Praison AI

Judge

Unified LLM-as-judge for evaluating agent outputs

How It Works

Diagram

Result

Judge

Input

Output

Expected

Criteria

Judge

LLM Call

Parse Response

Score

Reasoning

Passed?

Quick Start

Accuracy Check

python
from praisonaiagents.eval import Judge

result = Judge().run(output="4", expected="4")
print(f"Score: {result.score}/10")
print(f"Passed: {result.passed}")

Criteria Check

python
from praisonaiagents.eval import Judge

judge = Judge(criteria="Response is helpful and accurate")
result = judge.run(output="Hello! How can I help you today?")
print(f"Score: {result.score}/10")

With Agent

python
from praisonaiagents import Agent
from praisonaiagents.eval import Judge

agent = Agent(instructions="You are a math tutor")
result = Judge().run(
    agent=agent,
    input_text="What is 2+2?",
    expected="4"
)

Judge Types

Diagram

Custom Judges

Built-in Judges

Registry

Judge Registry

AccuracyJudge

CriteriaJudge

Your Custom Judge

AccuracyJudge

Compares output against expected output

 CriteriaJudge

Evaluates against custom criteria

Configuration

model string

LLM model to use for judging

temperature float

Temperature for LLM calls (lower = more consistent)

threshold float

Score threshold for passing (1-10 scale)

criteria string

Custom criteria for evaluation

python
from praisonaiagents.eval import Judge, JudgeConfig

config = JudgeConfig(
    model="gpt-4o",
    temperature=0.1,
    threshold=8.0,
    criteria="Response is accurate and well-formatted"
)

judge = Judge(config=config)

Custom Judges

  1. Create Custom Judge
    python
    from praisonaiagents.eval import Judge

class RecipeJudge(Judge):
    """Judge for evaluating recipe quality."""

    CRITERIA_PROMPT = """Evaluate this recipe:

CRITERIA: Recipe is complete with ingredients and steps

RECIPE:
{output}

Score 1-10 based on completeness and clarity.

SCORE: [1-10]
REASONING: [explanation]
"""
  2. Register Judge
    python
    from praisonaiagents.eval import add_judge

add_judge("recipe", RecipeJudge)
  3. Use Judge
    python
    from praisonaiagents.eval import get_judge

RecipeJudge = get_judge("recipe")
judge = RecipeJudge()
result = judge.run(output=recipe_text)

Response format

A custom judge’s prompt asks the model for a SCORE: line and a REASONING: line, which the shared parser turns into a numeric score and an explanation.

text
SCORE: 8
REASONING: Complete recipe with clear ingredients and steps.
RuleBehaviour
Label matchSCORE: and REASONING: are matched at the start of a line. Markdown-bolded (SCORE: 1) and bullet-prefixed labels are tolerated.
Score clampingValues below 1 become 1.0; values above 10 become 10.0.
Unparseable scoreIf no numeric score can be read, the verdict fails closed — score=1.0, passed=False, reasoning="Unable to parse response".
Shared parserAccuracyEvaluator, CriteriaEvaluator, and ComparisonEvaluator all parse responses with parse_score_reasoning from praisonaiagents.eval.

Registry Functions

FunctionDescription
add_judge(name, class)Register a custom judge
get_judge(name)Get a judge class by name
list_judges()List all registered judges
remove_judge(name)Remove a registered judge
python
from praisonaiagents.eval import add_judge, get_judge, list_judges

# List available judges
print(list_judges())  # ['accuracy', 'criteria']

# Get a judge
AccuracyJudge = get_judge("accuracy")

The same registry is reachable from the shell:

bash
praisonai eval list          # short form — preferred
praisonai eval list-judges   # long form — still supported

Output:

Registered Judge Types:
  - accuracy
  - criteria
  - recipe

JudgeResult

ResponseField float

Quality score from 1-10

ResponseField bool

True only when a numeric score was parsed AND score >= threshold. An unparseable verdict is always passed=False.

ResponseField string

Explanation for the score

ResponseField list

Improvement suggestions

python
result = judge.run(output="Hello!")

print(result.score)        # 8.5
print(result.passed)       # True
print(result.reasoning)    # "Good greeting..."
print(result.suggestions)  # ["Could add more context"]
print(result.to_dict())    # Full dictionary

CLI Usage

Accuracy Mode
praisonai eval judge --output "4" --expected "4"
Criteria Mode
praisonai eval judge --output "Hello!" --criteria "Response is friendly"
From File
praisonai eval judge --file output.txt --criteria "Code is correct"
List Judges (short)
praisonai eval list
List Judges (long)
praisonai eval list-judges

Async Support

python
import asyncio
from praisonaiagents.eval import Judge

async def evaluate():
    judge = Judge(criteria="Response is helpful")
    result = await judge.run_async(output="Hello!")
    return result

result = asyncio.run(evaluate())

Best Practices

Choose the Right Mode
  • Use accuracy mode when you have a known expected output
  • Use criteria mode for subjective quality evaluation
  • Create custom judges for domain-specific evaluation
Set Appropriate Thresholds
  • Default threshold is 7.0 (70%)
  • Increase for critical evaluations
  • Decrease for exploratory testing
Use Specific Criteria
  • Be specific in your criteria
  • Include measurable aspects
  • Avoid vague terms like “good” or “nice”

Related

Safety Eval

Detect harmful, biased, or inappropriate outputs with a fail-closed guarantee