Skip to main content
Score agent responses against your own criteria — recipe quality, data pipelines, manufacturing checks, or any custom rubric. 
from praisonaiagents import Agent
from praisonaiagents.eval import Judge

agent = Agent(name="assistant", instructions="Answer briefly and accurately")
output = agent.start("What is the capital of France?")

result = Judge(criteria="Answer is correct and concise").run(output=output)
print(f"Score: {result.score}/10")
The user runs the agent; the judge scores the final output against your custom criteria and returns a score with reasoning.

​
How It Works

​
Quick Start

1

Simple criteria

from praisonaiagents import Agent
from praisonaiagents.eval import Judge

agent = Agent(name="assistant", instructions="Be helpful")
output = agent.start("Explain photosynthesis in one sentence")

judge = Judge(criteria="Response is accurate and concise")
result = judge.run(output=output)
print(result.score, result.reasoning)
2

Domain-specific config

from praisonaiagents.eval import Judge, JudgeCriteriaConfig

config = JudgeCriteriaConfig(
    name="data_pipeline",
    description="Evaluate pipeline output",
    prompt_template="Evaluate this output:\n{output}\n\nScore 1-10 for data integrity and performance.",
    scoring_dimensions=["integrity", "performance"],
    threshold=7.0,
)

judge = Judge(criteria_config=config)
result = judge.run(output="ETL complete: 1.2M rows, 0 errors")
3

CLI recipe optimisation

praisonai recipe optimize my-recipe --criteria "No bottlenecks, validated output schema"

​
Configuration Options

​
JudgeCriteriaConfig

FieldTypeDefaultDescription
namestr—Criteria configuration name
descriptionstr—What is being evaluated
prompt_templatestr—Prompt with {output}, {input}, {expected} placeholders
scoring_dimensionsList[str]—Dimensions to score
thresholdfloat7.0Passing score

​
JudgeConfig

FieldTypeDefaultDescription
modelstrgpt-4o-miniLLM for judging
temperaturefloat0.1Sampling temperature
max_tokensint500Max response tokens
thresholdfloat7.0Passing score
criteriastrNoneSimple criteria string

​
Custom Optimisation Rules

Register domain-specific fix patterns with add_optimization_rule: 
from praisonaiagents.eval import add_optimization_rule

class WaterLeakRule:
    name = "water_leak"
    pattern = r"(leak|overflow|pressure.drop)"
    severity = "critical"

    def get_fix(self, context):
        return f"Check valve at {context.get('location', 'unknown')}"

add_optimization_rule("water_leak", WaterLeakRule)
FunctionDescription
add_optimization_rule(name, rule_class)Register a rule
get_optimization_rule(name)Get rule by name
list_optimization_rules()List registered rules

​
Best Practices

Vague rubrics like “good output” produce inconsistent scores. Use specific, testable dimensions.
Use 8–9 for production-critical checks; 6–7 for exploratory workflows.
Run the judge on good and bad examples before wiring it into optimisation loops.
Break evaluation into dimensions for clearer feedback and targeted fixes.

CLI Eval

Run evaluations from the command line

Evaluator Optimiser

Optimise agents with evaluator feedback loops