Dynamic Judge

Evaluate agent outputs against custom criteria for any domain

Score agent responses against your own criteria — recipe quality, data pipelines, manufacturing checks, or any custom rubric.

from praisonaiagents import Agent
from praisonaiagents.eval import Judge

agent = Agent(name="assistant", instructions="Answer briefly and accurately")
output = agent.start("What is the capital of France?")

result = Judge(criteria="Answer is correct and concise").run(output=output)
print(f"Score: {result.score}/10")

The user runs the agent; the judge scores the final output against your custom criteria and returns a score with reasoning.

How It Works

Quick Start

1. Simple criteria

   from praisonaiagents import Agent
   from praisonaiagents.eval import Judge
   
   agent = Agent(name="assistant", instructions="Be helpful")
   output = agent.start("Explain photosynthesis in one sentence")
   
   judge = Judge(criteria="Response is accurate and concise")
   result = judge.run(output=output)
   print(result.score, result.reasoning)

2. Domain-specific config

   from praisonaiagents.eval import Judge, JudgeCriteriaConfig
   
   config = JudgeCriteriaConfig(
       name="data_pipeline",
       description="Evaluate pipeline output",
       prompt_template="Evaluate this output:\n{output}\n\nScore 1-10 for data integrity and performance.",
       scoring_dimensions=["integrity", "performance"],
       threshold=7.0,
   )
   
   judge = Judge(criteria_config=config)
   result = judge.run(output="ETL complete: 1.2M rows, 0 errors")

3. CLI recipe optimisation

   praisonai recipe optimize my-recipe --criteria "No bottlenecks, validated output schema"

Configuration Options

JudgeCriteriaConfig

JudgeConfig

Custom Optimisation Rules

Register domain-specific fix patterns with add_optimization_rule:

from praisonaiagents.eval import add_optimization_rule

class WaterLeakRule:
    name = "water_leak"
    pattern = r"(leak|overflow|pressure.drop)"
    severity = "critical"

    def get_fix(self, context):
        return f"Check valve at {context.get('location', 'unknown')}"

add_optimization_rule("water_leak", WaterLeakRule)

Best Practices

Related