LLM-as-Judge

Evaluate agent outputs using LLM-based quality assessment

python

from praisonaiagents import Agent

judge = Agent(name="judge", instructions="Evaluate agent outputs for quality and accuracy.")
agent = Agent(name="writer", instructions="Write a short essay.")
output = agent.start("Write about climate change.")
judge.start(f"Rate this response from 1-10: {output}")

Use an LLM to evaluate and score agent outputs for accuracy, quality, and custom criteria.

The user runs a task agent, then the judge agent scores the output against your rubric.

Diagram

Quick Start

Evaluate Output

Python

python

from praisonaiagents.eval import Judge

# Simple accuracy check
result = Judge().run(output="4", expected="4", input="What is 2+2?")
print(f"Score: {result.score}/10, Passed: {result.passed}")

TypeScript

typescript

import { Judge } from 'praisonai';

// Simple accuracy check
const result = await new Judge().run({
  output: "4",
  expected: "4", 
  input: "What is 2+2?"
});
console.log(`Score: ${result.score}/10, Passed: ${result.passed}`);

CLI

bash

# Judge with expected output
praisonai eval judge --input "The answer is 4" --expected "4"

# Judge with criteria
praisonai eval judge --input "Hello, how can I help?" --criteria "Response is helpful"

How It Works

Diagram

Phase	What happens
1. Produce	The task agent generates an output
2. Submit	You pass the output plus `expected` or `criteria` to `Judge.run()`
3. Score	The judge LLM rates the output on a 1–10 scale
4. Return	`JudgeResult` reports the score, pass/fail, and improvement suggestions

Choosing an Evaluation Approach

Match the judge setup to how strict and comparative your scoring needs to be.

Diagram

Evaluation Modes

Accuracy Evaluation

Compare agent output against an expected result:

Python

python

from praisonaiagents.eval import Judge

judge = Judge()
result = judge.run(
    output="Python is a high-level programming language",
    expected="Python is a programming language",
    input="What is Python?"
)

print(f"Score: {result.score}/10")
print(f"Reasoning: {result.reasoning}")

TypeScript

typescript

import { Judge } from 'praisonai';

const judge = new Judge();
const result = await judge.run({
  output: "Python is a high-level programming language",
  expected: "Python is a programming language",
  input: "What is Python?"
});

console.log(`Score: ${result.score}/10`);
console.log(`Reasoning: ${result.reasoning}`);

Criteria Evaluation

Evaluate output against custom criteria:

Python

python

from praisonaiagents.eval import Judge

judge = Judge(criteria="Response is helpful, accurate, and concise")
result = judge.run(output="Hello! I'm here to help you with any questions.")

if result.passed:
    print("✅ Output meets criteria")
else:
    print("❌ Output needs improvement")
    for suggestion in result.suggestions:
        print(f"  • {suggestion}")

TypeScript

typescript

import { Judge } from 'praisonai';

const judge = new Judge({ 
  criteria: "Response is helpful, accurate, and concise" 
});
const result = await judge.run({
  output: "Hello! I'm here to help you with any questions."
});

if (result.passed) {
  console.log("✅ Output meets criteria");
} else {
  console.log("❌ Output needs improvement");
  result.suggestions.forEach(s => console.log(`  • ${s}`));
}

Recipe/Workflow Evaluation

Evaluate multi-agent workflow execution:

Python

python

from praisonaiagents.eval import RecipeJudge

judge = RecipeJudge(mode="context")  # or "memory", "knowledge"
result = judge.run(
    output="Final workflow output...",
    expected="Complete analysis with citations"
)

TypeScript

typescript

import { RecipeJudge } from 'praisonai';

const judge = new RecipeJudge({ mode: "context" });
const result = await judge.run({
  output: "Final workflow output...",
  expected: "Complete analysis with citations"
});

Configuration

JudgeConfig

model string

LLM model to use for evaluation

temperature number

Temperature for consistent scoring (lower = more consistent)

maxTokens number

Maximum tokens for LLM response

threshold number

Score threshold for passing (1-10 scale)

criteria string

Custom evaluation criteria

Python

python

from praisonaiagents.eval import Judge, JudgeConfig

config = JudgeConfig(
    model="gpt-4o",
    temperature=0.0,
    threshold=8.0,
    criteria="Response must be technically accurate"
)

judge = Judge(config=config)

TypeScript

typescript

import { Judge, JudgeConfig } from 'praisonai';

const config: JudgeConfig = {
  model: "gpt-4o",
  temperature: 0.0,
  threshold: 8.0,
  criteria: "Response must be technically accurate"
};

const judge = new Judge({ config });

JudgeResult

The result object contains:

Field	Type	Description
`score`	number	Quality score (1-10)
`passed`	boolean	`True` only when a numeric score was parsed AND `score >= threshold`. An unparseable verdict is always `passed=False`.
`reasoning`	string	Explanation for the score
`output`	string	The judged output
`expected`	string?	Expected output (if provided)
`criteria`	string?	Criteria used (if provided)
`suggestions`	string[]	Improvement suggestions
`timestamp`	number	When evaluation occurred

Fail-Closed Parsing

If the judge’s LLM response can’t be parsed for a numeric SCORE:, the verdict fails closed.

Empty responses, JSON {}, and prose-only replies like "the answer is wrong" all fail closed.

Markdown-bolded labels like SCORE: 1 (the default output style of most models) and bullet-prefixed labels are tolerated — the parser strips **, *, ` `, and leading –/>` before reading the value. Prompts no longer need to warn the model against bolding.

Diagram

The safety grader (BaseLLMGrader) already followed this pattern — Judge now matches it. See Safety Evaluation.

Judge with Agent

Evaluate an agent’s response directly:

Python

python

from praisonaiagents import Agent
from praisonaiagents.eval import Judge

agent = Agent(
    name="Math Helper",
    instructions="You solve math problems"
)

judge = Judge()
result = judge.run(
    agent=agent,
    input="What is 15 * 7?",
    expected="105"
)

print(f"Agent scored: {result.score}/10")

TypeScript

typescript

import { Agent, Judge } from 'praisonai';

const agent = new Agent({
  name: "Math Helper",
  instructions: "You solve math problems"
});

const judge = new Judge();
const result = await judge.run({
  agent,
  input: "What is 15 * 7?",
  expected: "105"
});

console.log(`Agent scored: ${result.score}/10`);

Custom Judges

Register Custom Judge

Python

python

from praisonaiagents.eval import Judge, add_judge, get_judge, list_judges

class CodeQualityJudge(Judge):
    """Judge for evaluating code quality."""

    def __init__(self, **kwargs):
        super().__init__(
            criteria="Code is clean, efficient, and well-documented",
            **kwargs
        )

# Register
add_judge("code_quality", CodeQualityJudge)

# Use
JudgeClass = get_judge("code_quality")
judge = JudgeClass()

# List all judges
print(list_judges())  # ['accuracy', 'criteria', 'recipe', 'code_quality']

TypeScript

typescript

import { Judge, addJudge, getJudge, listJudges } from 'praisonai';

class CodeQualityJudge extends Judge {
  constructor(options = {}) {
    super({
      ...options,
      criteria: "Code is clean, efficient, and well-documented"
    });
  }
}

// Register
addJudge("code_quality", CodeQualityJudge);

// Use
const JudgeClass = getJudge("code_quality");
const judge = new JudgeClass();

// List all judges
console.log(listJudges()); // ['accuracy', 'criteria', 'recipe', 'code_quality']

Domain-Agnostic Evaluation

Use JudgeCriteriaConfig for any domain:

Python

python

from praisonaiagents.eval import Judge, JudgeCriteriaConfig

# Water flow optimization
config = JudgeCriteriaConfig(
    name="water_flow",
    description="Evaluate water flow optimization",
    prompt_template="""Evaluate the water flow configuration:
{output}

Score based on:
- Flow rate efficiency
- Pressure optimization  
- Resource conservation

SCORE: [1-10]
REASONING: [explanation]
SUGGESTIONS: [improvements]""",
    scoring_dimensions=["flow_rate", "pressure", "efficiency"],
    threshold=7.0
)

judge = Judge(criteria_config=config)
result = judge.run(output="Flow rate: 50L/min, Pressure: 2.5 bar")

TypeScript

typescript

import { Judge, JudgeCriteriaConfig } from 'praisonai';

const config: JudgeCriteriaConfig = {
  name: "water_flow",
  description: "Evaluate water flow optimization",
  promptTemplate: `Evaluate the water flow configuration:
{output}

Score based on:
- Flow rate efficiency
- Pressure optimization
- Resource conservation

SCORE: [1-10]
REASONING: [explanation]
SUGGESTIONS: [improvements]`,
  scoringDimensions: ["flow_rate", "pressure", "efficiency"],
  threshold: 7.0
};

const judge = new Judge({ criteriaConfig: config });
const result = await judge.run({ 
  output: "Flow rate: 50L/min, Pressure: 2.5 bar" 
});

Async Evaluation

Python

python

import asyncio
from praisonaiagents.eval import Judge

async def evaluate_outputs():
    judge = Judge(criteria="Response is helpful")

    outputs = [
        "Hello! How can I help?",
        "I don't know.",
        "Let me help you with that!"
    ]

    results = await asyncio.gather(*[
        judge.run_async(output=output)
        for output in outputs
    ])

    for output, result in zip(outputs, results):
        print(f"{output[:30]}... → {result.score}/10")

asyncio.run(evaluate_outputs())

TypeScript

typescript

import { Judge } from 'praisonai';

async function evaluateOutputs() {
  const judge = new Judge({ criteria: "Response is helpful" });

  const outputs = [
    "Hello! How can I help?",
    "I don't know.",
    "Let me help you with that!"
  ];

  const results = await Promise.all(
    outputs.map(output => judge.runAsync({ output }))
  );

  outputs.forEach((output, i) => {
    console.log(`${output.slice(0, 30)}... → ${results[i].score}/10`);
  });
}

evaluateOutputs();

CLI Reference

bash

# Basic judge
praisonai eval judge --input "Output to evaluate"

# With expected output (accuracy mode)
praisonai eval judge --input "4" --expected "4"

# With criteria
praisonai eval judge --input "Hello!" --criteria "Response is friendly"

# Custom threshold
praisonai eval judge --input "Test" --threshold 8.0

# JSON output
praisonai eval judge --input "Test" --json

# Custom model
praisonai eval judge --input "Test" --model gpt-4o

Best Practices

Use Low Temperature

Set temperature: 0.1 or lower for consistent scoring across evaluations.

Define Clear Criteria

Be specific about what constitutes a good output. Vague criteria lead to inconsistent scores.

Set Appropriate Thresholds

7.0: Standard quality bar
8.0: High quality requirement
6.0: Lenient evaluation

Review Suggestions

The suggestions array provides actionable improvements. Use them to iterate on agent prompts.

Evaluation Framework

Complete evaluation suite

Agent Testing

Test your agents

LLM-as-Judge

Quick Start

Python

TypeScript

CLI

How It Works

Choosing an Evaluation Approach

Evaluation Modes

Accuracy Evaluation

Python

TypeScript

Criteria Evaluation

Python

TypeScript

Recipe/Workflow Evaluation

Python

TypeScript

Configuration

JudgeConfig

Python

TypeScript

JudgeResult

Fail-Closed Parsing

Judge with Agent

Python

TypeScript

Custom Judges

Register Custom Judge

Python

TypeScript

Domain-Agnostic Evaluation

Python

TypeScript

Async Evaluation

Python

TypeScript

CLI Reference

Best Practices

Related