Agent Evaluation

Comprehensive evaluation framework for testing and benchmarking AI agents

Agent Evaluation

PraisonAI provides a comprehensive evaluation framework for testing and benchmarking AI agents. The evaluation system supports multiple evaluation types with zero performance impact when not in use.

Evaluation Types

!Eval Mode for Testing

Type	Description	Use Case
Accuracy	Compare output against expected output using LLM-as-judge	Verify correctness
Performance	Measure runtime and memory usage	Benchmark speed
Reliability	Verify expected tool calls are made	Test tool usage
Criteria	Evaluate against custom criteria	Quality assessment
Last Trace	Grade the most recently completed traced run from a stored pointer	CI gating + local “did my last run pass?” check

Installation

The evaluation framework is included in praisonaiagents:

bash

pip install praisonaiagents

Python Usage

Accuracy Evaluation

Compare agent outputs against expected results using an LLM judge:

python

from praisonaiagents import Agent
from praisonaiagents.eval import AccuracyEvaluator

# Create agent
agent = Agent(instructions="You are a math tutor. Answer concisely.")

# Create evaluator
evaluator = AccuracyEvaluator(
    agent=agent,
    input_text="What is 2 + 2?",
    expected_output="4",
    num_iterations=3,  # Run multiple times for statistical significance

)

# Run evaluation
result = evaluator.run(print_summary=True)

print(f"Average Score: {result.avg_score}/10")
print(f"Passed: {result.passed}")

Performance Evaluation

Benchmark agent runtime and memory usage:

python

from praisonaiagents import Agent
from praisonaiagents.eval import PerformanceEvaluator

agent = Agent(instructions="You are a helpful assistant.")

evaluator = PerformanceEvaluator(
    agent=agent,
    input_text="What is the capital of France?",
    num_iterations=10,  # Number of benchmark runs
    warmup_runs=2,      # Warmup runs before measurement
    track_memory=True,  # Track memory usage

)

result = evaluator.run(print_summary=True)

print(f"Average Time: {result.avg_run_time:.4f}s")
print(f"Min Time: {result.min_run_time:.4f}s")
print(f"Max Time: {result.max_run_time:.4f}s")
print(f"P95 Time: {result.p95_run_time:.4f}s")
print(f"Avg Memory: {result.avg_memory:.2f} MB")

Reliability Evaluation

Verify that agents call the expected tools:

python

from praisonaiagents import Agent
from praisonaiagents.eval import ReliabilityEvaluator

def search_web(query: str) -> str:
    """Search the web."""
    return f"Results for: {query}"

def calculate(expression: str) -> str:
    """Calculate expression."""
    # Use an AST allow-list for user math — see /features/custom-tool-safe-eval
    return str(float(expression))  # replace with safe_eval in production

agent = Agent(
    instructions="You have search and calculator tools.",
    tools=[search_web, calculate]
)

evaluator = ReliabilityEvaluator(
    agent=agent,
    input_text="Search for weather and calculate 25 * 4",
    expected_tools=["search_web", "calculate"],
    forbidden_tools=["delete_file"],  # Should NOT be called

)

result = evaluator.run(print_summary=True)

print(f"Passed: {result.passed}")
print(f"Pass Rate: {result.pass_rate:.1%}")

Criteria Evaluation

Evaluate outputs against custom criteria using LLM-as-judge:

python

from praisonaiagents import Agent
from praisonaiagents.eval import CriteriaEvaluator

agent = Agent(instructions="You are a customer service agent.")

# Numeric scoring (1-10)
evaluator = CriteriaEvaluator(
    criteria="Response is helpful, empathetic, and provides a clear solution",
    agent=agent,
    input_text="My order hasn't arrived yet.",
    scoring_type="numeric",  # Score 1-10
    threshold=7.0,           # Pass if score >= 7
    num_iterations=2,

)

result = evaluator.run(print_summary=True)

print(f"Average Score: {result.avg_score}/10")
print(f"Pass Rate: {result.pass_rate:.1%}")

# Binary scoring (pass/fail)
binary_evaluator = CriteriaEvaluator(
    criteria="Response does not contain offensive language",
    agent=agent,
    input_text="Tell me a joke",
    scoring_type="binary",

)

binary_result = binary_evaluator.run(print_summary=True)

Failure Callbacks

Handle evaluation failures with callbacks:

python

from praisonaiagents.eval import CriteriaEvaluator

def handle_failure(score):
    print(f"ALERT: Evaluation failed with score {score.score}")
    print(f"Reasoning: {score.reasoning}")
    # Send alert, log to monitoring system, etc.

evaluator = CriteriaEvaluator(
    criteria="Response is professional",
    agent=agent,
    input_text="Help me",
    on_fail=handle_failure,
    threshold=8.0
)

evaluator.run()

Evaluate Pre-generated Outputs

Evaluate outputs without running the agent:

python

from praisonaiagents.eval import AccuracyEvaluator, CriteriaEvaluator

# Accuracy evaluation of pre-generated output
accuracy_eval = AccuracyEvaluator(
    func=lambda x: "unused",  # Placeholder
    input_text="What is 2+2?",
    expected_output="4"
)

result = accuracy_eval.evaluate_output("The answer is 4")

# Criteria evaluation of pre-generated output
criteria_eval = CriteriaEvaluator(
    criteria="Response is helpful and accurate",
    func=lambda x: "unused"
)

result = criteria_eval.evaluate_output("Here's how to solve that...")

Saving Results

Save evaluation results to files:

python

evaluator = AccuracyEvaluator(
    agent=agent,
    input_text="Test input",
    expected_output="Expected output",
    save_results_path="results/{name}_{eval_id}.json"  # Supports placeholders
)

result = evaluator.run()
# Results automatically saved to file

Building agents from YAML (without running them)

AgentsGenerator.build_agents() returns the constructed agent instances from an agents.yaml without running them. The with block keeps the per-agent tool_timeout thread pool alive while you drive the agents yourself (for an evaluator, a sandbox, a one-off introspection).

python

from praisonai.agents_generator import AgentsGenerator
from praisonai.framework_adapters.registry import get_default_registry
from praisonai.llm.config import build_config_list

registry = get_default_registry()
adapter = registry.create(registry.pick_default())

with AgentsGenerator(
    agent_file="agents.yaml",
    framework=adapter.name,
    config_list=build_config_list(),
    adapter=adapter,
) as gen:
    agents = gen.build_agents()
    for agent in agents:
        print(agent.name)

If your YAML sets framework: crewai, pass that framework to AgentsGenerator(...) as well so the eval build and your own build stay aligned.

Card

Full constructor signature and lifecycle methods

CLI Usage

Accuracy Evaluation

bash

praisonai eval accuracy \
    --prompt "What is 2+2?" \  # Direct prompt (no agents.yaml needed)
    --expected "4"

# Or with agents.yaml:
praisonai eval accuracy \
    --agent agents.yaml \
    --input "What is 2+2?" \
    --expected "4" \
    --iterations 3 \
    --output results.json \
    --verbose

Performance Evaluation

bash

praisonai eval performance \
    --agent agents.yaml \
    --input "Hello" \
    --iterations 10 \
    --warmup 2 \
    --memory \
    --output perf_results.json

Reliability Evaluation

bash

praisonai eval reliability \
    --agent agents.yaml \
    --input "Search for weather" \
    --expected-tools "search_web,calculate" \
    --forbidden-tools "delete_file" \
    --output reliability.json

Criteria Evaluation

bash

praisonai eval criteria \
    --agent agents.yaml \
    --input "Help me with my order" \
    --criteria "Response is helpful and professional" \
    --scoring numeric \
    --threshold 7.0 \
    --iterations 2 \
    --output criteria.json

Last Trace Evaluation

Grade the last completed traced run from a stored pointer — no manual JSON export needed.

bash

praisonai eval last-trace --min-tool-calls 1 --json
# {"passed": true, "score": 1.0, "tool_call_count": 1}

Options:

Option	Type	Default	Description
`--min-tool-calls`	`int`	`1`	Minimum number of tool calls required to pass. `0` disables the gate.
`--json`	flag	`False`	Emit result as a single-line JSON object to stdout.

Exit codes:

Code	Meaning
`0`	Pass (`result.passed` is `True`)
`1`	Fail (`result.passed` is `False`), or malformed trace file
`2`	No completed traced run found (missing pointer, or pointer path no longer exists on disk)

See Grade Your Last Run for the full run → record → grade flow and the praisonaiagents.trace Python API.

List Judges

List every judge type registered via add_judge() (built-ins and any custom judges loaded through your agents.yaml).

Short form (preferred)

praisonai eval list

Long form

praisonai eval list-judges

Both forms print:

Registered Judge Types:
  - accuracy
  - criteria
  - recipe

Use this after registering a custom judge (see Custom Judges) to confirm the registration took effect.

Batch Evaluation

Run multiple test cases from a JSON file:

bash

praisonai eval batch \
    --agent agents.yaml \
    --test-file tests.json \
    --batch-type accuracy \
    --output batch_results.json

Test file format (tests.json):

json

[
    {
        "input": "What is 2+2?",
        "expected": "4"
    },
    {
        "input": "What is the capital of France?",
        "expected": "Paris"
    }
]

CLI Options Reference

Common Options

Option	Short	Description
`--agent`	`-a`	Path to agents.yaml file
`--output`	`-o`	Output file for results
`--verbose`	`-v`	Enable verbose output
`--quiet`	`-q`	Suppress JSON output

--agent agents.yaml honors a framework: key inside the YAML. If omitted, the registry’s default framework is used.

Accuracy Options

Option	Short	Description
`--input`	`-i`	Input text for the agent
`--expected`	`-e`	Expected output
`--iterations`	`-n`	Number of iterations
`--model`	`-m`	LLM model for judging

Performance Options

Option	Short	Description
`--input`	`-i`	Input text for the agent
`--iterations`	`-n`	Number of benchmark iterations
`--warmup`	`-w`	Number of warmup runs
`--memory`		Track memory usage

Reliability Options

Option	Short	Description
`--input`	`-i`	Input text for the agent
`--expected-tools`	`-t`	Expected tools (comma-separated)
`--forbidden-tools`	`-f`	Forbidden tools (comma-separated)

Criteria Options

Option	Short	Description
`--input`	`-i`	Input text for the agent
`--criteria`	`-c`	Evaluation criteria
`--scoring`	`-s`	Scoring type (numeric/binary)
`--threshold`		Pass threshold for numeric scoring
`--iterations`	`-n`	Number of iterations
`--model`	`-m`	LLM model for judging

Last Trace Options

Option	Short	Description
`--min-tool-calls`		Minimum tool calls required to pass (default `1`; `0` disables the gate)
`--json`		Emit result as a single-line JSON object to stdout

Result Data Structures

AccuracyResult

python

result.evaluations  # List of individual scores
result.avg_score    # Average score (0-10)
result.min_score    # Minimum score
result.max_score    # Maximum score
result.std_dev      # Standard deviation
result.passed       # True if avg_score >= 7
result.to_dict()    # Convert to dictionary
result.to_json()    # Convert to JSON string

PerformanceResult

python

result.metrics          # List of PerformanceMetrics
result.avg_run_time     # Average runtime in seconds
result.min_run_time     # Minimum runtime
result.max_run_time     # Maximum runtime
result.median_run_time  # Median runtime
result.p95_run_time     # 95th percentile runtime
result.avg_memory       # Average memory usage (MB)
result.max_memory       # Peak memory usage (MB)

ReliabilityResult

python

result.tool_results  # List of ToolCallResult
result.passed_calls  # Tools that passed
result.failed_calls  # Tools that failed
result.pass_rate     # Pass rate (0-1)
result.passed        # True if all checks passed
result.status        # "PASSED" or "FAILED"

CriteriaResult

python

result.evaluations  # List of CriteriaScore
result.criteria     # The evaluation criteria
result.scoring_type # "numeric" or "binary"
result.threshold    # Pass threshold
result.avg_score    # Average score
result.pass_rate    # Pass rate (0-1)
result.passed       # True if passed threshold

LLM Judge in Interactive Tests

The interactive test runner integrates LLM-as-judge evaluation for automated response quality assessment. This allows you to validate not just tool calls and file outputs, but also the quality of agent responses.

Using Judge in CSV Tests

Add a judge_rubric column to your CSV test file:

csv

id,name,prompts,judge_rubric,judge_threshold,judge_model
test_01,Helpful Response,"Explain Python decorators",Response is clear and accurate,7.0,gpt-4o-mini
test_02,Code Quality,"Create a function to sort a list",Code is correct and well-documented,8.0,gpt-4o-mini

Judge Configuration

Option	Default	Description
`judge_rubric`	(empty)	Evaluation criteria for the judge
`judge_threshold`	7.0	Minimum score to pass (1-10 scale)
`judge_model`	gpt-4o-mini	Model used for evaluation

CLI Options for Judge

bash

# Run with judge evaluation
praisonai test interactive --csv tests.csv

# Skip judge even if rubric is present
praisonai test interactive --csv tests.csv --no-judge

# Use a different judge model
praisonai test interactive --csv tests.csv --judge-model gpt-4o

Judge Output

When judge evaluation is enabled, results include:

Score: 1-10 rating based on rubric
Passed: Whether score meets threshold
Reasoning: Detailed explanation of the score

Example artifact (judge_result.json):

json

{
  "score": 8.5,
  "passed": true,
  "reasoning": "SCORE: 8.5\nREASONING: The response clearly explains...",
  "threshold": 7.0,
  "model": "gpt-4o-mini"
}

Writing Effective Rubrics

Good rubrics are:

Specific: “Response includes code example” vs “Response is good”
Measurable: “Explains at least 3 benefits” vs “Comprehensive”
Relevant: Focus on what matters for the test case

Examples:

csv

# Good rubrics
"Response contains working Python code with proper error handling"
"Explanation covers syntax, use cases, and at least one example"
"File was created with correct content and proper formatting"

# Avoid vague rubrics
"Response is helpful"
"Code is good"
"Answer is correct"

Best Practices

Use Multiple Iterations: Run evaluations multiple times for statistical significance
Warmup Runs: Use warmup runs for performance benchmarks to avoid cold-start effects
Save Results: Always save results for tracking and comparison
Custom Criteria: Write specific, measurable criteria for criteria evaluations
Batch Testing: Use batch evaluation for regression testing
CI/CD Integration: Integrate evaluations into your CI/CD pipeline

Examples

See the examples directory for complete examples:

GitHub Advanced Test Rubrics

The github-advanced test suite uses specialized LLM judge rubrics for evaluating GitHub workflow quality:

Available Rubrics

Rubric	Description	Key Criteria
PR Quality	Evaluates pull request quality	Title clarity, body completeness, issue reference, branch naming
Code Quality	Evaluates code changes	Correctness, tests pass, coverage, type hints, no regressions
Workflow Correctness	Evaluates GitHub workflow	Repo created, issue created, PR links issue
CI/CD Quality	Evaluates CI configuration	Valid YAML, checkout step, setup step, triggers
Documentation	Evaluates docs changes	Links valid, content accurate, formatting correct
Multi-Agent	Evaluates agent collaboration	Handoff, task completion, context preservation

Rubric Structure

Each rubric contains weighted criteria:

python

from tests.live.interactive.github_advanced.judge_rubric import (
    PR_QUALITY_RUBRIC,
    evaluate_with_rubric,
)

# Get evaluation prompt
prompt = PR_QUALITY_RUBRIC.get_prompt()

# Evaluate with context
result = evaluate_with_rubric(
    rubric=PR_QUALITY_RUBRIC,
    context={
        "pr_title": "Fix subtract sign bug",
        "pr_body": "Closes #1. Fixed the subtract function.",
        "branch": "fix/subtract-sign",
    },
    judge_model="gpt-4o-mini",
)

print(result["overall_score"])  # 0-10
print(result["passed"])  # True/False

Scenario to Rubric Mapping

Scenario	Rubrics Applied
GH_01	PR Quality, Code Quality, Workflow Correctness
GH_02	PR Quality, CI/CD Quality, Workflow Correctness
GH_03	PR Quality, Code Quality, Workflow Correctness
GH_04	PR Quality, Documentation, Workflow Correctness
GH_05	PR Quality, Multi-Agent, Workflow Correctness