Evaluation Suite

Run Context, Loop, Harness, and Accuracy evaluators as one CI gate

Evaluation Suite runs every evaluator you enable in one pass and returns a single result you can gate CI on.

The four core evaluators — Context, Loop, Harness, Accuracy — share one protocol and roll up into a single EvalSuiteResult with an aggregated score and pass/fail.

Quick Start

1. Run two evaluators as one gate

   Pass evaluators to EvalSuite and read the aggregated result.

   from praisonaiagents import Agent
   from praisonaiagents.eval import EvalSuite, AccuracyEvaluator, HarnessEvaluator
   
   agent = Agent(name="Assistant", instructions="Be helpful.")
   
   suite = EvalSuite(evaluators=[
       AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"),
       HarnessEvaluator(trace={"tool_calls": [{"name": "write_file"}], "artifacts": ["out.txt"]}),
   ], name="ci-gate")
   
   result = suite.run(print_summary=True)
   print(result.overall_score, result.success)

2. Run all four in one suite

   Reuse artifacts your agent already produces, then score every dimension at once.

   from praisonaiagents import Agent
   from praisonaiagents.eval import (
       EvalSuite, AccuracyEvaluator, ContextEvaluator,
       HarnessEvaluator, EvaluationLoop, LoopEvaluator,
   )
   
   agent = Agent(name="Assistant", instructions="Be helpful.")
   loop_result = EvaluationLoop(agent=agent, criteria="Clear plan", threshold=8.0).run("Draft a launch plan.")
   
   suite = EvalSuite(evaluators=[
       AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"),
       ContextEvaluator(trace_events=loop_result.trace_events, agent_order=["Assistant"]),
       HarnessEvaluator(trace={"tool_calls": [{"name": "write_file"}], "artifacts": ["plan.md"]}, required_artifacts=["plan.md"]),
   ], name="chl-gate")
   
   result = suite.run(print_summary=True)
   print(result.overall_score, result.success)

How It Works

EvalSuite fans out to each evaluator, collects every score, and aggregates one weighted overall_score.

Every EvalSuiteResult also carries .fingerprint — an id for the setup this run was scored under; see Run Fingerprint.

Configuration Options

Each evaluator is documented on its own page; the suite only orchestrates them.

Common Patterns

Gate CI on the aggregated score

Exit non-zero when the overall score drops below a threshold so a broken agent fails the build.

import sys
from praisonaiagents.eval import EvalSuite, AccuracyEvaluator

result = EvalSuite(evaluators=[
    AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"),
]).run()

if result.overall_score < 8.0 or not result.success:
    print(f"Eval gate failed: {result.overall_score:.1f}/10")
    sys.exit(1)

Export the result as JSON

Persist the aggregated result for downstream dashboards.

import json
from praisonaiagents.eval import EvalSuite, AccuracyEvaluator

result = EvalSuite(evaluators=[
    AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"),
]).run()

with open("eval-report.json", "w") as f:
    json.dump(result.to_dict(), f, indent=2, default=str)

Skip an evaluator conditionally

Drop AccuracyEvaluator when no LLM key is present and run the deterministic evaluators only.

import os
from praisonaiagents.eval import EvalSuite, AccuracyEvaluator, HarnessEvaluator

evaluators = [HarnessEvaluator(trace={"tool_calls": [{"name": "write_file"}], "artifacts": ["out.txt"]})]
if os.getenv("OPENAI_API_KEY"):
    evaluators.append(AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"))

result = EvalSuite(evaluators=evaluators).run()

Compare across runs

Guard with assert_comparable before comparing two runs, so scores from a moved setup are refused instead of silently charted.

from praisonaiagents.eval import assert_comparable

assert_comparable(previous.fingerprint, current.fingerprint)

Best Practices

Related