Skip to main content
Evaluation Suite runs every evaluator you enable in one pass and returns a single result you can gate CI on. The four core evaluators — Context, Loop, Harness, Accuracy — share one protocol and roll up into a single EvalSuiteResult with an aggregated score and pass/fail.

​
Quick Start

1

Run two evaluators as one gate

Pass evaluators to EvalSuite and read the aggregated result.
from praisonaiagents import Agent
from praisonaiagents.eval import EvalSuite, AccuracyEvaluator, HarnessEvaluator

agent = Agent(name="Assistant", instructions="Be helpful.")

suite = EvalSuite(evaluators=[
    AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"),
    HarnessEvaluator(trace={"tool_calls": [{"name": "write_file"}], "artifacts": ["out.txt"]}),
], name="ci-gate")

result = suite.run(print_summary=True)
print(result.overall_score, result.success)
2

Run all four in one suite

Reuse artifacts your agent already produces, then score every dimension at once.
from praisonaiagents import Agent
from praisonaiagents.eval import (
    EvalSuite, AccuracyEvaluator, ContextEvaluator,
    HarnessEvaluator, EvaluationLoop, LoopEvaluator,
)

agent = Agent(name="Assistant", instructions="Be helpful.")
loop_result = EvaluationLoop(agent=agent, criteria="Clear plan", threshold=8.0).run("Draft a launch plan.")

suite = EvalSuite(evaluators=[
    AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"),
    ContextEvaluator(trace_events=loop_result.trace_events, agent_order=["Assistant"]),
    HarnessEvaluator(trace={"tool_calls": [{"name": "write_file"}], "artifacts": ["plan.md"]}, required_artifacts=["plan.md"]),
], name="chl-gate")

result = suite.run(print_summary=True)
print(result.overall_score, result.success)

​
How It Works

EvalSuite fans out to each evaluator, collects every score, and aggregates one weighted overall_score.
EvaluatorScores
AccuracyEvaluatorOutput correctness against expected output (LLM-as-judge)
ContextEvaluatorMulti-agent handoff fidelity and context-budget compliance
LoopEvaluatorLoop health — convergence, wasted iterations, doom-loop guards
HarnessEvaluatorTest-harness traces — tool calls, artifacts, schema parity

​
Configuration Options

Each evaluator is documented on its own page; the suite only orchestrates them.

Context Evaluator

Multi-agent handoff and budget scoring

Loop Evaluator

Loop convergence and doom-loop health

Harness Evaluator

Test-harness trace scoring

Eval SDK Reference

Auto-generated reference for EvalSuite

​
Common Patterns

​
Gate CI on the aggregated score

Exit non-zero when the overall score drops below a threshold so a broken agent fails the build. 
import sys
from praisonaiagents.eval import EvalSuite, AccuracyEvaluator

result = EvalSuite(evaluators=[
    AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"),
]).run()

if result.overall_score < 8.0 or not result.success:
    print(f"Eval gate failed: {result.overall_score:.1f}/10")
    sys.exit(1)

​
Export the result as JSON

Persist the aggregated result for downstream dashboards. 
import json
from praisonaiagents.eval import EvalSuite, AccuracyEvaluator

result = EvalSuite(evaluators=[
    AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"),
]).run()

with open("eval-report.json", "w") as f:
    json.dump(result.to_dict(), f, indent=2, default=str)

​
Skip an evaluator conditionally

Drop AccuracyEvaluator when no LLM key is present and run the deterministic evaluators only. 
import os
from praisonaiagents.eval import EvalSuite, AccuracyEvaluator, HarnessEvaluator

evaluators = [HarnessEvaluator(trace={"tool_calls": [{"name": "write_file"}], "artifacts": ["out.txt"]})]
if os.getenv("OPENAI_API_KEY"):
    evaluators.append(AccuracyEvaluator(agent=agent, input_text="What is 2+2?", expected_output="4"))

result = EvalSuite(evaluators=evaluators).run()

​
Best Practices

Add the evaluator that catches the regression you fear — Accuracy for wrong answers, Context for lost handoffs, Loop for runaway iterations, Harness for missing artifacts.
Run the suite on a known-good agent first, then set the CI gate just below that overall_score so real regressions fail the build without flaky false alarms.
Harness, Context, and Loop make zero LLM calls — they run offline and reproducibly. Keep AccuracyEvaluator behind an API-key check so keyless CI stays green.
Run one evaluator directly while iterating on it; switch to EvalSuite once you gate CI on more than one dimension so you get a single aggregated pass/fail.

Context Evaluator

Score multi-agent handoff fidelity

Loop Evaluator

Score loop health and convergence

Harness Evaluator

Score Interactive Test Harness traces

Judge

LLM-as-judge used by AccuracyEvaluator