Trials Engine

Run K isolated attempts per eval case to measure pass rate, not one lucky sample

The trials engine runs each eval case K independent times so you get a pass rate and a frontier of shaky cases, not one lucky (or unlucky) sample.

Quick Start

1. Run K attempts per case

   Build an agent, wrap your cases in an EvalPackage, and call run_trials.

   from praisonaiagents import Agent
   from praisonaiagents.eval import EvalPackage, EvalCase, run_trials
   
   agent = Agent(name="summariser", instructions="Summarise the input in one sentence.")
   
   package = EvalPackage(name="summariser-suite", cases=[
       EvalCase(
           name="short",
           input="PraisonAI is an agent framework.",
           verify=lambda output, expected: "agent" in output.lower(),
       ),
       EvalCase(
           name="long",
           input="LLMs are large language models.",
           criteria=["mentions language models"],
       ),
   ])
   
   report = run_trials(agent, package, k=8, concurrency=4)
   print(report.pass_rates())   # {"short": 0.875, "long": 1.0}
   print(report.frontier())     # ["short"]

2. Async variant

   Use arun_trials inside an existing event loop.

   from praisonaiagents.eval import arun_trials
   
   report = await arun_trials(agent, package, k=8, concurrency=4)
   print(report.pass_rates())

3. Save and reload

   Persist the full report — including per-attempt trajectories — and load it back.

   from praisonaiagents.eval import TrialReport
   
   report.save("trials.json")               # creates parent dirs
   report = TrialReport.load("trials.json")  # full round-trip

4. Gate CI on the pass rate

   Read summary() for a stable, machine-parseable dict and fail loudly below your bar.

   summary = report.summary()
   assert summary["overall_pass_rate"] >= 0.9, summary["frontier"]

How It Works

Each case runs K times on a fresh, isolated copy of the agent; every completed attempt is scored, then results are assembled into a TrialReport.

Scoring — one contract, four fallbacks

Scoring runs only on stop_reason="completed" attempts and resolves in this exact order.

1. EvalCase.verify callable — if present, it wins. A raising verify is caught and recorded as a failed TrialScore with reason="verify error: ..." — one bad metric never aborts the report.
2. Tool assertions — if case.metadata["expected_tools"] is set, the engine reuses ReliabilityEvaluator._extract_tool_calls. Score is 1 − (missing / expected); passes only when every expected tool was called.
3. Criteria / expected via Judge — falls through to the existing Judge. The judge’s 0–10 score is normalised to [0, 1].
4. No scorer configured — a non-empty completion counts as a pass (reason="no scorer configured"); an empty completion is a fail.

Configuration Options

run_trials() and arun_trials() share the same signature.

TrialScore

TrialScore.to_dict() returns {"value", "passed", "reason"}.

TrialAttempt

TrialAttempt also has to_dict(), from_dict(), and to_eval_result() (adapts to the existing EvalResult shape).

TrialReport methods

New EvalCase / EvalResult fields

verify return values are coerced: bool → TrialScore(1.0/0.0), int/float → TrialScore(value=x, passed=x >= 0.5), TrialScore → returned as-is, anything else → truthiness fallback.

Isolation guarantees

Every attempt runs on an independent copy of the agent so K attempts are K real samples and your live agent is never mutated.

• Fresh agent_id and _session_id per attempt.
• chat_history rebound to a new list (not shared with the caller).
• Independent system-prompt cache.
• memory and knowledge writes severed (set to None for the attempt).

Timeouts & wedged agents

Common Patterns

Frontier-first iteration

Run k=8, look at report.frontier(), and focus improvement work there — mastered and zero-pass cases are lower-value to iterate on.

report = run_trials(agent, package, k=8, concurrency=4)
for case in report.frontier():
    print(case, report.pass_rates()[case])

CI gating with summary()

Assert a threshold on the machine-parseable dict inside a pytest job.

def test_agent_pass_rate():
    report = run_trials(agent, package, k=8, concurrency=4)
    summary = report.summary()
    assert summary["overall_pass_rate"] >= 0.9, summary["frontier"]

Save trajectories → SFT dataset

Persist the report, then export it to a trainer-ready dataset.

report.save("trials.json")
# praisonai-train from-trials trials.json  →  train.jsonl

Add --qc to quality-check the export. from-trials --qc keeps English agent trajectories by default — the Tamil script-purity check is auto-disabled for exports, so nothing is dropped as low_script_purity. See Export trials with QC for the qc_cfg opt-in and the ExportSummary.skipped_qc / qc_drops fields.

Which scorer should I use?

Pick the strongest signal your case can provide.

Best Practices

Related