Harness Evaluator

Score Interactive Test Harness traces and roll them into an EvalSuite report

Harness Evaluator scores an Interactive Test Harness trace against tool-call, schema-parity, artifact, and judge gates so harness runs plug straight into EvalSuite.

Quick Start

1. Score a trace

   Pass a trace dict with tool calls and artifacts, then call run:

   from praisonaiagents.eval import HarnessEvaluator
   
   trace = {
       "tool_calls": [{"name": "read_file"}, {"name": "write_file"}],
       "artifacts": ["out.txt"],
   }
   
   evaluator = HarnessEvaluator(trace=trace, name="smoke_scenario")
   result = evaluator.run(print_summary=True)
   print(result.passed, result.score)

2. Add gates

   Require specific artifacts and a minimum number of tool calls:

   from praisonaiagents.eval import HarnessEvaluator
   
   trace = {
       "tool_calls": [{"name": "read_file"}, {"name": "write_file"}],
       "artifacts": ["out.txt", "report.md"],
       "judge_score": 9.0,
   }
   
   evaluator = HarnessEvaluator(
       trace=trace,
       required_artifacts=["out.txt", "report.md"],
       min_tool_calls=2,
       judge_threshold=7.0,
       name="build_report",
   )
   result = evaluator.run(print_summary=True)

3. Gate on tool-schema parity

   from praisonaiagents.eval import HarnessEvaluator
   
   result = HarnessEvaluator(
       trace=trace,
       expected_schema_hash="a1b2c3d4e5f6a7b8",   # sha256(tool_schema)[:16]
       name="plugin_parity",
   ).run()
   
   print(result.schema_consistent)

4. Aggregate in an EvalSuite

   Add the evaluator to an EvalSuite and export an EvalReport:

   from praisonaiagents.eval import HarnessEvaluator, EvalSuite
   
   evaluator = HarnessEvaluator(
       trace={"tool_calls": 2, "artifacts": ["out.txt"]},
       required_artifacts=["out.txt"],
       name="scenario_a",
   )
   
   suite = EvalSuite(evaluators=[evaluator], name="harness_suite")
   report = suite.run(print_summary=True)

How It Works

A trace passes through four independent gates; the score is the fraction of gates that pass.

The trace reads tool calls from tool_calls → tool_trace → tools and artifacts from artifacts → files → outputs, so real harness traces score unmodified. An integer under a tool-calls key means “N calls”; dict-shaped artifacts are unwrapped via path/name/file/filename/artifact.

Configuration Options

Methods

Helper

harness_row_to_eval_case(row) maps a CSV harness row (id/name/scenario, prompt/input, optional fixture, rubric, expected) into an EvalCase with metadata["source"] = "harness".

run() returns a HarnessResult exposing passed, score, tool_call_count, tool_calls_sufficient, schema_hash, schema_consistent, artifacts_complete, missing_artifacts, judge_score, and judge_passed, plus to_dict(), to_json(), and print_summary().

Trace shape

Keys are order-insensitive — the first non-empty match wins.

Common Patterns

Aggregate CSV harness scenarios into an EvalReport with harness_row_to_eval_case():

import csv
from praisonaiagents.eval import harness_row_to_eval_case

with open("scenarios.csv") as f:
    cases = [harness_row_to_eval_case(row) for row in csv.DictReader(f)]

Enforce tool-schema parity between a native run and a plugin run:

from praisonaiagents.eval import HarnessEvaluator

evaluator = HarnessEvaluator(
    trace=plugin_run_trace,
    expected_schema_hash=native_schema_hash,   # from the native baseline run
    name="parity_check",
)
result = evaluator.run(print_summary=True)
assert result.schema_consistent

Gate CI on artifact completeness for a suite of scenarios:

from praisonaiagents.eval import HarnessEvaluator, EvalSuite

evaluators = [
    HarnessEvaluator(trace=t, required_artifacts=["out.txt"], name=t["id"])
    for t in scenario_traces
]
report = EvalSuite(evaluators=evaluators, name="ci_gate").run(print_summary=True)

Best Practices

Related