Skip to main content
Harness Evaluator scores an Interactive Test Harness trace against tool-call, schema-parity, artifact, and judge gates so harness runs plug straight into EvalSuite.

​
Quick Start

1

Score a trace

Pass a trace dict with tool calls and artifacts, then call run:
from praisonaiagents.eval import HarnessEvaluator

trace = {
    "tool_calls": [{"name": "read_file"}, {"name": "write_file"}],
    "artifacts": ["out.txt"],
}

evaluator = HarnessEvaluator(trace=trace, name="smoke_scenario")
result = evaluator.run(print_summary=True)
print(result.passed, result.score)
2

Add gates

Require specific artifacts and a minimum number of tool calls:
from praisonaiagents.eval import HarnessEvaluator

trace = {
    "tool_calls": [{"name": "read_file"}, {"name": "write_file"}],
    "artifacts": ["out.txt", "report.md"],
    "judge_score": 9.0,
}

evaluator = HarnessEvaluator(
    trace=trace,
    required_artifacts=["out.txt", "report.md"],
    min_tool_calls=2,
    judge_threshold=7.0,
    name="build_report",
)
result = evaluator.run(print_summary=True)
3

Gate on tool-schema parity

from praisonaiagents.eval import HarnessEvaluator

result = HarnessEvaluator(
    trace=trace,
    expected_schema_hash="a1b2c3d4e5f6a7b8",   # sha256(tool_schema)[:16]
    name="plugin_parity",
).run()

print(result.schema_consistent)
4

Aggregate in an EvalSuite

Add the evaluator to an EvalSuite and export an EvalReport:
from praisonaiagents.eval import HarnessEvaluator, EvalSuite

evaluator = HarnessEvaluator(
    trace={"tool_calls": 2, "artifacts": ["out.txt"]},
    required_artifacts=["out.txt"],
    name="scenario_a",
)

suite = EvalSuite(evaluators=[evaluator], name="harness_suite")
report = suite.run(print_summary=True)

​
How It Works

A trace passes through four independent gates; the score is the fraction of gates that pass.
GateWhat it checksWhen it passes
ArtifactsEvery required_artifacts entry is present in the traceNo missing artifacts
Tool callsNumber of tool calls in the tracecount >= min_tool_calls
Schema hashsha256[:16] of tool_schema vs expected_schema_hashHashes match (or no hash set)
Judgejudge_score from the tracescore >= judge_threshold (or no score present)
The trace reads tool calls from tool_calls → tool_trace → tools and artifacts from artifacts → files → outputs, so real harness traces score unmodified. An integer under a tool-calls key means “N calls”; dict-shaped artifacts are unwrapped via path/name/file/filename/artifact.

​
Configuration Options

OptionTypeDefaultDescription
traceDict[str, Any]requiredHarness trace dict (tool_calls/tool_trace/tools, artifacts/files/outputs, tool_schema, judge_score).
required_artifactsOptional[List[str]]NoneArtifact paths/names that must be present to pass.
min_tool_callsint0Minimum number of tool calls required (0 = no gate).
expected_schema_hashOptional[str]NoneIf set, the trace’s tool-schema sha256[:16] hash must match this value.
judge_thresholdfloat7.0Judge score (0–10) at/above which the judge gate passes.
nameOptional[str]NoneOptional evaluation name.
save_results_pathOptional[str]NoneOptional path to persist the result JSON.
verboseboolFalseEnable verbose logging.
Methods
MethodReturnsPurpose
run(print_summary=False)HarnessResultComputes the 4 gates and aggregates
to_eval_result(case_name=None)EvalResultConverts the latest run for EvalReport export
Helper harness_row_to_eval_case(row) maps a CSV harness row (id/name/scenario, prompt/input, optional fixture, rubric, expected) into an EvalCase with metadata["source"] = "harness". run() returns a HarnessResult exposing passed, score, tool_call_count, tool_calls_sufficient, schema_hash, schema_consistent, artifacts_complete, missing_artifacts, judge_score, and judge_passed, plus to_dict(), to_json(), and print_summary().

Eval Module Reference

Full Python API for the eval package

​
Trace shape

Keys are order-insensitive — the first non-empty match wins.
FieldAccepted keysAccepted forms
Tool callstool_calls → tool_trace → toolslist of dicts/objects, or an int (counted)
Artifactsartifacts → files → outputslist of strings, list of dicts (path/name/file/filename/artifact), or a dict (values used)
Tool schematool_schemaany JSON-serialisable dict
Judge scorejudge_scorefloat (malformed values fail the judge gate)

​
Common Patterns

Aggregate CSV harness scenarios into an EvalReport with harness_row_to_eval_case(): 
import csv
from praisonaiagents.eval import harness_row_to_eval_case

with open("scenarios.csv") as f:
    cases = [harness_row_to_eval_case(row) for row in csv.DictReader(f)]
Enforce tool-schema parity between a native run and a plugin run: 
from praisonaiagents.eval import HarnessEvaluator

evaluator = HarnessEvaluator(
    trace=plugin_run_trace,
    expected_schema_hash=native_schema_hash,   # from the native baseline run
    name="parity_check",
)
result = evaluator.run(print_summary=True)
assert result.schema_consistent
Gate CI on artifact completeness for a suite of scenarios: 
from praisonaiagents.eval import HarnessEvaluator, EvalSuite

evaluators = [
    HarnessEvaluator(trace=t, required_artifacts=["out.txt"], name=t["id"])
    for t in scenario_traces
]
report = EvalSuite(evaluators=evaluators, name="ci_gate").run(print_summary=True)

​
Best Practices

A non-numeric judge_score fails the judge gate but does not raise, so one broken scenario won’t abort a mixed CI run. A trace with no judge_score treats the judge as “not configured” — the judge gate simply passes and does not count against you.
Tool calls are read from tool_calls, then tool_trace, then tools, and artifacts from artifacts, then files, then outputs. Whatever key your harness emits, the evaluator finds it — no reshaping step before scoring.
Set min_tool_calls when a scenario must actually exercise tools (a smoke test that made zero calls is suspect). Set expected_schema_hash when you need the tool surface to stay identical between two runs — for example native vs. plugin-provided tools.
Pass save_results_path to write the result JSON, or call result.to_json() yourself, so failed CI runs keep the per-gate breakdown for debugging.

Evaluation Suite

Run all four evaluators as one CI gate

Context Evaluator

Score multi-agent handoff fidelity

Evaluation Loop

Iterative agent → judge → improve loop

Judge

LLM-as-judge for evaluating outputs

Evaluation

Evaluators, suites, and reports

CHL Engineering

The Context / Harness / Loop rubric this evaluator scores against.