> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Harness Evaluator
> Score Interactive Test Harness traces and roll them into an EvalSuite report
Harness Evaluator scores an Interactive Test Harness trace against tool-call, schema-parity, artifact, and judge gates so harness runs plug straight into `EvalSuite`.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
Trace[๐ Harness Trace] --> Eval[๐งช HarnessEvaluator]
Eval --> Result[โ
HarnessResult]
Result --> Suite[๐ EvalSuite โ EvalReport]
classDef input fill:#8B0000,stroke:#7C90A0,color:#fff
classDef process fill:#189AB4,stroke:#7C90A0,color:#fff
classDef result fill:#10B981,stroke:#7C90A0,color:#fff
classDef report fill:#6366F1,stroke:#7C90A0,color:#fff
class Trace input
class Eval process
class Result result
class Suite report
```
## Quick Start
Pass a trace dict with tool calls and artifacts, then call `run`:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import HarnessEvaluator
trace = {
"tool_calls": [{"name": "read_file"}, {"name": "write_file"}],
"artifacts": ["out.txt"],
}
evaluator = HarnessEvaluator(trace=trace, name="smoke_scenario")
result = evaluator.run(print_summary=True)
print(result.passed, result.score)
```
Require specific artifacts and a minimum number of tool calls:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import HarnessEvaluator
trace = {
"tool_calls": [{"name": "read_file"}, {"name": "write_file"}],
"artifacts": ["out.txt", "report.md"],
"judge_score": 9.0,
}
evaluator = HarnessEvaluator(
trace=trace,
required_artifacts=["out.txt", "report.md"],
min_tool_calls=2,
judge_threshold=7.0,
name="build_report",
)
result = evaluator.run(print_summary=True)
```
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import HarnessEvaluator
result = HarnessEvaluator(
trace=trace,
expected_schema_hash="a1b2c3d4e5f6a7b8", # sha256(tool_schema)[:16]
name="plugin_parity",
).run()
print(result.schema_consistent)
```
Add the evaluator to an `EvalSuite` and export an `EvalReport`:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import HarnessEvaluator, EvalSuite
evaluator = HarnessEvaluator(
trace={"tool_calls": 2, "artifacts": ["out.txt"]},
required_artifacts=["out.txt"],
name="scenario_a",
)
suite = EvalSuite(evaluators=[evaluator], name="harness_suite")
report = suite.run(print_summary=True)
```
***
## How It Works
A trace passes through four independent gates; the score is the fraction of gates that pass.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
Trace[๐ Harness Trace] --> Art{๐ Artifacts complete?}
Trace --> Tool{๐ง Tool calls โฅ min?}
Trace --> Schema{๐ Schema hash matches?}
Trace --> Judge{โ๏ธ Judge โฅ threshold?}
Art --> Score[๐ Aggregate score]
Tool --> Score
Schema --> Score
Judge --> Score
classDef input fill:#8B0000,stroke:#7C90A0,color:#fff
classDef gate fill:#189AB4,stroke:#7C90A0,color:#fff
classDef result fill:#10B981,stroke:#7C90A0,color:#fff
class Trace input
class Art,Tool,Schema,Judge gate
class Score result
```
| Gate | What it checks | When it passes |
| --------------- | -------------------------------------------------------- | ------------------------------------------------ |
| **Artifacts** | Every `required_artifacts` entry is present in the trace | No missing artifacts |
| **Tool calls** | Number of tool calls in the trace | `count >= min_tool_calls` |
| **Schema hash** | sha256\[:16] of `tool_schema` vs `expected_schema_hash` | Hashes match (or no hash set) |
| **Judge** | `judge_score` from the trace | `score >= judge_threshold` (or no score present) |
The trace reads tool calls from `tool_calls` โ `tool_trace` โ `tools` and artifacts from `artifacts` โ `files` โ `outputs`, so real harness traces score unmodified. An integer under a tool-calls key means "N calls"; dict-shaped artifacts are unwrapped via `path`/`name`/`file`/`filename`/`artifact`.
***
## Configuration Options
| Option | Type | Default | Description |
| ---------------------- | --------------------- | -------- | -------------------------------------------------------------------------------------------------------------------- |
| `trace` | `Dict[str, Any]` | required | Harness trace dict (`tool_calls`/`tool_trace`/`tools`, `artifacts`/`files`/`outputs`, `tool_schema`, `judge_score`). |
| `required_artifacts` | `Optional[List[str]]` | `None` | Artifact paths/names that must be present to pass. |
| `min_tool_calls` | `int` | `0` | Minimum number of tool calls required (`0` = no gate). |
| `expected_schema_hash` | `Optional[str]` | `None` | If set, the trace's tool-schema sha256\[:16] hash must match this value. |
| `judge_threshold` | `float` | `7.0` | Judge score (0โ10) at/above which the judge gate passes. |
| `name` | `Optional[str]` | `None` | Optional evaluation name. |
| `save_results_path` | `Optional[str]` | `None` | Optional path to persist the result JSON. |
| `verbose` | `bool` | `False` | Enable verbose logging. |
**Methods**
| Method | Returns | Purpose |
| -------------------------------- | --------------- | ----------------------------------------------- |
| `run(print_summary=False)` | `HarnessResult` | Computes the 4 gates and aggregates |
| `to_eval_result(case_name=None)` | `EvalResult` | Converts the latest run for `EvalReport` export |
**Helper**
`harness_row_to_eval_case(row)` maps a CSV harness row (`id`/`name`/`scenario`, `prompt`/`input`, optional `fixture`, `rubric`, `expected`) into an `EvalCase` with `metadata["source"] = "harness"`.
`run()` returns a `HarnessResult` exposing `passed`, `score`, `tool_call_count`, `tool_calls_sufficient`, `schema_hash`, `schema_consistent`, `artifacts_complete`, `missing_artifacts`, `judge_score`, and `judge_passed`, plus `to_dict()`, `to_json()`, and `print_summary()`.
Full Python API for the eval package
***
## Trace shape
Keys are order-insensitive โ the first non-empty match wins.
| Field | Accepted keys | Accepted forms |
| ----------- | ------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Tool calls | `tool_calls` โ `tool_trace` โ `tools` | list of dicts/objects, or an `int` (counted) |
| Artifacts | `artifacts` โ `files` โ `outputs` | list of strings, list of dicts (`path`/`name`/`file`/`filename`/`artifact`), or a dict (values used) |
| Tool schema | `tool_schema` | any JSON-serialisable dict |
| Judge score | `judge_score` | float (malformed values fail the judge gate) |
***
## Common Patterns
Aggregate CSV harness scenarios into an `EvalReport` with `harness_row_to_eval_case()`:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
import csv
from praisonaiagents.eval import harness_row_to_eval_case
with open("scenarios.csv") as f:
cases = [harness_row_to_eval_case(row) for row in csv.DictReader(f)]
```
Enforce tool-schema parity between a native run and a plugin run:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import HarnessEvaluator
evaluator = HarnessEvaluator(
trace=plugin_run_trace,
expected_schema_hash=native_schema_hash, # from the native baseline run
name="parity_check",
)
result = evaluator.run(print_summary=True)
assert result.schema_consistent
```
Gate CI on artifact completeness for a suite of scenarios:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import HarnessEvaluator, EvalSuite
evaluators = [
HarnessEvaluator(trace=t, required_artifacts=["out.txt"], name=t["id"])
for t in scenario_traces
]
report = EvalSuite(evaluators=evaluators, name="ci_gate").run(print_summary=True)
```
***
## Best Practices
A non-numeric `judge_score` fails the judge gate but does not raise, so one broken scenario won't abort a mixed CI run. A trace with no `judge_score` treats the judge as "not configured" โ the judge gate simply passes and does not count against you.
Tool calls are read from `tool_calls`, then `tool_trace`, then `tools`, and artifacts from `artifacts`, then `files`, then `outputs`. Whatever key your harness emits, the evaluator finds it โ no reshaping step before scoring.
Set `min_tool_calls` when a scenario must actually exercise tools (a smoke test that made zero calls is suspect). Set `expected_schema_hash` when you need the tool surface to stay identical between two runs โ for example native vs. plugin-provided tools.
Pass `save_results_path` to write the result JSON, or call `result.to_json()` yourself, so failed CI runs keep the per-gate breakdown for debugging.
***
## Related
Run all four evaluators as one CI gate
Score multi-agent handoff fidelity
Iterative agent โ judge โ improve loop
LLM-as-judge for evaluating outputs
Evaluators, suites, and reports
The Context / Harness / Loop rubric this evaluator scores against.