EvalPort Adapter

Export and import eval suites in the EvalPort open spec

The EvalPort adapter converts native eval suites and reports to and from the EvalPort open spec (version 1.0.0-rc.5) so your agents interoperate with any EvalPort-compatible tool.

Run an Agent through a suite, then export both the suite and results as spec-conformant EvalPort documents:

python

import json
from praisonaiagents import Agent
from praisonaiagents.eval import (
    EvalCase, EvalPackage, EvalSuite, HarnessEvaluator,
    to_evalport, report_to_evalport,
)

agent = Agent(name="Support", instructions="Answer support questions.")

package = EvalPackage(
    name="support_eval",
    cases=[EvalCase(name="refund", input="refund policy?", expected="30 days")],
)

evaluators = [HarnessEvaluator(trace={"tool_calls": 0}, name=c.name) for c in package.cases]
report = EvalSuite(evaluators=evaluators, name=package.name).run()

with open("suite.json", "w") as f:
    json.dump(to_evalport(package), f, indent=2)
with open("results.json", "w") as f:
    json.dump(report_to_evalport(report, run_id="ci-1"), f, indent=2)

An Agent produces an EvalReport; one line exports it as an EvalPort ResultSet:

python

from praisonaiagents.eval import report_to_evalport

result_set = report_to_evalport(report)   # report: EvalReport from a suite run

Diagram

Quick Start

Export a suite

Build an EvalPackage and export it as an EvalPort suite dict:

python

import json
from praisonaiagents.eval import EvalCase, EvalPackage, to_evalport

package = EvalPackage(
    name="math_eval",
    description="Basic math evaluation",
    cases=[EvalCase(name="addition", input="What is 2 + 2?", expected="4")],
    thresholds={"accuracy": 0.9},
)

suite = to_evalport(package)
print(json.dumps(suite, indent=2))

Import a suite from another tool

Load an EvalPort suite dict from any compatible tool, then run it. from_evalport prefers spec-shaped test_cases but also accepts legacy cases:

python

from praisonaiagents.eval import from_evalport, EvalSuite

suite_dict = {
    "id": "external_suite",
    "test_cases": [{"id": "greeting", "input": "Say hello", "graders": ["praisonai_verdict"]}],
}

package = from_evalport(suite_dict)
print(package.name, len(package))

Export a run report

Run a suite to get an EvalReport, then persist it as an EvalPort ResultSet:

python

import json
from praisonaiagents.eval import report_to_evalport

result_set = report_to_evalport(report)   # report: EvalReport from a suite run
with open("results.json", "w") as f:
    json.dump(result_set, f, indent=2)

Pin run_id / started_at for CI

Pass explicit run_id and started_at so CI runs produce stable, diffable output:

python

result_set = report_to_evalport(
    report,
    run_id="ci-build-1234",
    started_at="2026-09-30T00:00:00+00:00",
)

How It Works

The adapter maps native dataclasses to plain EvalPort dicts and back, so PraisonAI and external harnesses trade suites over the shared spec.

Diagram

Three module-level functions cover every direction — pick one by what you hold:

Diagram

Function	Direction	Returns
`to_evalport(package)`	`EvalPackage` → suite	`Dict[str, Any]`
`from_evalport(suite)`	suite → `EvalPackage`	`EvalPackage`
`report_to_evalport(report, *, run_id=None, started_at=None)`	`EvalReport` → result set	`Dict[str, Any]`

A round trip is lossless — from_evalport(to_evalport(pkg)) reproduces the original package’s to_dict().

Spec Mapping

Native models map 1:1 onto EvalPort concepts.

PraisonAI	EvalPort (spec `1.0.0-rc.5`)	Notes
`EvalPackage.name`	`suite.id` and `suite.name`	required
`EvalPackage.description`	`suite.description`	omitted when empty
`EvalPackage.version`	`suite.metadata.praisonai.version`	package version, not the spec version
`EvalPackage.cases[*]`	`suite.test_cases[*]`	renamed from `cases`
`EvalPackage.thresholds`	`suite.metadata.praisonai.thresholds`	under reserved namespace
`EvalPackage.seed`	`suite.metadata.praisonai.seed`	under reserved namespace
(implicit)	`suite.graders`	verdict grader + one per unique criterion; required by spec
`EvalCase.name`	`test_case.id`	fallback on import: `test_case.name` → `"case"`
`EvalCase.input`	`test_case.input`	required
`EvalCase.expected`	`test_case.expected_output`	fallback: `expected`; omitted when `None`
`EvalCase.criteria`	`test_case.graders` = `["praisonai_verdict", ...criteria]`	always references the verdict grader first, deduplicated
`EvalCase.timeout_seconds` (positive)	`test_case.timeout_ms`	integer ms, `max(1, round(sec*1000))`; spec requires `>= 1`
`EvalCase.timeout_seconds` (nonpositive)	`test_case.metadata.praisonai.timeout_seconds`	preserves round-trip fidelity
`EvalCase.metadata`	`test_case.metadata`	dict copy, never mutated on import
`EvalReport.package_name`	`result_set.suite_id`	renamed
—	`result_set.run_id`	auto UUID (`uuid.uuid4().hex`)
—	`result_set.started_at`	auto RFC 3339 UTC
—	`result_set.runner`	`{"name": "praisonaiagents", "version": <pkg version if importable>}`
`EvalReport` totals	`result_set.summary.*`	`total`, `passed`, `failed`, `pass_rate`, `avg_score` (renamed, clamped to `[0,1]`)
`EvalReport.thresholds_met`	`result_set.metadata.praisonai.thresholds_met`	under reserved namespace
`EvalResult.case_name`	`result.test_case_id`	renamed
`EvalResult.score`	`result.grader_results[0].score`	verdict grader; clamped to `[0,1]`, raw kept in `grader_result.metadata`
`EvalResult.criteria_scores`	`result.grader_results[1:]`	one `grader_result` per criterion
`EvalResult.latency_ms`	`result.duration_ms`	integer; omitted when `None` or negative
`EvalResult.error`	`result.error = {"type": "runner_error", "message": str}`	structured per spec
`EvalResult.actual_output`	`result.actual_output`	coerced to string
`EvalResult.record`	`result.metadata.praisonai.record`	under reserved namespace

Each grader_result carries a clamped [0,1] score:

json

{"grader_id": "praisonai_verdict", "type": "custom", "score": 0.95, "passed": true}

A suite dict from to_evalport looks like this:

json

{
  "version": "1.0.0-rc.5",
  "id": "math_eval",
  "name": "math_eval",
  "graders": [
    {
      "id": "praisonai_verdict",
      "type": "custom",
      "params": {"handler": "praisonaiagents.eval"},
      "description": "PraisonAI per-case verdict (EvalResult.passed/score)"
    }
  ],
  "test_cases": [
    {
      "id": "addition",
      "input": "What is 2 + 2?",
      "graders": ["praisonai_verdict"],
      "expected_output": "4",
      "timeout_ms": 30000
    }
  ],
  "metadata": {
    "praisonai": {
      "version": "1.0.0",
      "thresholds": {"accuracy": 0.9},
      "seed": null
    }
  },
  "description": "Basic math evaluation"
}

A result set from report_to_evalport looks like this:

json

{
  "version": "1.0.0-rc.5",
  "suite_id": "math_eval",
  "run_id": "7f0c8c1a5d3f4a2b9e5c8d6a4b2f1e0c",
  "started_at": "2026-09-30T12:00:00+00:00",
  "runner": {"name": "praisonaiagents", "version": "0.x.y"},
  "results": [
    {
      "test_case_id": "addition",
      "passed": true,
      "grader_results": [
        {"grader_id": "praisonai_verdict", "type": "custom", "score": 1.0, "passed": true}
      ],
      "actual_output": "4",
      "duration_ms": 12
    }
  ],
  "summary": {
    "total": 1,
    "passed": 1,
    "failed": 0,
    "pass_rate": 1.0,
    "avg_score": 1.0
  },
  "metadata": {
    "praisonai": {"thresholds_met": {"accuracy": true}}
  }
}

Configuration Options

The adapter is dependency-free and exposes three functions — no config classes.

Card

Full Python API for the eval package, including to_evalport, from_evalport, and report_to_evalport

Common Patterns

Round-trip a suite locally to prove fidelity:

python

from praisonaiagents.eval import EvalCase, EvalPackage, to_evalport, from_evalport

package = EvalPackage(name="demo", cases=[EvalCase(name="c1", input="ping")])
restored = from_evalport(to_evalport(package))
assert restored.to_dict() == package.to_dict()

Import a third-party EvalPort suite and run it through HarnessEvaluator:

python

from praisonaiagents.eval import from_evalport, HarnessEvaluator, EvalSuite

package = from_evalport(external_suite_dict)   # from a Benchmark Hub / other harness
evaluators = [
    HarnessEvaluator(trace={"tool_calls": 1, "artifacts": ["out.txt"]}, name=case.name)
    for case in package.cases
]
report = EvalSuite(evaluators=evaluators, name=package.name).run(print_summary=True)

Prove structural conformance for both documents with the external openeval package:

python

from praisonaiagents.eval import to_evalport, report_to_evalport
from openeval.validate import validate_suite, validate_result_set   # opt-in, install separately

suite = to_evalport(package)
result_set = report_to_evalport(report, run_id="ci-build-1234")

assert validate_suite(suite).valid
assert validate_result_set(result_set).valid

Best Practices

= 1)”> to_evalport writes a positive timeout_seconds as timeout_ms (integer milliseconds, max(1, round(sec*1000))) because the spec requires timeout_ms >= 1. A nonpositive timeout_seconds cannot be represented, so it is preserved under test_case.metadata.praisonai.timeout_seconds — the importer restores the original value, keeping round trips lossless.

Every test case references praisonai_verdict first

The exported graders list on each test case always starts with "praisonai_verdict" (the spec requires at least one grader per case) followed by any criterion names, deduplicated in order. On import the adapter strips "praisonai_verdict" before returning criteria, so a round trip reproduces the original criteria exactly.

Pin run_id / started_at for deterministic CI

report_to_evalport accepts keyword-only run_id and started_at. Left unset, run_id defaults to a random UUID and started_at to the current UTC time — pass explicit values in CI so output stays stable and diffable.

python

result_set = report_to_evalport(
    report,
    run_id="ci-build-1234",
    started_at="2026-09-30T00:00:00+00:00",
)

Scores are clamped to [0, 1]

Any raw score outside [0, 1] is clamped; the pre-clamp value survives under grader_result.metadata["openeval.raw_score"]. When clamping changes the summary average, the raw value is kept under metadata.praisonai.average_score. A non-numeric or NaN score becomes null and forces passed: false (“not verified”).

Keep the core SDK dependency-free

The adapter never imports evalport-sdk or openeval. Install validation packages only where you run validate_suite() / validate_result_set(), so the core SDK stays lightweight everywhere else.

Evaluation Suite

Run every evaluator as one CI gate

Harness Evaluator

Score Interactive Test Harness traces

Judge

LLM-as-judge for evaluating outputs

Evaluation

Evaluators, suites, and reports

EvalPort Adapter

Quick Start

How It Works

Spec Mapping

Configuration Options

Common Patterns

Best Practices

Related