Skip to main content
Context Evaluator scores how much of one agent’s output reaches the next agent — plus whether each agent stayed inside its token budget — all without a live LLM call.

​
Quick Start

1

Score a 2-agent workflow

from praisonaiagents import Agent, Task, PraisonAIAgents
from praisonaiagents.eval import ContextEvaluator

researcher = Agent(name="researcher", instructions="Research the topic")
writer     = Agent(name="writer",     instructions="Write the final report")

research = Task(description="Research AI agents", agent=researcher)
report   = Task(description="Write the report", agent=writer)

workflow = PraisonAIAgents(agents=[researcher, writer], tasks=[research, report])
workflow.start()

result = ContextEvaluator(
    trace_events=workflow.trace_events,
    agent_order=["researcher", "writer"],
).run(print_summary=True)

print(result.overall_score)   # 0-10
2

Add a token-budget ledger

from praisonaiagents.eval import ContextEvaluator

result = ContextEvaluator(
    trace_events=workflow.trace_events,
    agent_order=["researcher", "writer"],
    budget_ledger=[
        {"agent_name": "researcher", "used_tokens": 8_200, "budget_tokens": 10_000},
        {"agent_name": "writer",     "used_tokens": 4_100, "budget_tokens": 6_000},
    ],
).run(print_summary=True)

print(result.handoff_score)   # context flow (0-10)
print(result.budget_score)    # budget compliance (0-10)
3

Run inside an EvalSuite

from praisonaiagents.eval import ContextEvaluator, EvalSuite, AccuracyEvaluator

suite = EvalSuite(evaluators=[
    ContextEvaluator(
        trace_events=workflow.trace_events,
        agent_order=["researcher", "writer"],
    ),
    AccuracyEvaluator(agent=writer, input_text="Summarize AI agents", expected_output="..."),
])
report = suite.run()

​
How It Works

Context Evaluator reads the workflow trace and measures how much of each agent’s response tokens survive into the next agent’s request.
SignalSource eventScore
Exact content matchllm_response → llm_request10/10
Partial overlaptoken intersectionmin(10, overlap × 12 + 2)
Below 5low overlapflags content_loss_detected=True
Budget metused <= budget10/10
Over budgetoverrunmax(1, 10 - overrun × 10)
No budget (0)neutral5.0

​
When to choose which dimension

Pick the dimensions that match the data you have.

​
Configuration Options

Full parameter narrative lives in the SDK source.
OptionTypeDefaultDescription
trace_eventslist[Any]NoneTrace events, each with event_type / agent_name / data (objects or dicts)
agent_orderlist[str]NoneOrdered agent names for the handoff chain (needs ≥ 2 for handoff scoring)
budget_ledgerlist[dict]NoneList of {"agent_name", "used_tokens", "budget_tokens"} dicts
namestrNoneEvaluation name
save_results_pathstrNonePath to persist result JSON
verboseboolFalseEnable verbose logging
Methods
MethodReturnsPurpose
evaluate_handoff()list[ContextHandoffResult]Token overlap for each adjacent agent pair
evaluate_budget()list[BudgetComplianceResult]Budget adherence per ledger entry
run(print_summary=False)ContextEvalResultRuns both and aggregates
ContextEvalResult properties: handoff_score, budget_score, overall_score (average of dimensions present), content_loss_detected, plus to_dict().

​
Common Patterns

Gate a workflow in CI, catch handoff regressions, or combine with the Harness Evaluator. 
import sys
from praisonaiagents.eval import ContextEvaluator

result = ContextEvaluator(
    trace_events=workflow.trace_events,
    agent_order=["researcher", "writer"],
).run()

if result.overall_score < 8:
    sys.exit(1)

​
Best Practices

Handoff scoring needs both llm_request and llm_response events for each agent. Pass the complete workflow.trace_events, not a single summary string.
Scoring walks adjacent pairs in agent_order. List the agents in the exact sequence they run so each handoff is compared correctly.
Entries with budget_tokens of 0 score a neutral 5.0. Set a real budget only where you want a gate.
No live model call runs, so scores are reproducible and require no API key.

Evaluation Suite

Run all four evaluators as one CI gate

Evaluation Loop

Iterative agent → judge → improve loop

Judge

LLM-as-judge for evaluating outputs

Harness Evaluator

Score Interactive Test Harness traces

CHL Engineering

The Context / Harness / Loop rubric this evaluator scores against.