Context Evaluator

Score multi-agent handoff fidelity and token-budget compliance without an LLM

Context Evaluator scores how much of one agent’s output reaches the next agent — plus whether each agent stayed inside its token budget — all without a live LLM call.

Quick Start

1. Score a 2-agent workflow

   from praisonaiagents import Agent, Task, PraisonAIAgents
   from praisonaiagents.eval import ContextEvaluator
   
   researcher = Agent(name="researcher", instructions="Research the topic")
   writer     = Agent(name="writer",     instructions="Write the final report")
   
   research = Task(description="Research AI agents", agent=researcher)
   report   = Task(description="Write the report", agent=writer)
   
   workflow = PraisonAIAgents(agents=[researcher, writer], tasks=[research, report])
   workflow.start()
   
   result = ContextEvaluator(
       trace_events=workflow.trace_events,
       agent_order=["researcher", "writer"],
   ).run(print_summary=True)
   
   print(result.overall_score)   # 0-10

2. Add a token-budget ledger

   from praisonaiagents.eval import ContextEvaluator
   
   result = ContextEvaluator(
       trace_events=workflow.trace_events,
       agent_order=["researcher", "writer"],
       budget_ledger=[
           {"agent_name": "researcher", "used_tokens": 8_200, "budget_tokens": 10_000},
           {"agent_name": "writer",     "used_tokens": 4_100, "budget_tokens": 6_000},
       ],
   ).run(print_summary=True)
   
   print(result.handoff_score)   # context flow (0-10)
   print(result.budget_score)    # budget compliance (0-10)

3. Run inside an EvalSuite

   from praisonaiagents.eval import ContextEvaluator, EvalSuite, AccuracyEvaluator
   
   suite = EvalSuite(evaluators=[
       ContextEvaluator(
           trace_events=workflow.trace_events,
           agent_order=["researcher", "writer"],
       ),
       AccuracyEvaluator(agent=writer, input_text="Summarize AI agents", expected_output="..."),
   ])
   report = suite.run()

How It Works

Context Evaluator reads the workflow trace and measures how much of each agent’s response tokens survive into the next agent’s request.

When to choose which dimension

Pick the dimensions that match the data you have.

Configuration Options

Full parameter narrative lives in the SDK source.

Methods

ContextEvalResult properties: handoff_score, budget_score, overall_score (average of dimensions present), content_loss_detected, plus to_dict().

Common Patterns

Gate a workflow in CI, catch handoff regressions, or combine with the Harness Evaluator.

import sys
from praisonaiagents.eval import ContextEvaluator

result = ContextEvaluator(
    trace_events=workflow.trace_events,
    agent_order=["researcher", "writer"],
).run()

if result.overall_score < 8:
    sys.exit(1)

from praisonaiagents.eval import ContextEvaluator, EvalSuite, AccuracyEvaluator

suite = EvalSuite(evaluators=[
    ContextEvaluator(
        trace_events=workflow.trace_events,
        agent_order=["researcher", "writer"],
        budget_ledger=[
            {"agent_name": "researcher", "used_tokens": 8_200, "budget_tokens": 10_000},
            {"agent_name": "writer",     "used_tokens": 4_100, "budget_tokens": 6_000},
        ],
    ),
    AccuracyEvaluator(agent=writer, input_text="...", expected_output="..."),
])
report = suite.run()

Best Practices

Related