eval • AI Agent SDK

PraisonAI Agents Evaluation Framework.

eval

PraisonAI Agents Evaluation Framework.

Provides comprehensive evaluation capabilities for AI agents with zero performance impact when not in use through lazy loading.

Evaluator Types: – AccuracyEvaluator: Compare output against expected output using LLM-as-judge – PerformanceEvaluator: Measure runtime and memory usage – ReliabilityEvaluator: Verify expected tool calls are made – CriteriaEvaluator: Evaluate against custom criteria – ContextEvaluator: Score context budget compliance and multi-agent handoff fidelity – ComparisonEval: Side-by-side comparison of two agent outputs – SafetyEval: Detect harmful, biased, or inappropriate outputs – LoopEvaluator: Score loop health (convergence, wasted iterations, doom-loop guards) – EvalSuite: Orchestrator for running multiple evaluations

Example:

>>> from praisonaiagents.eval import AccuracyEvaluator
>>> evaluator = AccuracyEvaluator(
...     agent=my_agent,
...     input_text="What is 2+2?",
...     expected_output="4"
... )
>>> result = evaluator.run(print_summary=True)

Import

from praisonaiagents import eval

Constants

Related Documentation