Evaluation Loop

Iterative improvement loop that runs agent → judges → improves until quality threshold met

How It Works

Diagram

Quick Start

Agent.run_until()

python

from praisonaiagents import Agent

agent = Agent(name="analyzer", instructions="Analyze systems thoroughly")

result = agent.run_until(
    "Analyze the authentication flow",
    criteria="Analysis is thorough and actionable",
    threshold=8.0,
)

print(f"Score: {result.final_score}/10")
print(f"Success: {result.success}")

EvaluationLoop Class

python

from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

agent = Agent(name="writer", instructions="Write compelling content")

loop = EvaluationLoop(
    agent=agent,
    criteria="Content is engaging and well-structured",
    threshold=8.0,
    max_iterations=5,
)

result = loop.run("Write a product description for an AI assistant")
print(result.final_report)

With Callback

python

from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

def on_iteration(iteration_result):
    print(f"Iteration {iteration_result.iteration}: {iteration_result.score}/10")

agent = Agent(name="coder", instructions="Write clean code")

result = agent.run_until(
    "Write a function to validate email addresses",
    criteria="Code is correct, readable, and handles edge cases",
    on_iteration=on_iteration,
)

Goal-gated variant

Passing goal= delegates to the tool-using goal loop — an independent judge gates completion against acceptance criteria.

python

from praisonaiagents import Agent
from praisonaiagents.goal import GoalCriteria

agent = Agent(name="ops", instructions="You are an ops assistant", autonomy=True)

result = agent.run_until(
    "Do the task",
    goal="A PR is open with green CI",
    goal_criteria=GoalCriteria(outcome="PR merged", verification="PR URL exists"),
    max_iterations=15,          # forwarded as max_turns
    judge_model="gpt-4o-mini",  # optional independent judge
)
# NOTE: this returns AutonomyResult, not EvaluationLoopResult
print(result.completion_reason)  # "goal_met" | "budget_paused"

If a bare string criteria is passed alongside goal, it is promoted to GoalCriteria(outcome=criteria) so it is not silently dropped.

Modes

Diagram

Optimize Mode

Stops as soon as the threshold is met. Best for production use.

python

loop = EvaluationLoop(
    agent=agent,
    criteria="...",
    mode="optimize"  # default
)

Review Mode

Runs all iterations regardless of score. Best for analysis.

python

loop = EvaluationLoop(
    agent=agent,
    criteria="...",
    mode="review"
)

Configuration

agent Agent

The Agent instance to evaluate

criteria string

Evaluation criteria for the Judge (e.g., “Response is thorough and accurate”)

threshold float

Score threshold for success (1-10 scale)

max_iterations int

Maximum number of iterations before stopping

mode string

"optimize" (stop on success) or "review" (run all iterations)

on_iteration Callable

Optional callback called after each iteration with IterationResult

verbose bool

Enable verbose logging

metric Callable[[str], float]

Numeric metric on (output). When set, the loop scores with this callable instead of the LLM Judge. Non-finite results (NaN / inf) are floored to 0.0.

Keep the Best Iteration

The result exposes the highest-scoring iteration, not just the last one.

python

result = agent.run_until("Draft a launch email", criteria="clear and persuasive")

print(result.best_score)    # highest score across all iterations
print(result.best_output)   # output of the best iteration
print(result.final_output)  # output of the last iteration (may differ)

success is now best.score >= threshold, not iterations[-1].score >= threshold. A run that improves and then regresses reports the improvement instead of the regression.

ResponseField IterationResult

The highest-scoring iteration

ResponseField IterationResult

The highest-scoring iteration (falls back to max() over iterations if best is unset)

ResponseField float

Score of the best iteration

ResponseField string

Output of the best iteration

Score with a Numeric Metric

Pass a metric to score iterations with an empirical function instead of the LLM Judge.

python

from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

def word_overlap(output: str, expected: str) -> float:
    a, b = set(output.lower().split()), set(expected.lower().split())
    return len(a & b) / max(len(b), 1)

agent = Agent(name="answerer", instructions="Answer the question.")

loop = EvaluationLoop(
    agent=agent,
    criteria="",
    threshold=0.8,
    max_iterations=3,
    mode="review",
    metric=lambda output: word_overlap(output, "gold answer"),
)

result = loop.run("Answer the question.")
print(result.best_score, result.best_output)

The Judge remains the default — the metric only takes over when set. This works with any empirical score (rouge_l, accuracy, latency, or a custom function).

Results

EvaluationLoopResult

ResponseField bool

Whether the loop achieved the threshold

ResponseField float

Score from the last iteration (1-10)

ResponseField list[float]

All scores across iterations

ResponseField string

Output from the last iteration

ResponseField list[string]

All findings/suggestions collected

ResponseField int

Number of iterations completed

ResponseField float

Total time taken

python

result = agent.run_until("Analyze the codebase", criteria="...")

# Access results
print(result.success)              # True
print(result.final_score)          # 8.5
print(result.score_history)        # [6.0, 7.2, 8.5]
print(result.num_iterations)       # 3
print(result.accumulated_findings) # ["Consider edge cases", ...]

# Generate report
print(result.final_report)         # Markdown report

# Serialize
print(result.to_json())            # JSON string
print(result.to_dict())            # Dictionary

IterationResult

Each iteration produces an IterationResult:

python

for iteration in result.iterations:
    print(f"Iteration {iteration.iteration}")
    print(f"  Score: {iteration.score}/10")
    print(f"  Reasoning: {iteration.reasoning}")
    print(f"  Findings: {iteration.findings}")
    print(f"  Output: {iteration.output[:100]}...")

Async Support

python

import asyncio
from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

async def main():
    agent = Agent(name="analyzer", instructions="Analyze systems")

    # Using EvaluationLoop directly
    loop = EvaluationLoop(agent=agent, criteria="Analysis is thorough")
    result = await loop.run_async("Analyze the auth flow")

    # Or using Agent method
    result = await agent.run_until_async(
        "Analyze the auth flow",
        criteria="Analysis is thorough",
    )

    print(result.final_score)

asyncio.run(main())

Best Practices

Write Specific Criteria

Be specific in your criteria to get consistent results:

python

# ❌ Vague
criteria="Response is good"

# ✅ Specific
criteria="Response includes: 1) Clear problem statement, 2) Step-by-step solution, 3) Code examples"

Set Appropriate Thresholds

8.0 (default): Good for most use cases
9.0+: High quality, may require more iterations
7.0: Acceptable quality, faster completion

Limit Iterations for Cost Control

Each iteration makes LLM calls. Set max_iterations based on your budget:

python

loop = EvaluationLoop(
    agent=agent,
    criteria="...",
    max_iterations=3,  # Limit for cost control
)

Use Callbacks for Monitoring

Track progress in real-time:

python

def on_iteration(r):
    print(f"[{r.iteration}] Score: {r.score} - {r.reasoning[:50]}...")
    if r.score < 6:
        print("  ⚠️ Low score, may need more iterations")

result = agent.run_until("...", criteria="...", on_iteration=on_iteration)

Judge

LLM-as-judge for evaluating outputs

Evaluator-Optimizer

Multi-agent evaluator-optimizer pattern

Goal Loop

Gate a tool-using loop on an independent acceptance-criteria judge

Prompt Optimizer

Optimise an agent’s instructions against an eval set

Trials Engine

Run K isolated attempts per eval case for a pass rate and frontier

Last-Trace Eval

Grade the last completed traced run from the CLI with one command.

Evaluation Loop

How It Works

Quick Start

Agent.run_until()

EvaluationLoop Class

With Callback

Goal-gated variant

Modes

Configuration

Keep the Best Iteration

Score with a Numeric Metric

Results

EvaluationLoopResult

IterationResult

Async Support

Best Practices

Related