Evaluation Loop - PraisonAI

EvaluationLoop implements the “Ralph Loop” pattern: run an agent, judge the output, provide feedback, and repeat until the quality threshold is met.

How It Works

Quick Start

Agent.run_until()
EvaluationLoop Class
With Callback
Goal-gated variant

from praisonaiagents import Agent

agent = Agent(name="analyzer", instructions="Analyze systems thoroughly")

result = agent.run_until(
    "Analyze the authentication flow",
    criteria="Analysis is thorough and actionable",
    threshold=8.0,
)

print(f"Score: {result.final_score}/10")
print(f"Success: {result.success}")

from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

agent = Agent(name="writer", instructions="Write compelling content")

loop = EvaluationLoop(
    agent=agent,
    criteria="Content is engaging and well-structured",
    threshold=8.0,
    max_iterations=5,
)

result = loop.run("Write a product description for an AI assistant")
print(result.final_report)

from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

def on_iteration(iteration_result):
    print(f"Iteration {iteration_result.iteration}: {iteration_result.score}/10")

agent = Agent(name="coder", instructions="Write clean code")

result = agent.run_until(
    "Write a function to validate email addresses",
    criteria="Code is correct, readable, and handles edge cases",
    on_iteration=on_iteration,
)

Passing goal= delegates to the tool-using goal loop — an independent judge gates completion against acceptance criteria.

from praisonaiagents import Agent
from praisonaiagents.goal import GoalCriteria

agent = Agent(name="ops", instructions="You are an ops assistant", autonomy=True)

result = agent.run_until(
    "Do the task",
    goal="A PR is open with green CI",
    goal_criteria=GoalCriteria(outcome="PR merged", verification="PR URL exists"),
    max_iterations=15,          # forwarded as max_turns
    judge_model="gpt-4o-mini",  # optional independent judge
)
# NOTE: this returns AutonomyResult, not EvaluationLoopResult
print(result.completion_reason)  # "goal_met" | "budget_paused"

When goal is passed, run_until() delegates to run_goal() and returns an AutonomyResult (success / output / completion_reason) instead of the usual EvaluationLoopResult. Adjust downstream code that accesses result.final_score / result.iterations.

If a bare string criteria is passed alongside goal, it is promoted to GoalCriteria(outcome=criteria) so it is not silently dropped.

Modes

Optimize Mode

Stops as soon as the threshold is met. Best for production use.

loop = EvaluationLoop(
    agent=agent,
    criteria="...",
    mode="optimize"  # default
)

Review Mode

Runs all iterations regardless of score. Best for analysis.

loop = EvaluationLoop(
    agent=agent,
    criteria="...",
    mode="review"
)

Configuration

agent

Agent

required

The Agent instance to evaluate

criteria

string

required

Evaluation criteria for the Judge (e.g., “Response is thorough and accurate”)

threshold

float

default:"8.0"

Score threshold for success (1-10 scale)

max_iterations

int

default:"5"

Maximum number of iterations before stopping

mode

string

default:"optimize"

"optimize" (stop on success) or "review" (run all iterations)

on_iteration

Callable

Optional callback called after each iteration with IterationResult

verbose

bool

default:"false"

Enable verbose logging

metric

Callable[[str], float]

Numeric metric on (output). When set, the loop scores with this callable instead of the LLM Judge. Non-finite results (NaN / inf) are floored to 0.0.

Keep the Best Iteration

The result exposes the highest-scoring iteration, not just the last one.

result = agent.run_until("Draft a launch email", criteria="clear and persuasive")

print(result.best_score)    # highest score across all iterations
print(result.best_output)   # output of the best iteration
print(result.final_output)  # output of the last iteration (may differ)

success is now best.score >= threshold, not iterations[-1].score >= threshold. A run that improves and then regresses reports the improvement instead of the regression.

best

IterationResult

The highest-scoring iteration

best_iteration

IterationResult

The highest-scoring iteration (falls back to max() over iterations if best is unset)

best_score

float

Score of the best iteration

best_output

string

Output of the best iteration

Score with a Numeric Metric

Pass a metric to score iterations with an empirical function instead of the LLM Judge.

from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

def word_overlap(output: str, expected: str) -> float:
    a, b = set(output.lower().split()), set(expected.lower().split())
    return len(a & b) / max(len(b), 1)

agent = Agent(name="answerer", instructions="Answer the question.")

loop = EvaluationLoop(
    agent=agent,
    criteria="",
    threshold=0.8,
    max_iterations=3,
    mode="review",
    metric=lambda output: word_overlap(output, "gold answer"),
)

result = loop.run("Answer the question.")
print(result.best_score, result.best_output)

The Judge remains the default — the metric only takes over when set. This works with any empirical score (rouge_l, accuracy, latency, or a custom function).

Non-finite metric results (NaN / inf) are floored to 0.0 so they never win selection.

Results

EvaluationLoopResult

success

bool

Whether the loop achieved the threshold

final_score

float

Score from the last iteration (1-10)

score_history

list[float]

All scores across iterations

final_output

string

Output from the last iteration

accumulated_findings

list[string]

All findings/suggestions collected

num_iterations

int

Number of iterations completed

total_duration_seconds

float

Total time taken

result = agent.run_until("Analyze the codebase", criteria="...")

# Access results
print(result.success)              # True
print(result.final_score)          # 8.5
print(result.score_history)        # [6.0, 7.2, 8.5]
print(result.num_iterations)       # 3
print(result.accumulated_findings) # ["Consider edge cases", ...]

# Generate report
print(result.final_report)         # Markdown report

# Serialize
print(result.to_json())            # JSON string
print(result.to_dict())            # Dictionary

IterationResult

Each iteration produces an IterationResult:

for iteration in result.iterations:
    print(f"Iteration {iteration.iteration}")
    print(f"  Score: {iteration.score}/10")
    print(f"  Reasoning: {iteration.reasoning}")
    print(f"  Findings: {iteration.findings}")
    print(f"  Output: {iteration.output[:100]}...")

Async Support

import asyncio
from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

async def main():
    agent = Agent(name="analyzer", instructions="Analyze systems")
    
    # Using EvaluationLoop directly
    loop = EvaluationLoop(agent=agent, criteria="Analysis is thorough")
    result = await loop.run_async("Analyze the auth flow")
    
    # Or using Agent method
    result = await agent.run_until_async(
        "Analyze the auth flow",
        criteria="Analysis is thorough",
    )
    
    print(result.final_score)

asyncio.run(main())

Best Practices

Write Specific Criteria

Be specific in your criteria to get consistent results:

# ❌ Vague
criteria="Response is good"

# ✅ Specific
criteria="Response includes: 1) Clear problem statement, 2) Step-by-step solution, 3) Code examples"

Set Appropriate Thresholds

8.0 (default): Good for most use cases
9.0+: High quality, may require more iterations
7.0: Acceptable quality, faster completion

Limit Iterations for Cost Control

Each iteration makes LLM calls. Set max_iterations based on your budget:

loop = EvaluationLoop(
    agent=agent,
    criteria="...",
    max_iterations=3,  # Limit for cost control
)

Use Callbacks for Monitoring

Track progress in real-time:

def on_iteration(r):
    print(f"[{r.iteration}] Score: {r.score} - {r.reasoning[:50]}...")
    if r.score < 6:
        print("  ⚠️ Low score, may need more iterations")

result = agent.run_until("...", criteria="...", on_iteration=on_iteration)

Judge

LLM-as-judge for evaluating outputs

Evaluator-Optimizer

Multi-agent evaluator-optimizer pattern

Goal Loop

Gate a tool-using loop on an independent acceptance-criteria judge

Prompt Optimizer

Optimise an agent’s instructions against an eval set

EvaluationLoop uses lazy loading - the Judge is only imported when you actually run an evaluation, ensuring zero performance impact when not in use.

​How It Works

​Quick Start

​Modes

Optimize Mode

Review Mode

​Configuration

​Keep the Best Iteration

​Score with a Numeric Metric

​Results

​EvaluationLoopResult

​IterationResult

​Async Support

​Best Practices

​Related

Judge

Evaluator-Optimizer

Goal Loop

Prompt Optimizer

How It Works

Quick Start

Modes

Configuration

Keep the Best Iteration

Score with a Numeric Metric

Results

EvaluationLoopResult

IterationResult

Async Support

Best Practices

Related