Evaluation Loop

EvaluationLoop implements the “Ralph Loop” pattern: run an agent, judge the output, provide feedback, and repeat until the quality threshold is met.

How It Works

Quick Start

Agent.run_until()
EvaluationLoop Class
With Callback
Goal-gated variant

from praisonaiagents import Agent

agent = Agent(name="analyzer", instructions="Analyze systems thoroughly")

result = agent.run_until(
    "Analyze the authentication flow",
    criteria="Analysis is thorough and actionable",
    threshold=8.0,
)

print(f"Score: {result.final_score}/10")
print(f"Success: {result.success}")

from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

agent = Agent(name="writer", instructions="Write compelling content")

loop = EvaluationLoop(
    agent=agent,
    criteria="Content is engaging and well-structured",
    threshold=8.0,
    max_iterations=5,
)

result = loop.run("Write a product description for an AI assistant")
print(result.final_report)

from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

def on_iteration(iteration_result):
    print(f"Iteration {iteration_result.iteration}: {iteration_result.score}/10")

agent = Agent(name="coder", instructions="Write clean code")

result = agent.run_until(
    "Write a function to validate email addresses",
    criteria="Code is correct, readable, and handles edge cases",
    on_iteration=on_iteration,
)

Passing goal= delegates to the tool-using goal loop — an independent judge gates completion against acceptance criteria.

from praisonaiagents import Agent
from praisonaiagents.goal import GoalCriteria

agent = Agent(name="ops", instructions="You are an ops assistant", autonomy=True)

result = agent.run_until(
    "Do the task",
    goal="A PR is open with green CI",
    goal_criteria=GoalCriteria(outcome="PR merged", verification="PR URL exists"),
    max_iterations=15,          # forwarded as max_turns
    judge_model="gpt-4o-mini",  # optional independent judge
)
# NOTE: this returns AutonomyResult, not EvaluationLoopResult
print(result.completion_reason)  # "goal_met" | "budget_paused"

When goal is passed, run_until() delegates to run_goal() and returns an AutonomyResult (success / output / completion_reason) instead of the usual EvaluationLoopResult. Adjust downstream code that accesses result.final_score / result.iterations.

If a bare string criteria is passed alongside goal, it is promoted to GoalCriteria(outcome=criteria) so it is not silently dropped.

Modes

Optimize Mode

Stops as soon as the threshold is met. Best for production use.

loop = EvaluationLoop(
    agent=agent,
    criteria="...",
    mode="optimize"  # default
)

Review Mode

Runs all iterations regardless of score. Best for analysis.

loop = EvaluationLoop(
    agent=agent,
    criteria="...",
    mode="review"
)

Configuration

Agent

required

The Agent instance to evaluate

string

required

Evaluation criteria for the Judge (e.g., “Response is thorough and accurate”)

float

default:"8.0"

Score threshold for success (1-10 scale)

int

default:"5"

Maximum number of iterations before stopping

string

default:"optimize"

"optimize" (stop on success) or "review" (run all iterations)

Callable

Optional callback called after each iteration with IterationResult

bool

default:"false"

Enable verbose logging

Callable[[str], float]

Numeric metric on (output). When set, the loop scores with this callable instead of the LLM Judge. Non-finite results (NaN / inf) are floored to 0.0.

Keep the Best Iteration

The result exposes the highest-scoring iteration, not just the last one.

result = agent.run_until("Draft a launch email", criteria="clear and persuasive")

print(result.best_score)    # highest score across all iterations
print(result.best_output)   # output of the best iteration
print(result.final_output)  # output of the last iteration (may differ)

success is now best.score >= threshold, not iterations[-1].score >= threshold. A run that improves and then regresses reports the improvement instead of the regression.

IterationResult

The highest-scoring iteration

IterationResult

The highest-scoring iteration (falls back to max() over iterations if best is unset)

float

Score of the best iteration

string

Output of the best iteration

Score with a Numeric Metric

Pass a metric to score iterations with an empirical function instead of the LLM Judge.

from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

def word_overlap(output: str, expected: str) -> float:
    a, b = set(output.lower().split()), set(expected.lower().split())
    return len(a & b) / max(len(b), 1)

agent = Agent(name="answerer", instructions="Answer the question.")

loop = EvaluationLoop(
    agent=agent,
    criteria="",
    threshold=0.8,
    max_iterations=3,
    mode="review",
    metric=lambda output: word_overlap(output, "gold answer"),
)

result = loop.run("Answer the question.")
print(result.best_score, result.best_output)

The Judge remains the default — the metric only takes over when set. This works with any empirical score (rouge_l, accuracy, latency, or a custom function).

Non-finite metric results (NaN / inf) are floored to 0.0 so they never win selection.

Results

EvaluationLoopResult

bool

Whether the loop achieved the threshold

float

Score from the last iteration (1-10)

list[float]

All scores across iterations

string

Output from the last iteration

list[string]

All findings/suggestions collected

int

Number of iterations completed

float

Total time taken

result = agent.run_until("Analyze the codebase", criteria="...")

# Access results
print(result.success)              # True
print(result.final_score)          # 8.5
print(result.score_history)        # [6.0, 7.2, 8.5]
print(result.num_iterations)       # 3
print(result.accumulated_findings) # ["Consider edge cases", ...]

# Generate report
print(result.final_report)         # Markdown report

# Serialize
print(result.to_json())            # JSON string
print(result.to_dict())            # Dictionary

IterationResult

Each iteration produces an IterationResult:

for iteration in result.iterations:
    print(f"Iteration {iteration.iteration}")
    print(f"  Score: {iteration.score}/10")
    print(f"  Reasoning: {iteration.reasoning}")
    print(f"  Findings: {iteration.findings}")
    print(f"  Output: {iteration.output[:100]}...")

Async Support

import asyncio
from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop

async def main():
    agent = Agent(name="analyzer", instructions="Analyze systems")
    
    # Using EvaluationLoop directly
    loop = EvaluationLoop(agent=agent, criteria="Analysis is thorough")
    result = await loop.run_async("Analyze the auth flow")
    
    # Or using Agent method
    result = await agent.run_until_async(
        "Analyze the auth flow",
        criteria="Analysis is thorough",
    )
    
    print(result.final_score)

asyncio.run(main())

Best Practices

Write Specific Criteria

Be specific in your criteria to get consistent results:

# ❌ Vague
criteria="Response is good"

# ✅ Specific
criteria="Response includes: 1) Clear problem statement, 2) Step-by-step solution, 3) Code examples"

Set Appropriate Thresholds

8.0 (default): Good for most use cases
9.0+: High quality, may require more iterations
7.0: Acceptable quality, faster completion

Limit Iterations for Cost Control

Each iteration makes LLM calls. Set max_iterations based on your budget:

loop = EvaluationLoop(
    agent=agent,
    criteria="...",
    max_iterations=3,  # Limit for cost control
)

Use Callbacks for Monitoring

Track progress in real-time:

def on_iteration(r):
    print(f"[{r.iteration}] Score: {r.score} - {r.reasoning[:50]}...")
    if r.score < 6:
        print("  ⚠️ Low score, may need more iterations")

result = agent.run_until("...", criteria="...", on_iteration=on_iteration)

Judge

LLM-as-judge for evaluating outputs

Evaluator-Optimizer

Multi-agent evaluator-optimizer pattern

Goal Loop

Gate a tool-using loop on an independent acceptance-criteria judge

Prompt Optimizer

Optimise an agent’s instructions against an eval set

EvaluationLoop uses lazy loading - the Judge is only imported when you actually run an evaluation, ensuring zero performance impact when not in use.

Getting Started

Install

Core Concepts

Guides

Features

Models

Databases

Observability

Memory

Telemetry & Analytics

Knowledge

RAG

Persistence

Tools

Other Features

Developers

Configuration

Best Practices

Getting Started (No Code)

How It Works

Quick Start

Modes

Optimize Mode

Review Mode

Configuration

Keep the Best Iteration

Score with a Numeric Metric

Results

EvaluationLoopResult

IterationResult

Async Support

Best Practices

Judge

Evaluator-Optimizer

Goal Loop

Prompt Optimizer

​How It Works

​Quick Start

​Modes

Optimize Mode

Review Mode

​Configuration

​Keep the Best Iteration

​Score with a Numeric Metric

​Results

​EvaluationLoopResult

​IterationResult

​Async Support

​Best Practices

​Related

Judge

Evaluator-Optimizer

Goal Loop

Prompt Optimizer

How It Works

Quick Start

Modes

Configuration

Keep the Best Iteration

Score with a Numeric Metric

Results

EvaluationLoopResult

IterationResult

Async Support

Best Practices

Related