Loop Evaluator

Score loop health — convergence, wasted iterations, and doom-loop guard events

Loop Evaluator scores how healthy a loop run was — did it converge, were iterations wasted, and did a doom-loop guard fire — on top of EvaluationLoop‘s output-quality score.

A loop can succeed (score ≥ 8.0) while burning max iterations, triggering doom-loop recovery, or costing 5× the expected tokens. EvaluationLoop scores output quality; LoopEvaluator scores loop health on top of it — no extra LLM calls.

Quick Start

1. Score an already-run loop

   Run an EvaluationLoop, then score its health.

   from praisonaiagents import Agent
   from praisonaiagents.eval import EvaluationLoop, LoopEvaluator
   
   agent = Agent(name="writer", instructions="Write clear explanations.")
   loop = EvaluationLoop(agent=agent, criteria="Explanation is clear", threshold=8.0, max_iterations=5)
   
   loop_result = loop.run("Explain how HTTPS works.")
   
   health = LoopEvaluator().run(loop_result)
   health.print_summary()   # ✅ HEALTHY / iterations / waste / doom-loop

2. Tolerate wasted iterations

   Set an explicit threshold and allow one wasted iteration.

   health = LoopEvaluator(threshold=8.0, max_wasted_iterations=1).run(loop_result)
   assert health.passed

3. Pass structured guard events

   Feed doom-loop guard events. The evaluator is duck-typed — dicts or objects both work.

   # Any of these shapes is recognised:
   guard_events = [
       {"type": "doom_loop", "reason": "repeated tool call"},   # generic marker
       # or a DoomLoopEvent-style object with a .loop_type
       # or a DoomLoopResult-style dict {"is_loop": True}
   ]
   
   health = LoopEvaluator().run(loop_result, guard_events=guard_events)
   assert health.doom_loop_fired is True
   assert not health.passed

How It Works

LoopEvaluator derives health from a completed EvaluationLoopResult — it never re-runs the loop, so it makes zero LLM calls and is safe in CI.

sequenceDiagram
    participant Loop as EvaluationLoop
    participant Result as EvaluationLoopResult
    participant Eval as LoopEvaluator
    participant Health as LoopHealthResult

    Loop->>Result: run() → score_history, threshold, mode
    Result->>Eval: run(loop_result, guard_events)
    Eval->>Eval: Derive convergence + waste + doom-loop
    Eval-->>Health: iterations, waste, doom_loop_fired, passed

Mode-Aware Convergence

Convergence is computed differently in optimize vs review mode — the health verdict never disagrees with the loop’s own verdict.

• optimize mode: convergence = first iteration whose score ≥ threshold. wasted_iterations = num_iterations − iterations_to_success.
• review mode: EvaluationLoop runs every iteration and derives success from the final score, so a transient earlier high score is not treated as convergence. LoopEvaluator defers to loop_result.success (or score_history[-1] >= threshold). wasted_iterations is always 0.

Configuration Options

Constructor parameters for LoopEvaluator.

run(loop_result, guard_events=None) takes a completed EvaluationLoopResult (or any object exposing score_history, threshold, total_duration_seconds, mode, success) and an optional list of guard events.

LoopHealthResult Fields

run() returns a LoopHealthResult. These are the fields returned by to_dict().

to_dict(), to_json(), and print_summary() are all provided.

Recognised Guard-Event Shapes

Guard events are duck-typed — the evaluator detects a doom-loop without coupling to any specific class.

Accepted type / event string values (case-insensitive): doom_loop, doom-loop, doomloop, repetition, loop_detected, repeated_action, repeated_failure, no_progress, circular_plan, resource_exhaustion, repeated_output.

Common Patterns

CI Gate on Loop Health

Fail the build when a loop wastes iterations or a doom-loop fires.

import sys
from praisonaiagents.eval import LoopEvaluator

health = LoopEvaluator(max_wasted_iterations=0).run(loop_result)
if not health.passed:
    print(f"Loop unhealthy: {health.reasoning}")
    sys.exit(1)

Combine Quality and Health

Score output quality (EvaluationLoop) and loop health (LoopEvaluator) on the same run — they are complementary.

from praisonaiagents import Agent
from praisonaiagents.eval import EvaluationLoop, LoopEvaluator

agent = Agent(name="analyst", instructions="Analyze systems thoroughly.")
loop = EvaluationLoop(agent=agent, criteria="Analysis is thorough", threshold=8.0)

loop_result = loop.run("Analyze the auth flow")

print(f"Quality: {loop_result.final_score}/10, success={loop_result.success}")

health = LoopEvaluator().run(loop_result)
print(f"Health: passed={health.passed}, wasted={health.wasted_iterations}")

Wire Real Doom-Loop Guards

Plug in structured DoomLoopEvents (or loop_detection_plugin events) as guard_events.

from praisonaiagents.escalation.doom_loop import DoomLoopEvent  # emits .loop_type
from praisonaiagents.eval import LoopEvaluator

# guard_events collected during the agent run
health = LoopEvaluator().run(loop_result, guard_events=guard_events)
if health.doom_loop_fired:
    print(f"Doom-loop fired: {health.reasoning}")

Best Practices

Related