Crash-Loop Guard

Stop auto-resurrecting a channel that keeps crashing on resume before it burns the process into a tight restart loop

from praisonaiagents import Agent

agent = Agent(name="assistant", instructions="Be helpful.")
# A crash-loop guard protects this agent's channel from a tight restart loop
agent.start("Anyone there?")

Stop auto-resurrecting a channel that keeps crashing on resume, before it burns the process into a tight restart loop.

Quick Start

1. Simplest — enable via gateway.yaml

   # gateway.yaml
   lifecycle:
     restart_loop_guard:
       enabled: true

   praisonai gateway start --config gateway.yaml

   Defaults trip the breaker after 3 rapid restarts within 60 seconds for any channel.

2. Custom window

   # gateway.yaml
   lifecycle:
     restart_loop_guard:
       enabled: true
       max_restarts: 5
       window_seconds: 120

   Trip only after 5 restarts inside a 2-minute window — more tolerant of transient flaps.

3. Python — build the pure guard

   import time
   from praisonaiagents.gateway import RestartLoopGuard
   
   guard = RestartLoopGuard(max_restarts=3, window_seconds=60)
   
   # Each time a supervised channel boot crashes:
   if guard.record(now=time.monotonic()):
       # too many restarts too fast — stop auto-resuming this channel
       ...

   When the guard trips, the gateway logs the crash-loop-halted line and stops auto-resurrecting that channel.

How It Works

The guard runs around the channel supervisor loop. Each crashed boot is recorded; once the trailing window holds max_restarts events, the gateway stops resurrecting that channel.

Configuration Options

RestartLoopGuard constructor

from praisonaiagents.gateway import RestartLoopGuard

guard = RestartLoopGuard(max_restarts=3, window_seconds=60.0)

Methods:

lifecycle.restart_loop_guard: YAML block

Accepted at the top level of gateway.yaml or nested under gateway:.

lifecycle:
  restart_loop_guard:
    enabled: true
    max_restarts: 3
    window_seconds: 60

Imports

from praisonaiagents.gateway import RestartLoopGuard

How It Composes With Channel Supervision

The crash-loop guard is a rapid-fire breaker (seconds window) that runs around the supervisor loop. It is orthogonal to the Channel Supervision health-monitor machinery:

Tune all three for a layered defence: the guard catches immediate crash storms per channel; max_restarts_per_hour catches slow, persistent flapping per channel; the fleet breaker catches a systemic fault that restarts every channel at once.

Log signal

When the guard trips, the gateway emits:

ERROR: Channel 'telegram' crash-loop breaker tripped (>= 3 restarts in 60s); halting auto-resume: <original exception>

Fleet-level breaker

The per-channel guards above catch one bad channel. A systemic fault — a bad shared provider, a network partition, an org-wide expired token — restarts every channel at once, and each one silently burns its own max_restarts_per_hour budget in a fleet-wide reconnect storm. FleetSupervisionPolicy sits on top of the per-channel budgets and trips one operator-visible breaker instead.

FleetSupervisionPolicy — a pure aggregate breaker

FleetSupervisionPolicy is a pure, dependency-free, side-effect-free breaker — the aggregate sibling of RestartLoopGuard. Import it from praisonaiagents.gateway:

import time
from praisonaiagents.gateway import FleetSupervisionPolicy

policy = FleetSupervisionPolicy(
    fleet_restarts_per_hour=40,
    failing_channel_fraction=0.5,
    breaker_cooldown_s=120.0,
)

now = time.monotonic()

# Record a fleet restart — True once the aggregate breaker trips.
if policy.note_restart(now):
    ...  # hold channel restarts; a systemic storm is underway

# Or trip on failing fraction: half the fleet failing/parked.
if policy.note_fleet_state(failing=2, total=3, now=now):
    ...

# Non-recording check (re-arms cleanly once cooldown elapses).
policy.tripped(now)

# Clear all recorded history + any active trip.
policy.reset()

Recovery

The breaker re-arms cleanly after breaker_cooldown_s — once cooldown elapses, tripped() clears the event window so pre-trip events cannot immediately re-trip it. When wired into the gateway, the degraded-owner fact (see below) clears on the next monitor sweep without an external get_status() poll: wait one breaker_cooldown_s after the underlying cause is fixed and the entry disappears on its own.

Observability

When wired into the gateway, get_status() / gateway status / /health gain a fleet block:

{
  "enabled": true,
  "running": true,
  "interval": 300,
  "channels": { "...": { "restart_count": 4, "can_restart": false } },
  "fleet": {
    "breaker_tripped": true,
    "fleet_restarts_per_hour": 40,
    "failing_channels": 2,
    "total_channels": 3
  }
}

When the fleet breaker trips, the monitor records exactly one entry on the shared Degraded-Capability Registry:

/

channels failing” (or “channel crash-loop” when total == 0) | | retry_hint | “praisonai gateway doctor”` |

The log line on trip:

ERROR: Fleet crash-loop breaker TRIPPED: holding channel restarts (2/3 channels failing, reason=...). Backing off for 120s; run 'praisonai gateway doctor' to diagnose.

Best Practices

Related