Skip to main content

​
FleetSupervisionPolicy

Defined in the protocols module.
AI Agent Pure fleet-level crash-loop breaker for channel supervision (Issue #3840). Per-channel restart budgets (ChannelHealthMonitor / ChannelRestartHistory in praisonai-bot) throttle one misbehaving channel, but they are blind to a systemic fault — a bad shared provider, a network partition, an org-wide expired token — that makes every channel restart at once. Each channel then independently stays “under budget” while the fleet as a whole thrashes: a reconnect storm that floods logs, burns CPU, and risks an upstream rate-limit ban with no single operator-visible signal. This is the aggregate breaker that sits on top of the per-channel budgets. Like :class:RestartLoopGuard it is intentionally side-effect free (records timestamps only, no I/O, no heavy deps) so the decision lives in core and is provable in isolation; the wrapper owns the side effects (halting restarts, recording one gateway degraded-owner entry). The breaker trips when either aggregate signal crosses its threshold within the trailing window:
  • the fleet restart rate reaches fleet_restarts_per_hour restarts across all channels, or
  • the fraction of channels in a failing/parked state reaches failing_channel_fraction.
Once tripped it stays tripped for breaker_cooldown_s so the caller applies backpressure (stops auto-restarting, backs off) instead of feeding the storm; after the cooldown it re-arms automatically. Example::

​
Constructor

​
int
default:"40"
No description available.
​
float
default:"0.5"
No description available.
​
float
default:"120.0"
No description available.

​
Methods

note_restart()

Record a fleet restart at now and return whether the breaker is tripped.

note_fleet_state()

Trip the breaker when too large a fraction of the fleet is failing.

tripped()

Return whether the breaker is currently tripped without recording.

reset()

Clear recorded restart history and any active trip (clean recovery).

​
Source

View on GitHub

praisonaiagents/gateway/protocols.py at line 4469

Multimodal