Fleet Supervision Policy • AI Agent SDK
FleetSupervisionPolicy: Pure fleet-level crash-loop breaker for channel supervision (Issue #3840).
FleetSupervisionPolicy
Defined in the protocols module.
AI Agent
Pure fleet-level crash-loop breaker for channel supervision (Issue #3840).
Per-channel restart budgets (`ChannelHealthMonitor / ChannelRestartHistory in praisonai-bot`) throttle one misbehaving channel, but they are blind to a systemic fault — a bad shared provider, a network partition, an org-wide expired token — that makes every channel restart at once. Each channel then independently stays “under budget” while the fleet as a whole thrashes: a reconnect storm that floods logs, burns CPU, and risks an upstream rate-limit ban with no single operator-visible signal.
This is the aggregate breaker that sits on top of the per-channel budgets. Like :class:RestartLoopGuard it is intentionally side-effect free (records timestamps only, no I/O, no heavy deps) so the decision lives in core and is provable in isolation; the wrapper owns the side effects (halting restarts, recording one `gateway` degraded-owner entry).
The breaker trips when either aggregate signal crosses its threshold within the trailing window:
- the fleet restart rate reaches `
fleet_restarts_per_hour` restarts across - all channels, or
- the fraction of channels in a failing/parked state reaches
- `
failing_channel_fraction`.
Once tripped it stays tripped for `breaker_cooldown_s` so the caller applies backpressure (stops auto-restarting, backs off) instead of feeding the storm; after the cooldown it re-arms automatically.
Example::
policy = FleetSupervisionPolicy(fleet_restarts_per_hour=40)
if policy.note_restart(now=time.monotonic()):
# fleet breaker tripped — stop auto-restarting, surface degraded
...Constructor
intNo description available.
floatNo description available.
floatNo description available.
Methods
Record a fleet restart at `now` and return whether the breaker is tripped.
Trip the breaker when too large a fraction of the fleet is failing.
tripped()Return whether the breaker is currently tripped without recording.
reset()Clear recorded restart history and any active trip (clean recovery).
Source
View on GitHubpraisonaiagents/gateway/protocols.py at line 6384