Fleet Supervision Policy • AI Agent SDK

FleetSupervisionPolicy: Pure fleet-level crash-loop breaker for channel supervision (Issue #3840).

FleetSupervisionPolicy

Defined in the protocols module.

AI Agent

Pure fleet-level crash-loop breaker for channel supervision (Issue #3840).

Per-channel restart budgets (`ChannelHealthMonitor / ChannelRestartHistory in praisonai-bot`) throttle one misbehaving channel, but they are blind to a systemic fault — a bad shared provider, a network partition, an org-wide expired token — that makes every channel restart at once. Each channel then independently stays “under budget” while the fleet as a whole thrashes: a reconnect storm that floods logs, burns CPU, and risks an upstream rate-limit ban with no single operator-visible signal.

This is the aggregate breaker that sits on top of the per-channel budgets. Like :class:RestartLoopGuard it is intentionally side-effect free (records timestamps only, no I/O, no heavy deps) so the decision lives in core and is provable in isolation; the wrapper owns the side effects (halting restarts, recording one `gateway` degraded-owner entry).

The breaker trips when either aggregate signal crosses its threshold within the trailing window:

  • the fleet restart rate reaches `fleet_restarts_per_hour` restarts across
  • all channels, or
  • the fraction of channels in a failing/parked state reaches
  • `failing_channel_fraction`.

Once tripped it stays tripped for `breaker_cooldown_s` so the caller applies backpressure (stops auto-restarting, backs off) instead of feeding the storm; after the cooldown it re-arms automatically.

Example::

python
policy = FleetSupervisionPolicy(fleet_restarts_per_hour=40)
if policy.note_restart(now=time.monotonic()):
    # fleet breaker tripped — stop auto-restarting, surface degraded
    ...

Constructor

ParamField int

No description available.

ParamField float

No description available.

ParamField float

No description available.

Methods

Source

View on GitHub

praisonaiagents/gateway/protocols.py at line 6384


Related Documentation