Gateway Fleet Breaker

Stops fleet-wide reconnect storms when every channel restarts at once

from praisonaiagents import Agent

agent = Agent(name="Support", instructions="Help operators triage gateway health")
agent.start("Is the gateway holding channel restarts right now?")

The fleet breaker sits on top of the per-channel restart budget and trips one aggregate circuit when a systemic fault — a bad shared provider, a network partition, an org-wide expired token — makes every channel restart at once. Instead of a silent fleet-wide reconnect storm, restarts are held and a single operator-facing degraded fact appears with a next step.

Quick Start

1. Simple — nothing to change

   The breaker is on by default with sensible thresholds. Watch the new fleet block in health output:

   praisonai gateway status --deep

   "fleet": {
     "breaker_tripped": false,
     "fleet_restarts_per_hour": 40,
     "failing_channels": 0,
     "total_channels": 3
   }

   breaker_tripped: false with failing_channels: 0 means the fleet is healthy — no action needed.

2. Tune the thresholds

   Set any of the three thresholds under gateway.health. Tuning is optional; the defaults trip on a real storm.

   # gateway.yaml — tuning is optional; the defaults trip on a real storm.
   gateway:
     health:
       fleet_restarts_per_hour: 40
       failing_channel_fraction: 0.5
       breaker_cooldown_s: 120

What “tripped” means

A tripped breaker means the gateway detected a fleet-wide storm and is now holding channel restarts to avoid feeding it — not that a single channel failed.

The breaker trips when either aggregate signal crosses its threshold within the trailing hour: the fleet restart rate reaches fleet_restarts_per_hour, or the fraction of failing channels reaches failing_channel_fraction. It stays tripped for breaker_cooldown_s, then re-arms on its own.

Agent-centric quick start

Does it protect you out of the box? Yes — the defaults trip on a real storm, and the same doctor already knows what to say.

# gateway.yaml — tuning is optional; the defaults trip on a real storm.
gateway:
  health:
    fleet_restarts_per_hour: 40
    failing_channel_fraction: 0.5
    breaker_cooldown_s: 120

# See whether the breaker is holding right now:
praisonai gateway status --deep

# If health() flags the fleet is degraded, the same doctor already knows what to say:
praisonai gateway doctor

When the breaker is tripped, the degraded registry carries one gateway / fleet fact. Its retry_hint points straight at the next command:

{
  "owner_kind": "gateway",
  "owner_id": "fleet",
  "state": "stale",
  "reason": "channel crash-loop: 2/3 channels failing",
  "retry_hint": "praisonai gateway doctor"
}

The fact clears automatically once the storm subsides.

Configuration Options

All three thresholds live under gateway.health. Values verified against HealthMonitorConfig (praisonai_bot/gateway/health_monitor.py) and FleetSupervisionPolicy (praisonaiagents/gateway/protocols.py).

The fleet block on gateway status health output carries the live state:

Common Patterns

Trip earlier on a smaller fleet — lower the fraction so a couple of failing channels are enough:

gateway:
  health:
    failing_channel_fraction: 0.34

Hold longer during a known-flaky upstream — extend the cooldown so the breaker does not re-arm into the same storm:

gateway:
  health:
    breaker_cooldown_s: 600

Best Practices

Related