Channel Supervision

Self-healing gateway channels with operator pause/resume/reconnect controls

from praisonaiagents import Agent

agent = Agent(name="supervisor", instructions="Supervise messages across gateway channels.")
agent.start("Monitor all channels and flag inappropriate content.")

Channel supervision keeps gateway bots alive through network outages with unlimited retries and operator-level pause / resume / reconnect controls. Channels with bad or expired tokens never reach the supervisor — praisonai gateway start --preflight (default on) aborts before launch when a credential probe fails; see Pre-flight credential check. Similarly, praisonai gateway start --strict-tools (default on) aborts before launch when any tool named in the config cannot be resolved; see Pre-flight tool check.

# gateway.yaml — supervision enabled automatically
agents:
  assistant:
    instructions: "You are a helpful AI assistant."
    model: gpt-4o-mini

channels:
  telegram:
    token: "${TELEGRAM_BOT_TOKEN}"
    platform: telegram

The user messages the bot on Telegram; channel supervision reconnects the channel after network blips without operator intervention.

Quick Start

Channel supervision is automatically enabled for all gateway channels configured in gateway.yaml. No additional setup is required.

1. Basic Gateway Setup

   Create a simple gateway with supervision:

   # gateway.yaml
   agents:
     assistant:
       instructions: "You are a helpful AI assistant."
       model: "gpt-4o-mini"
   
   channels:
     telegram:
       token: "${TELEGRAM_BOT_TOKEN}"
       platform: telegram

   praisonai gateway start --config gateway.yaml

   The telegram channel is now under supervision with unlimited retry capability.

2. Control Channel Operations

   Pause a problematic channel while investigating issues:

   praisonai gateway pause telegram

   Resume when ready:

   praisonai gateway resume telegram

   Force reconnect to reset error state:

   praisonai gateway reconnect telegram

How It Works

Channel supervision provides resilient error handling through error classification and unlimited retries:

Channel States

The supervision system tracks five distinct channel states:

Recovery-triggered outbox re-drain

When a channel transitions back to RUNNING after a recoverable failure (monitor.attempt > 0), the supervisor schedules a background re-drain of the durable outbox so held replies go out promptly.

• Non-blocking — scheduled with asyncio.create_task(...), so supervision never waits on the re-drain.
• Silent no-op without a hook — the supervisor resolves drain_outbox on the supervised object first, then on bot.adapter. Bots/adapters without the hook are skipped.
• Best-effort — a failed re-drain is logged at WARNING and swallowed; the outbox’s attempt-and-age policy still governs those messages.
• Recovery only — a clean first start does not trigger it; only a real recovery (at least one recoverable failure) does.

Cross-references: Durable Delivery → When the Outbox Drains (all three drain triggers) and Outbound Resilience → Automatic crash-recovery on start (the separate boot-time DLQ drainer).

Introduced in PraisonAI PR #4046 (fixes #4043).

Proactive Health Monitoring

The health monitor periodically asks each channel “Are you really alive?” and restarts the ones that aren’t, without waiting for an exception to be raised.

Enable via YAML

gateway:
  host: 127.0.0.1
  port: 8765
  health:
    interval: 300            # seconds between sweeps (default 300)
    startup_grace: 60        # grace window after start (default 60)
    stale_after: 120         # no inbound activity (idle) → stale-socket (default 120)
    stuck_after: 900         # busy with no progress (inbound OR in-run) → stuck (default 900)
    max_restarts_per_hour: 10        # per channel
    # Fleet-level crash-loop breaker (aggregate across channels):
    fleet_restarts_per_hour: 40      # int >= 1 (default 40)
    failing_channel_fraction: 0.5    # float in (0.0, 1.0] (default 0.5)
    breaker_cooldown_s: 120          # float >= 0 (default 120)
    enabled: true            # default true

Enable via Python

from praisonai.gateway.health_monitor import HealthMonitorConfig
from praisonai.gateway.supervisor import ChannelSupervisor

health_config = HealthMonitorConfig(interval=120, stale_after=180)
supervisor = ChannelSupervisor(health_config=health_config)

Configuration Options

HealthReason values

Decision order in evaluate_channel_health():

Passive inbound liveness

Channel liveness is driven by whether messages are still flowing IN, not just whether the outbound probe (e.g. Telegram getMe) succeeds. Every inbound message refreshes the timestamp via fire_message_received → _note_inbound(), so a reachable-but-deaf channel will eventually trip stale-socket instead of being reported healthy forever. All adapters get this automatically; WhatsApp, Linear, and Signal are wired explicitly because they bypass the shared handler. In Signal specifically, an empty poll from the signal-cli-rest-api bridge is not counted as inbound activity — so a silently half-open bridge that returns [] forever will still trip stale-socket after stale_after seconds instead of being reported healthy indefinitely.

Run awareness

The evaluator knows about in-flight agent turns (HealthResult.active_runs) and tracks per-run progress (HealthResult.last_run_progress, refreshed on tool calls, streamed tokens, and agent emitter events). A busy channel is never killed mid-run — BUSY is non-recoverable on purpose. Only when a busy channel has made no progress at all (neither inbound transport activity nor in-run progress) for stuck_after seconds does it escalate to STUCK (recoverable). An actively-streaming long agent turn stays BUSY indefinitely; a genuinely-hung turn that emits nothing for stuck_after is still correctly flagged STUCK.

Restart guard-rails

• Startup grace — no restarts during the first startup_grace seconds after connect
• 5-minute cooldown — implicit cooldown after every restart (logged as restart cooldown active)
• max_restarts_per_hour — when the cap is hit, a warning is logged and the restart is skipped
• Crash-Loop Guard — a rapid-fire breaker (seconds window) that runs around the supervisor loop and halts auto-resume after a burst of crash-on-resume restarts, complementing the per-hour cap
• Fleet-level breaker — an aggregate breaker (fleet_restarts_per_hour, failing_channel_fraction, breaker_cooldown_s) that trips once when a systemic fault restarts every channel at the same time, instead of each channel silently burning its own budget

WARNING: channel telegram: max restarts per hour (10) reached, skipping restart

Fleet breaker status

get_status() (surfaced by gateway status and /health) includes a fleet block so operators can see the aggregate breaker at a glance:

{
  "enabled": true,
  "running": true,
  "interval": 300,
  "channels": { "telegram": { "restart_count": 4, "can_restart": false } },
  "fleet": {
    "breaker_tripped": true,
    "fleet_restarts_per_hour": 40,
    "failing_channels": 2,
    "total_channels": 3
  }
}

breaker_tripped: true means restarts are being held fleet-wide; the same event records one gateway/fleet entry on the Degraded-Capability Registry. See Fleet-level breaker for the full model.

Suspend / resume monitoring

For planned maintenance, suspend health checks on a channel without stopping supervision:

monitor.suspend_channel("telegram")   # skip health sweeps
monitor.resume_channel("telegram")    # resume sweeps

Operator Controls

Pause Channel

Temporarily stop a channel without losing configuration:

praisonai gateway pause telegram --url ws://127.0.0.1:8765

curl -X POST http://127.0.0.1:8765/api/channels/telegram/pause \
     -H "Authorization: Bearer YOUR_TOKEN"

Effect: Channel enters PAUSED state and stops processing messages. Supervision loop waits indefinitely until resumed.

Resume Channel

Resume a manually paused channel:

praisonai gateway resume telegram --url ws://127.0.0.1:8765

curl -X POST http://127.0.0.1:8765/api/channels/telegram/resume \
     -H "Authorization: Bearer YOUR_TOKEN"

Effect: Channel transitions from PAUSED to STOPPED, then automatically restarts to RUNNING.

Reconnect Channel

Force a complete reconnection and reset error state:

praisonai gateway reconnect telegram --url ws://127.0.0.1:8765

curl -X POST http://127.0.0.1:8765/api/channels/telegram/reconnect \
     -H "Authorization: Bearer YOUR_TOKEN"

Effect: Resets retry counter, clears error history, forces restart. Works from any state including FAILED.

Error Classification

The supervision system classifies errors to determine retry behavior:

The retry policy uses capped exponential backoff:

• Initial delay: 5 seconds
• Maximum delay: 300 seconds (5 minutes)
• Unlimited attempts for recoverable errors
• Jitter added to prevent thundering herd

Runtime credential rejection

A mid-flight 401/403 or revoked-token error is its own outcome — the channel parks in CREDENTIAL_UNAVAILABLE, not FAILED.

FAILED is terminal and assumes a full process restart is needed. A credential rejection is different: it self-heals the moment the operator fixes the token, so the supervisor stops hammering the invalid token and waits — it does not loop-retry — until a reconnect(), resume(), or config hot-reload wakes it, then restarts the same bot instance without a process restart.

This is the runtime counterpart of the boot-time case documented on Degraded Channel Isolation: boot-time = an empty token at config load; runtime = a valid-looking token the platform rejected while the channel was live. Both surface identically as status: "degraded", reason: "credential unavailable".

Recovering from CREDENTIAL_UNAVAILABLE

1. Fix the credential

   Rotate the env var, refresh the mounted secret file, or re-issue the OAuth token so a fresh, valid credential is available.

2. Trigger recovery

   Any of three paths wakes the parked channel:

   • Config hot-reload — rebuilds the bot with the new token; the reload’s abort signal wakes the parked channel. See Gateway Hot Reload.
   • praisonai gateway reconnect <channel> — enough when the token lives in an env var (base Bot.start() re-reads env-var tokens on restart) or when the bot implements refresh_credentials().
   • praisonai gateway resume <channel> — only if you had also paused it; the manual_pause flag is orthogonal to the credential state.
3. Verify recovery

   Check GET /health: the channel’s status returns to running and it drops out of any degraded view.

Implementing refresh_credentials() on custom bots

A bot can opt in to re-sourcing its token on wake by exposing a refresh_credentials() method — the only new API surface bot authors need for this feature.

from praisonai_bot.bots import Bot  # existing base class

# `my_secret_manager` is illustrative — swap in your own client.
class MySecretManagerBot(Bot):
    async def refresh_credentials(self):
        self._token = await my_secret_manager.rotate(self.platform)

• Duck-typed — subclassing Bot is optional; any object with a callable refresh_credentials attribute works.
• Sync or async — the supervisor awaits coroutines.
• Best-effort — a raising hook logs a warning and the restart proceeds anyway; if the credential is still bad the channel simply re-parks, so there is no infinite loop.
• When you need it — only when your credential lives outside env vars / mounted files and cannot be picked up by a hot-reload or by Bot.start() re-reading env vars. Most operators do not need this.

Monitoring via /health

The enhanced health endpoint includes supervision status for each channel:

curl http://127.0.0.1:8765/health

{
  "status": "healthy",
  "uptime": 3600,
  "agents": 2,
  "sessions": 5,
  "clients": 3,
  "channels": {
    "telegram": {
      "platform": "telegram",
      "running": true,
      "last_activity": 1672531500,
      "active_runs": 2,
      "supervision": {
        "state": "running",
        "last_error": null,
        "last_error_time": null,
        "next_retry_at": null,
        "total_recoveries": 3,
        "manual_pause": false
      },
      "health_monitor": {
        "enabled": true,
        "running": true,
        "interval": 300,
        "last_check": 1672531200,
        "suspended": false,
        "restart_count": 1,
        "can_restart": true
      }
    },
    "discord": {
      "platform": "discord", 
      "running": false,
      "supervision": {
        "state": "failed",
        "last_error": "Fatal error: unexpected programming error",
        "last_error_time": 1672531200,
        "next_retry_at": null,
        "total_recoveries": 0,
        "manual_pause": false
      }
    },
    "slack": {
      "platform": "slack",
      "running": false,
      "status": "degraded",
      "reason": "credential unavailable",
      "supervision": {
        "state": "credential-unavailable",
        "last_error": "credential unavailable",
        "last_error_time": 1672531200,
        "next_retry_at": null,
        "total_recoveries": 0,
        "manual_pause": false
      }
    }
  }
}

Key supervision fields:

• state: Current channel state (running, failed, paused, stopped, credential-unavailable)
• last_error: Most recent error message (if any)
• last_error_time: Unix timestamp of last error
• next_retry_at: Unix timestamp of next retry attempt (if scheduled)
• total_recoveries: Count of successful recoveries from errors
• manual_pause: Whether channel is manually paused by operator
• active_runs: Number of in-flight agent turns (busy count)
• last_activity: Unix timestamp of last inbound transport activity (used for stale-socket and stuck detection)
• health_monitor.enabled: Whether proactive monitoring is active
• health_monitor.restart_count: Restarts in the current hour
• health_monitor.can_restart: Whether guard-rails allow another restart

Best Practices

Related