Channel Supervision

The gateway now ships in the praisonai-bot package. praisonai serve gateway still works exactly as documented here; for a standalone install see praisonai-bot Migration.

from praisonaiagents import Agent

agent = Agent(name="supervisor", instructions="Supervise messages across gateway channels.")
agent.start("Monitor all channels and flag inappropriate content.")

Channel supervision keeps gateway bots alive through network outages with unlimited retries and operator-level pause / resume / reconnect controls. Channels with bad or expired tokens never reach the supervisor — praisonai gateway start --preflight (default on) aborts before launch when a credential probe fails; see Pre-flight credential check.

# gateway.yaml — supervision enabled automatically
agents:
  assistant:
    instructions: "You are a helpful AI assistant."
    model: gpt-4o-mini

channels:
  telegram:
    token: "${TELEGRAM_BOT_TOKEN}"
    platform: telegram

The user messages the bot on Telegram; channel supervision reconnects the channel after network blips without operator intervention.

Quick Start

Channel supervision is automatically enabled for all gateway channels configured in gateway.yaml. No additional setup is required.

Basic Gateway Setup

Create a simple gateway with supervision:

# gateway.yaml
agents:
  assistant:
    instructions: "You are a helpful AI assistant."
    model: "gpt-4o-mini"

channels:
  telegram:
    token: "${TELEGRAM_BOT_TOKEN}"
    platform: telegram

praisonai gateway start --config gateway.yaml

The telegram channel is now under supervision with unlimited retry capability.

Control Channel Operations

Pause a problematic channel while investigating issues:

praisonai gateway pause telegram

Resume when ready:

praisonai gateway resume telegram

Force reconnect to reset error state:

praisonai gateway reconnect telegram

How It Works

Channel supervision provides resilient error handling through error classification and unlimited retries:

Component	Responsibility
ChannelSupervisor	Manages channel lifecycle and error handling
BackoffPolicy	Controls retry timing with capped exponential backoff
Error Classification	Determines if errors are recoverable, fatal, or conflict
Operator Controls	Provides manual pause/resume/reconnect capabilities

Channel States

The supervision system tracks four distinct channel states:

State	Description	Auto-Retry	Operator Actions
`RUNNING`	Channel is actively connected and serving messages	N/A	pause, reconnect
`FAILED`	Fatal error occurred (e.g., Telegram conflict, invalid token)	❌ No	reconnect only
`PAUSED`	Manually paused by operator	❌ No	resume, reconnect
`STOPPED`	Clean shutdown or initial state	❌ No	Automatic restart

Proactive health monitoring can move a channel from RUNNING into a restart cycle without a raised exception — for example when transport activity goes stale (stale-socket) or a probe fails (disconnected).

Proactive Health Monitoring

The health monitor periodically asks each channel “Are you really alive?” and restarts the ones that aren’t, without waiting for an exception to be raised.

Enable via YAML

gateway:
  host: 127.0.0.1
  port: 8765
  health:
    interval: 300            # seconds between sweeps (default 300)
    startup_grace: 60        # grace window after start (default 60)
    stale_after: 120         # no inbound activity (idle) → stale-socket (default 120)
    stuck_after: 900         # busy with no progress (inbound OR in-run) → stuck (default 900)
    max_restarts_per_hour: 10
    enabled: true            # default true

Enable via Python

from praisonai.gateway.health_monitor import HealthMonitorConfig
from praisonai.gateway.supervisor import ChannelSupervisor

health_config = HealthMonitorConfig(interval=120, stale_after=180)
supervisor = ChannelSupervisor(health_config=health_config)

Configuration Options

Option	Type	Default	Description
`interval`	`float`	`300.0`	Seconds between health sweeps
`startup_grace`	`float`	`60.0`	Seconds after start before checks count
`stale_after`	`float`	`120.0`	Seconds without inbound transport activity (while idle) → `stale-socket`
`stuck_after`	`float`	`900.0`	Seconds a busy channel can go without any progress signal (inbound transport activity OR in-run progress — tool calls, streamed tokens, emitter events) → `stuck`
`max_restarts_per_hour`	`int`	`10`	Hard cap on restart attempts per channel per hour
`enabled`	`bool`	`True`	Whether monitoring is enabled

HealthReason values

Reason	Recoverable	When it fires
`healthy`	No	All checks pass
`not-running`	No	`health.is_running` is false
`startup-grace`	No	`uptime_seconds < startup_grace`
`disconnected`	Yes	`health.probe` exists and `probe.ok` is false
`stale-socket`	Yes	Idle and no inbound activity for `stale_after` seconds
`busy`	No	`active_runs > 0` with progress within `stuck_after` — protects long runs from mid-run kills
`stuck`	Yes	`active_runs > 0` and no progress (inbound or in-run) for `> stuck_after` — likely wedged
`error`	Yes	`health.error` set or `health.ok` is false

Decision order in evaluate_channel_health():

Passive inbound liveness

Channel liveness is driven by whether messages are still flowing IN, not just whether the outbound probe (e.g. Telegram getMe) succeeds. Every inbound message refreshes the timestamp via fire_message_received → _note_inbound(), so a reachable-but-deaf channel will eventually trip stale-socket instead of being reported healthy forever. All adapters get this automatically; WhatsApp and Linear are wired explicitly because they bypass the shared handler.

Run awareness

The evaluator knows about in-flight agent turns (HealthResult.active_runs) and tracks per-run progress (HealthResult.last_run_progress, refreshed on tool calls, streamed tokens, and agent emitter events). A busy channel is never killed mid-run — BUSY is non-recoverable on purpose. Only when a busy channel has made no progress at all (neither inbound transport activity nor in-run progress) for stuck_after seconds does it escalate to STUCK (recoverable). An actively-streaming long agent turn stays BUSY indefinitely; a genuinely-hung turn that emits nothing for stuck_after is still correctly flagged STUCK.

Before praisonaiagents 1.6.82, only inbound transport activity counted toward progress, so a single long agent turn (deep research, big refactor, slow model) was classified STUCK once it crossed stuck_after — even while actively streaming. The protocol now honours in-run progress (HealthResult.last_run_progress), so progressing long runs stay BUSY indefinitely. See PraisonAI #2400.

Restart guard-rails

Startup grace — no restarts during the first startup_grace seconds after connect
5-minute cooldown — implicit cooldown after every restart (logged as restart cooldown active)
max_restarts_per_hour — when the cap is hit, a warning is logged and the restart is skipped

WARNING: channel telegram: max restarts per hour (10) reached, skipping restart

Suspend / resume monitoring

For planned maintenance, suspend health checks on a channel without stopping supervision:

monitor.suspend_channel("telegram")   # skip health sweeps
monitor.resume_channel("telegram")    # resume sweeps

Operator Controls

Pause Channel

Temporarily stop a channel without losing configuration:

CLI
REST API

praisonai gateway pause telegram --url ws://127.0.0.1:8765

curl -X POST http://127.0.0.1:8765/api/channels/telegram/pause \
     -H "Authorization: Bearer YOUR_TOKEN"

Effect: Channel enters PAUSED state and stops processing messages. Supervision loop waits indefinitely until resumed.

Resume Channel

Resume a manually paused channel:

CLI
REST API

praisonai gateway resume telegram --url ws://127.0.0.1:8765

curl -X POST http://127.0.0.1:8765/api/channels/telegram/resume \
     -H "Authorization: Bearer YOUR_TOKEN"

Effect: Channel transitions from PAUSED to STOPPED, then automatically restarts to RUNNING.

Reconnect Channel

Force a complete reconnection and reset error state:

CLI
REST API

praisonai gateway reconnect telegram --url ws://127.0.0.1:8765

curl -X POST http://127.0.0.1:8765/api/channels/telegram/reconnect \
     -H "Authorization: Bearer YOUR_TOKEN"

Effect: Resets retry counter, clears error history, forces restart. Works from any state including FAILED.

Error Classification

The supervision system classifies errors to determine retry behavior:

Error Type	Examples	Behavior	Recovery
Recoverable	Network timeouts, DNS failures, temporary API errors	Unlimited retry with exponential backoff	Automatic
Conflict	Telegram “Conflict: terminated by other getUpdates”	Immediate failure, no retry	Manual `reconnect` after stopping duplicate
Non-Recoverable	Invalid bot token, missing permissions	Immediate failure, no retry	Manual `reconnect` after fixing config

The retry policy uses capped exponential backoff:

Initial delay: 5 seconds
Maximum delay: 300 seconds (5 minutes)
Unlimited attempts for recoverable errors
Jitter added to prevent thundering herd

Monitoring via `/health`

The enhanced health endpoint includes supervision status for each channel:

Request
Response

curl http://127.0.0.1:8765/health

{
  "status": "healthy",
  "uptime": 3600,
  "agents": 2,
  "sessions": 5,
  "clients": 3,
  "channels": {
    "telegram": {
      "platform": "telegram",
      "running": true,
      "last_activity": 1672531500,
      "active_runs": 2,
      "supervision": {
        "state": "running",
        "last_error": null,
        "last_error_time": null,
        "next_retry_at": null,
        "total_recoveries": 3,
        "manual_pause": false
      },
      "health_monitor": {
        "enabled": true,
        "running": true,
        "interval": 300,
        "last_check": 1672531200,
        "suspended": false,
        "restart_count": 1,
        "can_restart": true
      }
    },
    "discord": {
      "platform": "discord", 
      "running": false,
      "supervision": {
        "state": "failed",
        "last_error": "Invalid bot token",
        "last_error_time": 1672531200,
        "next_retry_at": null,
        "total_recoveries": 0,
        "manual_pause": false
      }
    }
  }
}

Key supervision fields:

state: Current channel state (running, failed, paused, stopped)
last_error: Most recent error message (if any)
last_error_time: Unix timestamp of last error
next_retry_at: Unix timestamp of next retry attempt (if scheduled)
total_recoveries: Count of successful recoveries from errors
manual_pause: Whether channel is manually paused by operator
active_runs: Number of in-flight agent turns (busy count)
last_activity: Unix timestamp of last inbound transport activity (used for stale-socket and stuck detection)
health_monitor.enabled: Whether proactive monitoring is active
health_monitor.restart_count: Restarts in the current hour
health_monitor.can_restart: Whether guard-rails allow another restart

Best Practices

When to pause vs reconnect

Use pause for temporary investigations while keeping the channel configuration intact. Use reconnect when you need to reset error state after fixing underlying issues like network connectivity or API tokens.

Reading total_recoveries as a churn signal

High total_recoveries counts indicate frequent connection issues. Monitor this metric to identify unstable network conditions or platform-specific problems that may require infrastructure changes.

Hooking /health into monitoring systems

The /health endpoint is designed for integration with Prometheus, Datadog, or other monitoring systems. Set up alerts on state: "failed" and track total_recoveries trends to detect degrading connection quality.

Recovering from FAILED state

Channels in FAILED state require manual intervention. Use reconnect (not resume) to reset the error state and attempt a fresh connection. Always investigate the last_error to address root cause issues before reconnecting.

Tuning interval and stale_after for chatty vs quiet channels

Low-traffic channels need a larger stale_after (e.g. 600s) to avoid false stale-socket restarts. Chatty channels can use the default 120s.

Tuning stuck_after for long-running agent turns

If your agent regularly runs turns longer than 15 minutes (deep research, long tool chains), raise stuck_after so genuine progress isn’t classified as wedged. The default 900s covers most chat and triage workloads.

When to set max_restarts_per_hour low

When the upstream API is rate-limited, restart storms make the problem worse. Lower the cap (e.g. 3) so the guard-rail surfaces the issue in logs instead of hammering the API.

Dead Target Registry

Suppress permanently-dead channels — bot kicked, chat deleted, account deactivated

Gateway CLI

Complete CLI reference for gateway management

Gateway Error Handling

Error handling strategies for gateway bots

BotOS

Multi-platform orchestrator with the same supervision and health monitoring

Bot Loop Protection

Break runaway bot-to-bot reply loops — a pure decision protocol like evaluate_channel_health

​Quick Start

​How It Works

​Channel States

​Proactive Health Monitoring

​Enable via YAML

​Enable via Python

​Configuration Options

​HealthReason values

​Passive inbound liveness

​Run awareness

​Restart guard-rails

​Suspend / resume monitoring

​Operator Controls

​Pause Channel

​Resume Channel

​Reconnect Channel

​Error Classification

​Monitoring via /health

​Best Practices

​Related

Dead Target Registry

Gateway CLI

Gateway Error Handling

BotOS

Bot Loop Protection

Quick Start

How It Works

Channel States

Proactive Health Monitoring

Enable via YAML

Enable via Python

Configuration Options

HealthReason values

Passive inbound liveness

Run awareness

Restart guard-rails

Suspend / resume monitoring

Operator Controls

Pause Channel

Resume Channel

Reconnect Channel

Error Classification

Monitoring via `/health`

Best Practices

Related