> ## Documentation Index > Fetch the complete documentation index at: https://praison.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Channel Supervision > Self-healing gateway channels with operator pause/resume/reconnect controls The gateway now ships in the `praisonai-bot` package. `praisonai serve gateway` still works exactly as documented here; for a standalone install see [praisonai-bot Migration](/docs/guides/praisonai-bot-migration). ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent agent = Agent(name="supervisor", instructions="Supervise messages across gateway channels.") agent.start("Monitor all channels and flag inappropriate content.") ``` Channel supervision keeps gateway bots alive through network outages with unlimited retries and operator-level pause / resume / reconnect controls. Channels with bad or expired tokens never reach the supervisor β€” `praisonai gateway start --preflight` (default on) aborts before launch when a credential probe fails; see [Pre-flight credential check](/docs/features/gateway-cli#pre-flight-credential-check). ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # gateway.yaml β€” supervision enabled automatically agents: assistant: instructions: "You are a helpful AI assistant." model: gpt-4o-mini channels: telegram: token: "${TELEGRAM_BOT_TOKEN}" platform: telegram ``` The user messages the bot on Telegram; channel supervision reconnects the channel after network blips without operator intervention. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph LR subgraph "Channel Supervision State Machine" STOPPED[πŸ›‘ STOPPED] --> RUNNING[▢️ RUNNING] RUNNING --> FAILED[❌ FAILED] RUNNING --> PAUSED[⏸️ PAUSED] FAILED --> STOPPED PAUSED --> STOPPED RUNNING -->|operator| PAUSED PAUSED -->|operator| RUNNING FAILED -->|reconnect| STOPPED end classDef running fill:#10B981,stroke:#7C90A0,color:#fff classDef failed fill:#8B0000,stroke:#7C90A0,color:#fff classDef paused fill:#F59E0B,stroke:#7C90A0,color:#fff classDef stopped fill:#6366F1,stroke:#7C90A0,color:#fff class RUNNING running class FAILED failed class PAUSED paused class STOPPED stopped ``` ## Quick Start Channel supervision is automatically enabled for all gateway channels configured in `gateway.yaml`. No additional setup is required. Create a simple gateway with supervision: ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # gateway.yaml agents: assistant: instructions: "You are a helpful AI assistant." model: "gpt-4o-mini" channels: telegram: token: "${TELEGRAM_BOT_TOKEN}" platform: telegram ``` ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai gateway start --config gateway.yaml ``` The `telegram` channel is now under supervision with unlimited retry capability. Pause a problematic channel while investigating issues: ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai gateway pause telegram ``` Resume when ready: ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai gateway resume telegram ``` Force reconnect to reset error state: ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai gateway reconnect telegram ``` *** ## How It Works Channel supervision provides resilient error handling through error classification and unlimited retries: ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant Operator participant CLI participant Gateway participant Supervisor participant Bot participant Platform Operator->>CLI: pause telegram CLI->>Gateway: POST /api/channels/telegram/pause Gateway->>Supervisor: pause("telegram") Supervisor->>Bot: Signal abort Bot->>Platform: Disconnect Note over Bot: State: PAUSED Operator->>CLI: resume telegram CLI->>Gateway: POST /api/channels/telegram/resume Gateway->>Supervisor: resume("telegram") Supervisor->>Bot: Restart connection Bot->>Platform: Reconnect Note over Bot: State: RUNNING ``` | Component | Responsibility | | ------------------------ | -------------------------------------------------------- | | **ChannelSupervisor** | Manages channel lifecycle and error handling | | **BackoffPolicy** | Controls retry timing with capped exponential backoff | | **Error Classification** | Determines if errors are recoverable, fatal, or conflict | | **Operator Controls** | Provides manual pause/resume/reconnect capabilities | *** ## Channel States The supervision system tracks four distinct channel states: | State | Description | Auto-Retry | Operator Actions | | --------- | ------------------------------------------------------------- | ---------- | ----------------- | | `RUNNING` | Channel is actively connected and serving messages | N/A | pause, reconnect | | `FAILED` | Fatal error occurred (e.g., Telegram conflict, invalid token) | ❌ No | reconnect only | | `PAUSED` | Manually paused by operator | ❌ No | resume, reconnect | | `STOPPED` | Clean shutdown or initial state | ❌ No | Automatic restart | Proactive health monitoring can move a channel from `RUNNING` into a restart cycle without a raised exception β€” for example when transport activity goes stale (`stale-socket`) or a probe fails (`disconnected`). *** ## Proactive Health Monitoring The health monitor periodically asks each channel "Are you really alive?" and restarts the ones that aren't, without waiting for an exception to be raised. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph LR Timer[⏱️ Timer] --> Health[bot.health] Health --> Eval[evaluate_channel_health] Eval --> Reason[HealthReason] Reason -->|recoverable| Restart[supervisor reconnect] Reason -->|healthy| OK[βœ… no action] classDef process fill:#189AB4,stroke:#7C90A0,color:#fff classDef decision fill:#F59E0B,stroke:#7C90A0,color:#fff classDef action fill:#8B0000,stroke:#7C90A0,color:#fff classDef ok fill:#10B981,stroke:#7C90A0,color:#fff class Timer,Health,Eval process class Reason decision class Restart action class OK ok ``` ### Enable via YAML ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} gateway: host: 127.0.0.1 port: 8765 health: interval: 300 # seconds between sweeps (default 300) startup_grace: 60 # grace window after start (default 60) stale_after: 120 # no inbound activity (idle) β†’ stale-socket (default 120) stuck_after: 900 # busy with no progress (inbound OR in-run) β†’ stuck (default 900) max_restarts_per_hour: 10 enabled: true # default true ``` ### Enable via Python ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonai.gateway.health_monitor import HealthMonitorConfig from praisonai.gateway.supervisor import ChannelSupervisor health_config = HealthMonitorConfig(interval=120, stale_after=180) supervisor = ChannelSupervisor(health_config=health_config) ``` ### Configuration Options | Option | Type | Default | Description | | ----------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `interval` | `float` | `300.0` | Seconds between health sweeps | | `startup_grace` | `float` | `60.0` | Seconds after start before checks count | | `stale_after` | `float` | `120.0` | Seconds without **inbound** transport activity (while idle) β†’ `stale-socket` | | `stuck_after` | `float` | `900.0` | Seconds a **busy** channel can go without **any progress signal** (inbound transport activity OR in-run progress β€” tool calls, streamed tokens, emitter events) β†’ `stuck` | | `max_restarts_per_hour` | `int` | `10` | Hard cap on restart attempts per channel per hour | | `enabled` | `bool` | `True` | Whether monitoring is enabled | ### HealthReason values | Reason | Recoverable | When it fires | | --------------- | ----------- | -------------------------------------------------------------------------------------------- | | `healthy` | No | All checks pass | | `not-running` | No | `health.is_running` is false | | `startup-grace` | No | `uptime_seconds < startup_grace` | | `disconnected` | Yes | `health.probe` exists and `probe.ok` is false | | `stale-socket` | Yes | Idle and no **inbound** activity for `stale_after` seconds | | `busy` | **No** | `active_runs > 0` with progress within `stuck_after` β€” protects long runs from mid-run kills | | `stuck` | Yes | `active_runs > 0` and no progress (inbound or in-run) for `> stuck_after` β€” likely wedged | | `error` | Yes | `health.error` set or `health.ok` is false | Decision order in `evaluate_channel_health()`: ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph TB A[not is_running?] -->|yes| NR[not-running] A -->|no| B[uptime < startup_grace?] B -->|yes| SG[startup-grace] B -->|no| C[health.error set?] C -->|yes| E[error] C -->|no| D[probe fails?] D -->|yes| DC[disconnected] D -->|no| R[active_runs > 0?] R -->|yes| RS[idle > stuck_after?] RS -->|yes| ST[stuck] RS -->|no| BS[busy] R -->|no| F[last_activity stale?] F -->|yes| SS[stale-socket] F -->|no| G[not health.ok?] G -->|yes| E2[error] G -->|no| H[healthy] classDef check fill:#F59E0B,stroke:#7C90A0,color:#fff classDef result fill:#189AB4,stroke:#7C90A0,color:#fff class A,B,C,D,F,G,R,RS check class NR,SG,E,DC,SS,E2,H,ST,BS result ``` ### Passive inbound liveness Channel liveness is driven by whether messages are still flowing IN, not just whether the outbound probe (e.g. Telegram `getMe`) succeeds. Every inbound message refreshes the timestamp via `fire_message_received β†’ _note_inbound()`, so a reachable-but-deaf channel will eventually trip `stale-socket` instead of being reported healthy forever. All adapters get this automatically; WhatsApp and Linear are wired explicitly because they bypass the shared handler. ### Run awareness The evaluator knows about in-flight agent turns (`HealthResult.active_runs`) **and tracks per-run progress** (`HealthResult.last_run_progress`, refreshed on tool calls, streamed tokens, and agent emitter events). A busy channel is never killed mid-run β€” `BUSY` is non-recoverable on purpose. Only when a busy channel has made **no progress at all** (neither inbound transport activity nor in-run progress) for `stuck_after` seconds does it escalate to `STUCK` (recoverable). An actively-streaming long agent turn stays `BUSY` indefinitely; a genuinely-hung turn that emits nothing for `stuck_after` is still correctly flagged `STUCK`. Before praisonaiagents 1.6.82, only inbound transport activity counted toward progress, so a single long agent turn (deep research, big refactor, slow model) was classified `STUCK` once it crossed `stuck_after` β€” even while actively streaming. The protocol now honours in-run progress (`HealthResult.last_run_progress`), so progressing long runs stay `BUSY` indefinitely. See [PraisonAI #2400](https://github.com/MervinPraison/PraisonAI/pull/2400). ### Restart guard-rails * **Startup grace** β€” no restarts during the first `startup_grace` seconds after connect * **5-minute cooldown** β€” implicit cooldown after every restart (logged as `restart cooldown active`) * **`max_restarts_per_hour`** β€” when the cap is hit, a warning is logged and the restart is skipped ``` WARNING: channel telegram: max restarts per hour (10) reached, skipping restart ``` ### Suspend / resume monitoring For planned maintenance, suspend health checks on a channel without stopping supervision: ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} monitor.suspend_channel("telegram") # skip health sweeps monitor.resume_channel("telegram") # resume sweeps ``` *** ## Operator Controls ### Pause Channel Temporarily stop a channel without losing configuration: ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai gateway pause telegram --url ws://127.0.0.1:8765 ``` ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} curl -X POST http://127.0.0.1:8765/api/channels/telegram/pause \ -H "Authorization: Bearer YOUR_TOKEN" ``` **Effect**: Channel enters `PAUSED` state and stops processing messages. Supervision loop waits indefinitely until resumed. ### Resume Channel Resume a manually paused channel: ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai gateway resume telegram --url ws://127.0.0.1:8765 ``` ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} curl -X POST http://127.0.0.1:8765/api/channels/telegram/resume \ -H "Authorization: Bearer YOUR_TOKEN" ``` **Effect**: Channel transitions from `PAUSED` to `STOPPED`, then automatically restarts to `RUNNING`. ### Reconnect Channel Force a complete reconnection and reset error state: ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai gateway reconnect telegram --url ws://127.0.0.1:8765 ``` ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} curl -X POST http://127.0.0.1:8765/api/channels/telegram/reconnect \ -H "Authorization: Bearer YOUR_TOKEN" ``` **Effect**: Resets retry counter, clears error history, forces restart. Works from any state including `FAILED`. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph TD Start([Which control?]) --> Question{Channel state?} Question -->|RUNNING, temporary issue| Pause[πŸ‘€ pause] Question -->|FAILED, need reset| Reconnect[πŸ”„ reconnect] Question -->|PAUSED, issue resolved| Resume[▢️ resume] Pause --> PauseDesc[Stop processing
Keep configuration] Resume --> ResumeDesc[Restart from pause
Resume processing] Reconnect --> ReconnectDesc[Full reset
Clear error state] classDef action fill:#189AB4,stroke:#7C90A0,color:#fff classDef desc fill:#6366F1,stroke:#7C90A0,color:#fff class Pause,Resume,Reconnect action class PauseDesc,ResumeDesc,ReconnectDesc desc ``` *** ## Error Classification The supervision system classifies errors to determine retry behavior: | Error Type | Examples | Behavior | Recovery | | ------------------- | ---------------------------------------------------- | ---------------------------------------- | ------------------------------------------- | | **Recoverable** | Network timeouts, DNS failures, temporary API errors | Unlimited retry with exponential backoff | Automatic | | **Conflict** | Telegram "Conflict: terminated by other getUpdates" | Immediate failure, no retry | Manual `reconnect` after stopping duplicate | | **Non-Recoverable** | Invalid bot token, missing permissions | Immediate failure, no retry | Manual `reconnect` after fixing config | The retry policy uses capped exponential backoff: * Initial delay: 5 seconds * Maximum delay: 300 seconds (5 minutes) * Unlimited attempts for recoverable errors * Jitter added to prevent thundering herd *** ## Monitoring via `/health` The enhanced health endpoint includes supervision status for each channel: ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} curl http://127.0.0.1:8765/health ``` ```json theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} { "status": "healthy", "uptime": 3600, "agents": 2, "sessions": 5, "clients": 3, "channels": { "telegram": { "platform": "telegram", "running": true, "last_activity": 1672531500, "active_runs": 2, "supervision": { "state": "running", "last_error": null, "last_error_time": null, "next_retry_at": null, "total_recoveries": 3, "manual_pause": false }, "health_monitor": { "enabled": true, "running": true, "interval": 300, "last_check": 1672531200, "suspended": false, "restart_count": 1, "can_restart": true } }, "discord": { "platform": "discord", "running": false, "supervision": { "state": "failed", "last_error": "Invalid bot token", "last_error_time": 1672531200, "next_retry_at": null, "total_recoveries": 0, "manual_pause": false } } } } ``` Key supervision fields: * `state`: Current channel state (`running`, `failed`, `paused`, `stopped`) * `last_error`: Most recent error message (if any) * `last_error_time`: Unix timestamp of last error * `next_retry_at`: Unix timestamp of next retry attempt (if scheduled) * `total_recoveries`: Count of successful recoveries from errors * `manual_pause`: Whether channel is manually paused by operator * `active_runs`: Number of in-flight agent turns (busy count) * `last_activity`: Unix timestamp of last **inbound** transport activity (used for `stale-socket` and `stuck` detection) * `health_monitor.enabled`: Whether proactive monitoring is active * `health_monitor.restart_count`: Restarts in the current hour * `health_monitor.can_restart`: Whether guard-rails allow another restart *** ## Best Practices Use **pause** for temporary investigations while keeping the channel configuration intact. Use **reconnect** when you need to reset error state after fixing underlying issues like network connectivity or API tokens. High `total_recoveries` counts indicate frequent connection issues. Monitor this metric to identify unstable network conditions or platform-specific problems that may require infrastructure changes. The `/health` endpoint is designed for integration with Prometheus, Datadog, or other monitoring systems. Set up alerts on `state: "failed"` and track `total_recoveries` trends to detect degrading connection quality. Channels in `FAILED` state require manual intervention. Use `reconnect` (not `resume`) to reset the error state and attempt a fresh connection. Always investigate the `last_error` to address root cause issues before reconnecting. Low-traffic channels need a larger `stale_after` (e.g. 600s) to avoid false `stale-socket` restarts. Chatty channels can use the default 120s. If your agent regularly runs turns longer than 15 minutes (deep research, long tool chains), raise `stuck_after` so genuine progress isn't classified as wedged. The default 900s covers most chat and triage workloads. When the upstream API is rate-limited, restart storms make the problem worse. Lower the cap (e.g. 3) so the guard-rail surfaces the issue in logs instead of hammering the API. *** ## Related Suppress permanently-dead channels β€” bot kicked, chat deleted, account deactivated Complete CLI reference for gateway management Error handling strategies for gateway bots Multi-platform orchestrator with the same supervision and health monitoring Break runaway bot-to-bot reply loops β€” a pure decision protocol like `evaluate_channel_health`