Event-Loop Watchdog

Detect and self-recover from a wedged asyncio loop in long-lived gateways

LoopWatchdog detects a wedged asyncio loop and hands the process back to your supervisor before the gateway becomes a silent zombie.

Watchdog ladder

The gateway’s liveness coverage is a three-rung ladder — this page owns rungs 2 and 3; StartupWatchdog owns the front rung.

Using it from the gateway

Most operators run praisonai gateway start — turn the watchdog on with YAML or a CLI flag, no Python required.

# gateway.yaml
gateway:
  watchdog:
    enabled: true
    liveness_interval: 5      # seconds between loop probes
    liveness_strikes: 3       # hard-exit after N consecutive misses (wedge_after ≈ 15s)
    dump_file: /var/log/praisonai/loop-wedge.log   # optional; stderr always gets the dump

praisonai gateway start --config gateway.yaml

# Enable with defaults (probe every 5s, wedge after ~15s)
praisonai gateway start --config gateway.yaml --watchdog

# Set a specific wedge budget (splits into 3 strikes → probe interval = budget / 3)
praisonai gateway start --config gateway.yaml --watchdog --watchdog-timeout 15

# Works without a YAML file too (no-config WebSocket-only mode)
praisonai gateway start --watchdog

from praisonaiagents.gateway import GatewayConfig

config = GatewayConfig(
    watchdog=True,
    watchdog_timeout=30,   # optional; leave None for the default budget
)
# start the gateway with this config (Python-first)

import asyncio
from praisonaiagents import Agent
from praisonaiagents.gateway import LoopWatchdog

agent = Agent(name="Support", instructions="Reply to users.")

async def serve():
    # ... your long-lived gateway/serving loop here ...
    await asyncio.sleep(3600)

loop = asyncio.new_event_loop()
watchdog = LoopWatchdog()          # probe every 5s, wedge after 3 misses (~15s)
watchdog.arm(loop)
try:
    loop.run_until_complete(serve())
finally:
    watchdog.disarm()

Quick Start (custom runtime)

1. Simple usage (defaults)

   Build an agent, then arm the watchdog on the loop that serves it.

   import asyncio
   from praisonaiagents import Agent
   from praisonaiagents.gateway import LoopWatchdog
   
   agent = Agent(name="Support", instructions="Reply to users.")
   
   async def serve():
       # ... your long-lived gateway/serving loop here ...
       await asyncio.sleep(3600)
   
   loop = asyncio.new_event_loop()
   watchdog = LoopWatchdog()          # probe every 5s, wedge after 3 misses (~15s)
   watchdog.arm(loop)
   try:
       loop.run_until_complete(serve())
   finally:
       watchdog.disarm()

2. Tune the cadence

   Pass a LoopWatchdogPolicy to change how fast a wedge is caught.

   from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy
   
   watchdog = LoopWatchdog(LoopWatchdogPolicy(
       probe_interval_s=2.0,               # probe every 2s
       missed_probes_before_wedged=5,      # wedge only after ~10s of silence
       dump_file="/var/log/praisonai/loop-wedge.log",
   ))

3. Observe without exiting (dev / staging)

   Use dump_only to catch a hang without killing the process.

   from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy
   
   watchdog = LoopWatchdog(LoopWatchdogPolicy(on_wedge="dump_only"))
   watchdog.arm(loop)
   # ... later ...
   if watchdog.wedged:
       print("loop hung at least once; check the stack dump on stderr")

How It Works

A dedicated OS thread probes the loop, so it keeps working precisely when the loop does not.

sequenceDiagram
    participant Sup as Supervisor
    participant App as Embedder
    participant WD as LoopWatchdog<br/>(OS thread)
    participant Loop as asyncio loop

    App->>WD: arm(loop)
    loop every probe_interval_s
        WD->>Loop: call_soon_threadsafe(_ack)
        alt loop is healthy
            Loop-->>WD: _ack() runs
            Note over WD: missed = 0
        else loop is wedged
            Note over Loop: _ack never runs
            Note over WD: missed += 1
        end
    end
    Note over WD: missed >= threshold
    WD->>WD: faulthandler.dump_traceback(all_threads)
    WD->>Sup: os._exit(75)
    Sup-->>App: restart process

The watchdog uses os._exit, not sys.exit, on a confirmed wedge. Normal interpreter shutdown runs Py_FinalizeEx, which would itself hang trying to join the wedged loop thread — so os._exit hands the process straight back to the supervisor, which restarts it cleanly.

Health signal

When the watchdog is configured, GET /health (or gateway.health() in-process) includes a watchdog block.

The block is omitted entirely from health() when the watchdog isn’t configured — an always-on gateway sees no shape change.

Shutdown-phase deadline mode

arm_deadline(seconds) fires on a wall-clock overrun independent of loop responsiveness, so a wedged teardown on a still-alive loop still gets an all-thread stack dump + bounded os._exit(75) instead of an external SIGKILL.

graph LR
    subgraph "Shutdown-phase deadline"
        Stop[🛑 stop() / SIGTERM] --> Arm[⏱️ arm_deadline<br/>drain_timeout + grace]
        Arm --> Drain[🧹 ordered teardown]
        Drain --> Done{Clean by<br/>deadline?}
        Done -->|Yes| Cancel[✅ cancel — no exit]
        Done -->|No| Dump[🚨 dump all-thread stacks]
        Dump --> Exit[💥 os._exit(75)]
        Exit --> Sup[👷 supervisor restart]
    end

    classDef signal fill:#8B0000,stroke:#7C90A0,color:#fff
    classDef process fill:#189AB4,stroke:#7C90A0,color:#fff
    classDef decision fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef success fill:#10B981,stroke:#7C90A0,color:#fff
    classDef warn fill:#6366F1,stroke:#7C90A0,color:#fff

    class Stop signal
    class Arm,Drain process
    class Done decision
    class Cancel success
    class Dump,Exit warn
    class Sup success

Deadline mode is a sibling of the loop-probe mode — they cover different failure shapes.

# gateway.yaml
gateway:
  watchdog:
    enabled: true
    liveness_interval: 5
    liveness_strikes: 3
    shutdown_grace: 5      # seconds beyond drain_timeout before the deadline fires
    dump_file: /var/log/praisonai/loop-wedge.log

import asyncio
from praisonai_bot.gateway.server import Gateway

gateway = Gateway.from_yaml("gateway.yaml")
asyncio.create_task(gateway.serve())

# ... later, on shutdown:
await gateway.stop(drain_timeout=30, grace=10)
# → the deadline watchdog fires at ~40s if any teardown step is wedged

import asyncio
from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy

watchdog = LoopWatchdog(LoopWatchdogPolicy(shutdown_grace_s=5.0))
watchdog.arm(loop)  # loop-probe mode during serving

async def shutdown(drain_timeout=10.0):
    watchdog.arm_deadline(drain_timeout + watchdog.policy.shutdown_grace_s)
    try:
        await ordered_teardown()      # your adapter.disconnect(), WS drain, DB close, ...
    finally:
        watchdog.cancel()             # clean stops never self-exit

# ...
loop.run_until_complete(shutdown())
watchdog.disarm()

Configuration Options

LoopWatchdogPolicy holds every knob.

Computed property:

LoopWatchdog exposes a small surface:

YAML config keys

The gateway.watchdog block is a thin façade over LoopWatchdogPolicy for gateway operators.

An invalid value (for example liveness_interval: 0) fails closed: the gateway logs a warning and runs without a watchdog.

CLI flag reference

• --watchdog — enable the watchdog for this run (sets/overrides gateway.watchdog.enabled: true).
• --watchdog-timeout <seconds> — total wedge budget in seconds. Derives liveness_interval = budget / liveness_strikes (keeps the default 3 strikes unless YAML overrides). --watchdog-timeout 15 → probe every 5s. --watchdog-timeout 6 → probe every 2s.

Precedence

Highest wins: --watchdog[-timeout] CLI flag → explicit gateway.watchdog.* YAML (block or scalar, including watchdog: false) → GatewayConfig(watchdog=…, watchdog_timeout=…) field → off. Missing values fall back down the chain — an unset CLI flag does not clobber enabled: true in YAML, and an explicit gateway.watchdog: false is not re-enabled by the GatewayConfig field.

Choosing a cadence

Match the probe interval to how fast a wedge must be caught.

Shorter intervals bound recovery time but cost a scheduling roundtrip every N seconds; longer intervals reduce noise and false alerts under GC pauses.

Fail-open guarantees

A misbehaving watchdog must never kill a healthy gateway.

• arm() while already armed is a no-op.
• disarm() when not armed is safe.
• Any exception inside probe scheduling, dumping, or exiting is swallowed.
• A closed / stopped loop is not a wedge — scheduling onto it is a normal shutdown, so the missed counter resets and the watchdog keeps failing open.
• A disarm() racing an in-flight wedge suppresses the destructive os._exit — a deliberate shutdown never triggers a supervisor restart.

Common Patterns

Three ways teams wire the watchdog.

Long-lived embedder (production): keep the defaults and wire the process to your existing supervisor (systemd / Kubernetes / pm2). EX_TEMPFAIL (75) is the same exit code the rest of the restart-intent protocol uses, so any supervisor that already restarts the gateway handles this too.

from praisonaiagents.gateway import LoopWatchdog

watchdog = LoopWatchdog()          # defaults trip at ~15s
watchdog.arm(loop)

Diagnostic-only mode (staging): catch a hang without killing the process, inspect the dump, then flip to dump_and_exit in production.

from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy

watchdog = LoopWatchdog(LoopWatchdogPolicy(
    on_wedge="dump_only",
    dump_file="/var/log/praisonai/loop-wedge.log",
))
watchdog.arm(loop)

Custom exit code for a bespoke supervisor: override exit_code if you can’t use 75.

from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy

watchdog = LoopWatchdog(LoopWatchdogPolicy(exit_code=42))
watchdog.arm(loop)

Best Practices

Related