Event-Loop Watchdog

LoopWatchdog detects a wedged asyncio loop and hands the process back to your supervisor before the gateway becomes a silent zombie.

Running praisonai gateway start? You don’t need to write any Python — the gateway wires the watchdog for you. Turn it on with any of three opt-in surfaces: add gateway.watchdog: { enabled: true } to gateway.yaml, pass --watchdog on the CLI, or set GatewayConfig(watchdog=True) from Python. See Using it from the gateway below.

Watchdog ladder

The gateway’s liveness coverage is a three-rung ladder — this page owns rungs 2 and 3; StartupWatchdog owns the front rung.

Rung	Class / mode	Covers	Docs
1. Startup	`StartupWatchdog`	Imports, config/secret load, first `connect()` — before the loop exists	Startup Watchdog
2. Running loop	`LoopWatchdog` (probe mode)	The event loop while it serves	Below
3. Shutdown	`LoopWatchdog.arm_deadline`	Wedged teardown after `stop()` / SIGTERM	Shutdown-phase deadline mode

Looking for connection heartbeats? Gateway Liveness reaps silent client connections with PING/PONG frames. This page covers the event loop itself — when the process is up but the loop stops running. Each concern is orthogonal:

Concern	Startup Watchdog	Gateway Liveness	Loop Watchdog
What is dead?	A pre-loop startup step (no loop yet)	An individual client connection	The event loop itself (process is a zombie)
Detected how?	Wall-clock deadline before `confirm_loop_live()`	`PING` / `PONG` frames on the wire	OS-thread probe via `loop.call_soon_threadsafe`
Reaction	Dump stacks and `os._exit(75)` for restart	Close the connection (`LIVENESS_TIMEOUT`)	Dump stacks and `os._exit(75)` for restart
Config surface	Python `arm(deadline_s)` (embedder-only today)	`LivenessConfig` in `gateway.yaml`	`gateway.watchdog.*` YAML, and Python `arm(loop)` / `arm_deadline(deadline_s)`
Runs on	Dedicated OS thread (survives a pre-loop hang)	Gateway reaper task (async)	Dedicated OS thread (survives loop hang)

Using it from the gateway

Most operators run praisonai gateway start — turn the watchdog on with YAML or a CLI flag, no Python required.

YAML (recommended)
CLI flags
Python (via GatewayConfig)
Python (custom runtime)

Add a watchdog block under gateway and start as usual.

# gateway.yaml
gateway:
  watchdog:
    enabled: true
    liveness_interval: 5      # seconds between loop probes
    liveness_strikes: 3       # hard-exit after N consecutive misses (wedge_after ≈ 15s)
    dump_file: /var/log/praisonai/loop-wedge.log   # optional; stderr always gets the dump

praisonai gateway start --config gateway.yaml

The gateway arms the watchdog right before _server.serve() and disarms it in finally and on stop().

Pass --watchdog to enable it for a single run — the flags override the matching YAML fields.

# Enable with defaults (probe every 5s, wedge after ~15s)
praisonai gateway start --config gateway.yaml --watchdog

# Set a specific wedge budget (splits into 3 strikes → probe interval = budget / 3)
praisonai gateway start --config gateway.yaml --watchdog --watchdog-timeout 15

# Works without a YAML file too (no-config WebSocket-only mode)
praisonai gateway start --watchdog

restart replays these flags automatically via the persisted start-flags — you do not re-pass them.

Wiring up the gateway from Python? Turn the same event-loop watchdog on with GatewayConfig — no CLI flag and no YAML block required.

from praisonaiagents.gateway import GatewayConfig

config = GatewayConfig(
    watchdog=True,
    watchdog_timeout=30,   # optional; leave None for the default budget
)
# start the gateway with this config (Python-first)

This enables the same watchdog the YAML gateway.watchdog: block and the --watchdog flag enable. watchdog_timeout mirrors the --watchdog-timeout CLI flag; when set it must be > 0 (ValueError otherwise) and wins over the budget derived from a watchdog: block.

An explicit gateway.watchdog: false in YAML overrides this field and is honoured — a GatewayConfig(watchdog=True) does not silently re-enable it (fail-closed for operator intent).

Embedding your own asyncio runtime? Arm the watchdog on the loop directly.

import asyncio
from praisonaiagents import Agent
from praisonaiagents.gateway import LoopWatchdog

agent = Agent(name="Support", instructions="Reply to users.")

async def serve():
    # ... your long-lived gateway/serving loop here ...
    await asyncio.sleep(3600)

loop = asyncio.new_event_loop()
watchdog = LoopWatchdog()          # probe every 5s, wedge after 3 misses (~15s)
watchdog.arm(loop)
try:
    loop.run_until_complete(serve())
finally:
    watchdog.disarm()

Quick Start (custom runtime)

Simple usage (defaults)

Build an agent, then arm the watchdog on the loop that serves it.

import asyncio
from praisonaiagents import Agent
from praisonaiagents.gateway import LoopWatchdog

agent = Agent(name="Support", instructions="Reply to users.")

async def serve():
    # ... your long-lived gateway/serving loop here ...
    await asyncio.sleep(3600)

loop = asyncio.new_event_loop()
watchdog = LoopWatchdog()          # probe every 5s, wedge after 3 misses (~15s)
watchdog.arm(loop)
try:
    loop.run_until_complete(serve())
finally:
    watchdog.disarm()

Tune the cadence

Pass a LoopWatchdogPolicy to change how fast a wedge is caught.

from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy

watchdog = LoopWatchdog(LoopWatchdogPolicy(
    probe_interval_s=2.0,               # probe every 2s
    missed_probes_before_wedged=5,      # wedge only after ~10s of silence
    dump_file="/var/log/praisonai/loop-wedge.log",
))

Observe without exiting (dev / staging)

Use dump_only to catch a hang without killing the process.

from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy

watchdog = LoopWatchdog(LoopWatchdogPolicy(on_wedge="dump_only"))
watchdog.arm(loop)
# ... later ...
if watchdog.wedged:
    print("loop hung at least once; check the stack dump on stderr")

How It Works

A dedicated OS thread probes the loop, so it keeps working precisely when the loop does not.

Piece	Owner	Role
`LoopWatchdogPolicy`	Core (pure)	Configuration: cadence, threshold, on-wedge action, exit code, dump file
`LoopWatchdog.arm(loop)`	Embedder call	Spawns daemon OS thread `praisonai-loop-watchdog`
Watchdog thread	Runs off-loop	Schedules `_ack` via `call_soon_threadsafe` every interval
`faulthandler.dump_traceback(all_threads=True)`	Stdlib	All-thread stack dump on wedge
`os._exit(75)`	Stdlib	Bypasses `Py_FinalizeEx` — supervisor sees `EX_TEMPFAIL`
`GATEWAY_RESTART_EXIT_CODE`	Protocol	The `75` constant, shared with the restart-intent codes

The watchdog uses os._exit, not sys.exit, on a confirmed wedge. Normal interpreter shutdown runs Py_FinalizeEx, which would itself hang trying to join the wedged loop thread — so os._exit hands the process straight back to the supervisor, which restarts it cleanly.

Health signal

When the watchdog is configured, GET /health (or gateway.health() in-process) includes a watchdog block.

Field	Type	Description
`watchdog.enabled`	`bool`	Always `true` when the block is present (implied by the block existing at all).
`watchdog.armed`	`bool`	`true` while `_server.serve()` is running; flips to `false` on graceful `stop()`.
`watchdog.wedge_after_s`	`float`	`liveness_interval * liveness_strikes` — the effective wedge budget.

The block is omitted entirely from health() when the watchdog isn’t configured — an always-on gateway sees no shape change.

The watchdog’s last_lag_ms property also feeds pressure.event_loop_lag_p99_ms on the Pressure Telemetry health block. Note the lag anchor pins to the oldest still-unacked probe, so under a multi-interval stall the reported lag reflects the full stall, not just the gap since the newest probe.

Shutdown-phase deadline mode

arm_deadline(seconds) fires on a wall-clock overrun independent of loop responsiveness, so a wedged teardown on a still-alive loop still gets an all-thread stack dump + bounded os._exit(75) instead of an external SIGKILL. Deadline mode is a sibling of the loop-probe mode — they cover different failure shapes.

Concern	Loop-probe mode	Shutdown-phase deadline mode
What is dead?	The event loop itself (process is a zombie)	A teardown step while the loop is still responsive
Detected how?	Missed OS-thread probes on the loop	Wall-clock overrun of `drain_timeout + grace`
Armed by	`arm(loop)` in the embedder / `_server.serve()` in the gateway	`arm_deadline(deadline_s)` in `stop()` and in the supervisor-driven teardown `finally`
Disarmed by	`disarm()` after `serve()` returns / in `stop()`	`cancel()` in `stop()`’s `finally` — clean stops never self-exit
Runs on	Dedicated `praisonai-loop-watchdog` daemon thread	Dedicated `praisonai-shutdown-watchdog` daemon thread

YAML (recommended)
Python stop(grace=…)
Custom runtime arm_deadline

Add a shutdown_grace key to the same gateway.watchdog block. This is the extra budget added on top of drain_timeout before the deadline fires.

# gateway.yaml
gateway:
  watchdog:
    enabled: true
    liveness_interval: 5
    liveness_strikes: 3
    shutdown_grace: 5      # seconds beyond drain_timeout before the deadline fires
    dump_file: /var/log/praisonai/loop-wedge.log

If drain_timeout: 10 (see Graceful Drain) and shutdown_grace: 5, the deadline fires ~15 seconds into a wedged shutdown — well before systemd’s default TimeoutStopSec=90 reaches SIGKILL.

stop() accepts a per-call grace override that wins over policy.shutdown_grace_s.

import asyncio
from praisonai_bot.gateway.server import Gateway

gateway = Gateway.from_yaml("gateway.yaml")
asyncio.create_task(gateway.serve())

# ... later, on shutdown:
await gateway.stop(drain_timeout=30, grace=10)
# → the deadline watchdog fires at ~40s if any teardown step is wedged

Passing grace=None (the default) uses policy.shutdown_grace_s — set once in gateway.yaml and stops don’t need to specify it.

Embedding your own asyncio runtime? Wrap your shutdown in an arm_deadline / cancel pair.

import asyncio
from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy

watchdog = LoopWatchdog(LoopWatchdogPolicy(shutdown_grace_s=5.0))
watchdog.arm(loop)  # loop-probe mode during serving

async def shutdown(drain_timeout=10.0):
    watchdog.arm_deadline(drain_timeout + watchdog.policy.shutdown_grace_s)
    try:
        await ordered_teardown()      # your adapter.disconnect(), WS drain, DB close, ...
    finally:
        watchdog.cancel()             # clean stops never self-exit

# ...
loop.run_until_complete(shutdown())
watchdog.disarm()

Both modes share the same _dump_stacks + os._exit(exit_code) machinery and the same on_wedge="dump_only" diagnostic override.

Added in PraisonAI PR #5084 (issue #5079). Before this PR, the loop-probe watchdog was — correctly, per #3410 — disarmed the moment a deliberate stop() started, so a wedged teardown on a still-alive loop hung until systemd’s TimeoutStopSec fired SIGKILL, with no stack dump. Deadline mode closes that gap: the wrapper arms drain_timeout + shutdown_grace at the start of the stop path (and around the SIGTERM/SIGINT-driven teardown), cancels on clean completion, and self-exits with a diagnostic dump if the deadline elapses. It is fully opt-in and no-op when a watchdog is unconfigured; a cancel racing an in-flight expiry suppresses the destructive os._exit, and a stale generation after re-arm stays inert.

Configuration Options

LoopWatchdogPolicy holds every knob.

Option	Type	Default	Description
`probe_interval_s`	`float`	`5.0`	Seconds between liveness probes scheduled onto the loop. Also the cadence at which the watchdog thread wakes. Must be finite and `> 0` (raises `ValueError` on `0`, negative, `NaN`, or `inf`).
`missed_probes_before_wedged`	`int`	`3`	Number of consecutive probes that must fail to round-trip within their budget before the loop is declared wedged. Must be `>= 1`.
`on_wedge`	`"dump_and_exit"` \| `"dump_only"`	`"dump_and_exit"`	What to do once a wedge is confirmed. `"dump_and_exit"` dumps all-thread stacks then calls `os._exit(exit_code)`; `"dump_only"` dumps stacks but leaves the process running.
`exit_code`	`int`	`75` (`GATEWAY_RESTART_EXIT_CODE`)	Exit code used when `on_wedge == "dump_and_exit"`. Defaults to EX_TEMPFAIL so a supervisor restarts the process.
`dump_file`	`Optional[str]`	`None`	Optional path to also write the stack dump to (appended, line-buffered). `None` writes to stderr only.
`shutdown_grace_s`	`float`	`5.0`	Extra budget (seconds) added beyond a caller’s `drain_timeout` when arming the shutdown-phase deadline via `arm_deadline`. Must be finite and `>= 0` (zero is allowed; negative, `NaN`, `inf` raise `ValueError`). Only consulted by shutdown-phase deadline mode; the loop-probe mode ignores it.

Computed property:

Property	Returns	Description
`wedge_after_s`	`float`	`probe_interval_s * missed_probes_before_wedged` — approximate stall before a wedge is declared (defaults: `~15s`).

LoopWatchdog exposes a small surface:

Method / property	Description
`LoopWatchdog(policy=None)`	Construct with an explicit `LoopWatchdogPolicy` or accept the defaults.
`arm(loop)`	Start watching `loop` from a daemon OS thread named `praisonai-loop-watchdog`. Idempotent and fail-open.
`disarm()`	Stop watching. Safe to call multiple times / when not armed. Joins the worker (best-effort) with a `probe_interval_s + 1.0s` timeout.
`armed`	`True` iff the worker thread is alive.
`wedged`	`True` once a wedge has been confirmed (mainly useful for `dump_only`).
`arm_deadline(deadline_s)`	Start a wall-clock shutdown deadline on a dedicated daemon thread (`praisonai-shutdown-watchdog`). Non-positive / non-finite `deadline_s` is a silent no-op. Idempotent while armed. See Shutdown-phase deadline mode.
`cancel()`	Cancel an armed shutdown deadline. Sets the current generation’s stop event (suppressing an in-flight expiry even mid-dump). Safe when not armed.
`deadline_armed`	`True` iff a shutdown-phase deadline is currently armed.
`deadline_expired`	`True` once a shutdown deadline actually fired (mainly useful for `on_wedge="dump_only"` diagnostics). Cancelled generations do not flip this.

YAML config keys

The gateway.watchdog block is a thin façade over LoopWatchdogPolicy for gateway operators.

YAML key (under `gateway.watchdog.*`)	Type	Default	Maps to	Description
`enabled`	`bool` \| `"true"`/`"yes"`/`"on"`	`false`	—	Master switch. Omit or set to `false` and the gateway does not create a watchdog at all — zero cost.
`liveness_interval`	`float`	`5.0`	`LoopWatchdogPolicy.probe_interval_s`	Seconds between probes scheduled onto the loop.
`liveness_strikes`	`int`	`3`	`LoopWatchdogPolicy.missed_probes_before_wedged`	Consecutive missed probes before a wedge is declared.
`dump_file`	`str` \| `null`	`null`	`LoopWatchdogPolicy.dump_file`	Optional path to also write the stack dump to. `null` writes to stderr only.
`shutdown_grace`	`float`	`5.0`	`LoopWatchdogPolicy.shutdown_grace_s`	Extra budget (seconds) beyond `drain_timeout` before the shutdown-phase deadline fires. Only active when the wrapper’s stop path or supervisor-driven teardown runs.

An invalid value (for example liveness_interval: 0) fails closed: the gateway logs a warning and runs without a watchdog.

CLI flag reference

--watchdog — enable the watchdog for this run (sets/overrides gateway.watchdog.enabled: true).
--watchdog-timeout <seconds> — total wedge budget in seconds. Derives liveness_interval = budget / liveness_strikes (keeps the default 3 strikes unless YAML overrides). --watchdog-timeout 15 → probe every 5s. --watchdog-timeout 6 → probe every 2s.

Precedence

Highest wins: --watchdog[-timeout] CLI flag → explicit gateway.watchdog.* YAML (block or scalar, including watchdog: false) → GatewayConfig(watchdog=…, watchdog_timeout=…) field → off. Missing values fall back down the chain — an unset CLI flag does not clobber enabled: true in YAML, and an explicit gateway.watchdog: false is not re-enabled by the GatewayConfig field.

Choosing a cadence

Match the probe interval to how fast a wedge must be caught. Shorter intervals bound recovery time but cost a scheduling roundtrip every N seconds; longer intervals reduce noise and false alerts under GC pauses.

A long stop-the-world GC or a big synchronous blocking call inside a coroutine looks like a wedge if it exceeds wedge_after_s. If you legitimately do 20-second sync work on the loop, either raise wedge_after_s above it or — better — push the work off the loop with run_in_executor / asyncio.to_thread.

faulthandler writes one per-thread traceback on a wedge. Read the loop thread first (typically MainThread or the asyncio worker) — that’s where the actual hang is. Other threads may just be blocked waiting on that one.

Fail-open guarantees

A misbehaving watchdog must never kill a healthy gateway.

arm() while already armed is a no-op.
disarm() when not armed is safe.
Any exception inside probe scheduling, dumping, or exiting is swallowed.
A closed / stopped loop is not a wedge — scheduling onto it is a normal shutdown, so the missed counter resets and the watchdog keeps failing open.
A disarm() racing an in-flight wedge suppresses the destructive os._exit — a deliberate shutdown never triggers a supervisor restart.

Common Patterns

Three ways teams wire the watchdog. Long-lived embedder (production): keep the defaults and wire the process to your existing supervisor (systemd / Kubernetes / pm2). EX_TEMPFAIL (75) is the same exit code the rest of the restart-intent protocol uses, so any supervisor that already restarts the gateway handles this too.

from praisonaiagents.gateway import LoopWatchdog

watchdog = LoopWatchdog()          # defaults trip at ~15s
watchdog.arm(loop)

Diagnostic-only mode (staging): catch a hang without killing the process, inspect the dump, then flip to dump_and_exit in production.

from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy

watchdog = LoopWatchdog(LoopWatchdogPolicy(
    on_wedge="dump_only",
    dump_file="/var/log/praisonai/loop-wedge.log",
))
watchdog.arm(loop)

Custom exit code for a bespoke supervisor: override exit_code if you can’t use 75.

from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy

watchdog = LoopWatchdog(LoopWatchdogPolicy(exit_code=42))
watchdog.arm(loop)

Best Practices

Turn it on wherever a supervisor can restart the process

systemd’s Restart=on-failure, launchd’s KeepAlive, Docker’s restart: unless-stopped, and Kubernetes all handle exit code 75 correctly. If your supervisor doesn’t, the watchdog will still dump stacks but the process won’t come back — either fix the supervisor or use on_wedge="dump_only" for diagnostics.

Prefer YAML over CLI flags for permanent deployments

CLI flags survive restart via persisted start-flags, but a colleague reading gateway.yaml learns the watchdog is on. The YAML block is the discoverable configuration surface.

Only arm in long-lived processes

Short-lived CLI runs don’t need the watchdog — the process exits anyway. Arm it around gateways, bots, and other loops that must stay up for hours.

Always pair arm(loop) with disarm() in finally

The watchdog is a daemon thread and never blocks shutdown, but disarm() also joins it cleanly and prevents any race with a shutdown-time wedge.

Do not shorten the interval below your longest sync block

A 100ms interval false-positives under GC or big JSON serialisation. Keep probe_interval_s above your longest expected synchronous block on the loop.

Make sure your supervisor restarts on EX_TEMPFAIL

systemd (Restart=on-failure) restarts on 75; a naive nohup does not. See Exit Codes for the full restart-intent contract.

Startup Watchdog

Rung 1: guard the pre-loop startup window before this watchdog can take over.

Gateway Liveness

Connection-level heartbeats that reap silent clients (disambiguation).

Exit Codes

The shared EX_TEMPFAIL (75) restart-intent protocol.

Forensics

What to keep on unhealthy exits.

Crash-Loop Guard

Guard against restart storms.

Restart Continuation

Re-drive interrupted turns after the restart.

Graceful Drain

Bound in-flight turns on shutdown — the drain window the deadline mode wraps.

​Watchdog ladder

​Using it from the gateway

​Quick Start (custom runtime)

​How It Works

​Health signal

​Shutdown-phase deadline mode

​Configuration Options

​YAML config keys

​CLI flag reference

​Precedence

​Choosing a cadence

​Fail-open guarantees

​Common Patterns

​Best Practices

​Related

Startup Watchdog

Gateway Liveness

Exit Codes

Forensics

Crash-Loop Guard

Restart Continuation

Graceful Drain

Watchdog ladder

Using it from the gateway

Quick Start (custom runtime)

How It Works

Health signal

Shutdown-phase deadline mode

Configuration Options

YAML config keys

CLI flag reference

Precedence

Choosing a cadence

Fail-open guarantees

Common Patterns

Best Practices

Related