Gateway Startup Watchdog

Detect and self-recover from a wedged pre-loop startup — before the event-loop watchdog can take over

StartupWatchdog guards the window between process entry and a live event loop, so a hanging import, a blocking credential mint, or a deadlocked connect() never becomes a silent zombie.

Watchdog ladder

StartupWatchdog is the front rung of a three-rung ladder — it covers startup, LoopWatchdog covers the running loop and the shutdown drain.

Quick Start

1. Arm before heavy startup, confirm when the loop is live

   Arm the deadline before the first heavy import, then disarm the instant the loop begins serving.

   import asyncio
   from praisonaiagents.gateway import StartupWatchdog, LoopWatchdog
   
   sw = StartupWatchdog()
   sw.arm(deadline_s=30.0)          # trip if startup wedges for 30s
   
   # ... heavy imports, config/secret load, initial adapter connect() ...
   
   loop = asyncio.new_event_loop()
   LoopWatchdog().arm(loop)
   sw.confirm_loop_live()           # disarm; loop watchdog now owns liveness
   loop.run_forever()

2. Extend the deadline across slow-boot phases

   A legitimately slow boot reports progress at each phase, so it earns a fresh window instead of tripping.

   from praisonaiagents.gateway import StartupWatchdog
   
   sw = StartupWatchdog()
   sw.arm(deadline_s=15.0)
   
   load_config()
   sw.report_startup_progress("config loaded")     # extends the deadline
   
   connect_adapters()
   sw.report_startup_progress("adapters connected") # extends again (bounded)
   
   sw.confirm_loop_live()

3. Observe without exiting (dev / staging)

   Use on_expire="dump_only" to capture a stack dump on a wedge without killing the process.

   from praisonaiagents.gateway import StartupWatchdog
   
   sw = StartupWatchdog(
       on_expire="dump_only",
       dump_file="/var/log/praisonai/startup-wedge.log",
   )
   sw.arm(deadline_s=10.0)
   # ... startup ...
   if sw.expired:
       print("startup hung at least once; check the stack dump")

How It Works

A dedicated daemon OS thread counts down a wall-clock deadline, so it keeps working precisely when the main thread is wedged.

The watchdog is stdlib-only (threading, time, os, sys, math, faulthandler) — it has to run when the code it guards cannot, so its correctness never depends on the imports it protects.

On expiry the stack-dump header reads:

praisonai-startup-watchdog: gateway startup did not confirm a live event loop within ~Ns; dumping all-thread stacks

This is distinct from both the loop-wedge header and the shutdown-phase header, so a single glance at the log tells you which rung fired. See Exit Codes for the shared 75 contract.

Configuration Options

The constructor holds every knob; arm() takes the deadline.

arm(*, deadline_s: float, dump_stacks: bool = True):

StartupWatchdog exposes a small surface:

Common Patterns

Three ways teams wire the startup watchdog.

Minimal embedder (production): arm before heavy startup, confirm when the loop is live, and wire the process to your existing supervisor. EX_TEMPFAIL (75) is the same code the rest of the restart-intent protocol uses.

import asyncio
from praisonaiagents.gateway import StartupWatchdog, LoopWatchdog

sw = StartupWatchdog()
sw.arm(deadline_s=30.0)
# ... imports, config, connect() ...
loop = asyncio.new_event_loop()
LoopWatchdog().arm(loop)
sw.confirm_loop_live()
loop.run_forever()

Slow boot with progress reports: many adapters or cold DNS make a healthy boot take longer than a single window — report progress at every phase so the deadline extends.

from praisonaiagents.gateway import StartupWatchdog

sw = StartupWatchdog(max_extensions=20)   # generous for a slow cold start
sw.arm(deadline_s=10.0)
for adapter in adapters:
    adapter.connect()
    sw.report_startup_progress(f"connected {adapter.name}")
sw.confirm_loop_live()

Dump-only staging: catch a hang without killing the process, inspect the dump, then flip to dump_and_exit in production.

from praisonaiagents.gateway import StartupWatchdog

sw = StartupWatchdog(on_expire="dump_only", dump_file="/var/log/praisonai/startup-wedge.log")
sw.arm(deadline_s=10.0)

Fail-open guarantees

A misbehaving watchdog must never kill a healthy startup.

• arm() while already armed is a no-op.
• arm() with deadline_s <= 0, NaN, or inf is a silent no-op — nothing to guard.
• report_startup_progress() when not armed is a no-op.
• confirm_loop_live() when not armed is safe.
• A confirm_loop_live() racing an in-flight expiry — including during the pre-os._exit stderr flush — suppresses the destructive exit, so a legitimate late confirm never terminates a healthy startup.
• Any exception inside the watchdog is swallowed.

Best Practices

Related