Gateway Crash Forensics

Capture why a gateway died — OOM, supervisor SIGKILL, or parent death — without blocking shutdown

When the gateway dies unexpectedly, forensics writes one log line and one diagnostic file so the next boot can tell you why.

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users on chat channels.")

BotOS(agent=agent, platforms=["telegram"]).start()

The user keeps chatting until SIGTERM or OOM; forensics captures a snapshot line and a detached diagnostic file for the next operator review.

Quick Start

1. Default behaviour — nothing to configure

   Forensics is on by default. Start the gateway normally:

   praisonai gateway start --config gateway.yaml

   The gateway automatically creates ~/.praisonai/gateway/forensics/ at startup and registers signal handlers. No extra steps needed.

2. On the next crash, check the log line

   After a shutdown or crash, search your logs for the gateway-forensics: marker:

   WARNING  gateway-forensics: signal=SIGTERM pid=1234 ppid=1 supervised=yes loadavg_1m=0.50 traced=no maxrss_kb=99000

   This single line tells you the signal, whether the process was supervised, system load at crash time, and peak memory use.

3. Read the detached diagnostic file

   The gateway also writes a full diagnostic report that survives a SIGKILL of the process group (started with start_new_session=True):

   ls ~/.praisonai/gateway/forensics/
   # gateway-forensics-1234.log
   
   cat ~/.praisonai/gateway/forensics/gateway-forensics-1234.log
   # # gateway forensics ctx: {...}
   # ## process tree
   # ...
   # ## load
   # 0.50 0.40 0.30 2/123 4567
   # ## recent kernel OOM/killed
   # [Mon Jun 30 ...] Killed process 1234 (python) ...

How It Works

The gateway sets should_exit = True first, so draining always begins even if the forensic snapshot takes time under memory pressure. The snapshot and diagnostic are fire-and-forget — they never gate the drain.

What ends up in the log line

Any field that can’t be read is silently omitted from the line.

What ends up in the diagnostic file

Configuration Options

# gateway.yaml
gateway:
  forensics:
    enabled: true                                       # default true
    diagnostic_dir: ~/.praisonai/gateway/forensics      # default
    stop_timeout: 90                                    # optional; only used for the startup warning

Environment variable:

The startup warning

If stop_timeout (from config or PRAISONAI_STOP_TIMEOUT) is less than drain_timeout + 30s, the gateway logs once at startup:

WARNING  Supervisor stop-timeout (40s) < drain_timeout (30s) + headroom; gateway may be killed mid-drain with no explanation.

To silence it: raise your supervisor’s stop-timeout to at least drain_timeout + 30s, or set gateway.forensics.stop_timeout to match your real supervisor config.

Disabling

gateway:
  forensics:
    enabled: false

Set enabled: "false" when substituting from an environment variable. The log-line snapshot is suppressed too — there is no separate switch.

Common Patterns

systemd unit

# /etc/systemd/system/praisonai-gateway.service
[Service]
TimeoutStopSec=120
Environment=PRAISONAI_STOP_TIMEOUT=120
ExecStart=praisonai gateway start --config /etc/praisonai/gateway.yaml

Set TimeoutStopSec to at least drain_timeout + 30s. Export PRAISONAI_STOP_TIMEOUT so the startup warning matches reality.

Kubernetes

spec:
  terminationGracePeriodSeconds: 120
  containers:
    - name: gateway
      env:
        - name: PRAISONAI_STOP_TIMEOUT
          value: "120"
      volumeMounts:
        - name: forensics
          mountPath: /home/app/.praisonai/gateway/forensics
  volumes:
    - name: forensics
      persistentVolumeClaim:
        claimName: gateway-forensics-pvc

Mount a persistent volume at diagnostic_dir so reports survive pod termination.

Docker (foreground)

docker run \
  -v /host/forensics:/root/.praisonai/gateway/forensics \
  praisonai gateway start --config /config/gateway.yaml

Bind-mount diagnostic_dir onto the host for inspection after container exit.

Programmatic Use

Operators wiring custom supervision can use the pure helpers and the protocol directly:

from praisonai.gateway import (
    ShutdownForensicsProtocol,
    format_forensics_for_log,
    is_supervised,
    drain_timeout_has_headroom,
)

assert is_supervised(ppid=1, invocation_id=None) is True
assert drain_timeout_has_headroom(stop_timeout_s=90, drain_timeout_s=30) is True

class MyForensics:
    def snapshot(self, signal_name=None): ...
    def spawn_diagnostic(self, ctx, log_dir): ...

assert isinstance(MyForensics(), ShutdownForensicsProtocol)

format_forensics_for_log(ctx) renders a snapshot dict as a single log line prefixed with gateway-forensics:. Grep this marker to wire restart-cause dashboards.

Best Practices

grep "gateway-forensics:" /var/log/praisonai/gateway.log

Related