Redis Pub/Sub Outage Handling

Multi-instance gateway HA transport reconnects, surfaces outages, and counts dropped writes

The Redis pub/sub adapter powering multi-instance gateway fan-out reconnects on a dropped connection, re-subscribes every channel, and reports the outage through gateway status / doctor — so a cross-instance blackout is never silent.

When Redis drops, the adapter marks itself degraded, reconnects with bounded backoff, re-subscribes every channel, and clears the degraded record on recovery — while counting any writes that hit the disconnected transport.

Quick Start

1. Run an agent behind an HA gateway

   The agent code is unchanged — the resilience lives in the fan-out transport:

   from praisonaiagents import Agent
   
   agent = Agent(
       name="alerts-agent",
       instructions="Summarise incoming alert events in one sentence",
   )
   agent.start("Ready to fan out across gateway instances")

2. Enable Redis fan-out in gateway.yaml

   Point the push config at your Redis instance for cross-instance delivery:

   # gateway.yaml
   push:
     enabled: true
     redis:
       host: redis.internal
       port: 6379

3. Watch outage signals via gateway status

   The three redis_* fields tell you the transport’s health at a glance:

   praisonai gateway status --json
   # push.redis_connected:      bool  (false while degraded)
   # push.redis_degraded:       bool  (true during the outage window)
   # push.redis_dropped_writes: int   (cumulative)

How It Works

On a listener error the adapter drops its stale handles, marks itself degraded, then reconnects with exponential backoff (capped at 30s) and re-subscribes every tracked channel before resuming.

• Bounded exponential backoff — retries start at 1s, double each failure, and cap at 30s. The loop only exits if the adapter is disconnected.
• Automatic re-subscription — every channel tracked before the outage is re-subscribed on the fresh pub/sub handle.
• Degraded surfacing — the adapter records itself as owner ("route", "redis-pubsub") with state="stale", a redacted reason (exception string only — never the URL or password), and an actionable retry_hint.
• Dropped-write counting — publish, set_presence, remove_presence, store_message, and delete_message calls made while disconnected increment dropped_writes instead of vanishing.
• Clears on recovery — the degraded record is removed once Redis is reachable again.

Health Signals

health() (and gateway status) expose three fields on the push status block.

During an outage, the degraded transport is listed under degraded_owners:

{
  "push": {
    "redis_connected": false,
    "redis_degraded": true,
    "redis_dropped_writes": 3,
    "online_clients": 12
  },
  "degraded_owners": [
    {
      "owner_kind": "route",
      "owner_id": "redis-pubsub",
      "state": "stale",
      "reason": "redis pub/sub disconnected: Connection refused",
      "retry_hint": "check Redis connectivity; see `praisonai gateway doctor`"
    }
  ]
}

After recovery the redis_* fields return to healthy and the route:redis-pubsub entry is gone. Watch for the Redis push adapter reconnected (server_id=<uuid>) log line.

Doctor Integration

The degraded record shows up in praisonai gateway doctor (see Extensible Gateway Doctor) with the actionable retry_hint. No new plugin is needed — the adapter registers itself with the shared DegradedCapabilityRegistry.

praisonai gateway doctor
# route:redis-pubsub  stale
#   redis pub/sub disconnected: Connection refused
#   → check Redis connectivity; see `praisonai gateway doctor`

Best Practices

Related