Gateway Cross-Replica Idempotency

Exactly-once webhook admission across every gateway replica via a Redis-backed idempotency store

Turn on store_backend: redis and every replica of your gateway admits the same webhook exactly once, so a duplicate delivery never runs your agent twice.

Diagram

Cross-Replica Idempotency

first claim

duplicate

📥 Webhook
replica A

🔐 reserve
Redis SET NX PX

📥 Same webhook
replica B

✅ Admit · run once

🚫 Reject

Quick Start

  1. Single replica — nothing to do

    One gateway process dedups with the durable SQLite default. Nothing to set.

    python
    from praisonaiagents import Agent
    from praisonai_bot.gateway import WebSocketGateway
    
    agent = Agent(name="Support", instructions="Help users")
    WebSocketGateway(agent=agent).start()   # store_backend defaults to durable sqlite
  2. Multiple replicas — set store_backend: redis

    Two or more gateway processes need a shared claim so the same webhook fanned to both is admitted only once. Set it in gateway.yaml and make sure a Redis URL is configured — the store reuses the gateway’s push RedisConfig.

    yaml
    gateway:
      push:
        redis:
          url: redis://localhost:6379   # reused for the idempotency store
    hooks:
      idempotency:
        store_backend: redis            # cluster-wide exactly-once admission
      hooks:
        - path: gmail
          agent: assistant
  3. Or wire it in Python

    Pass the same store_backend through the gateway’s hooks config where you construct it.

    python
    from praisonaiagents import Agent
    from praisonai_bot.gateway import WebSocketGateway
    
    agent = Agent(name="Support", instructions="Help users")
    
    gateway = WebSocketGateway(
        agent=agent,
        config={
            "push": {"redis": {"url": "redis://localhost:6379"}},
            "hooks": {"idempotency": {"store_backend": "redis"}},
        },
    )
    gateway.start()

How It Works

Each replica claims the key (platform, account, channel_id, message_id) with a single SET NX PX; the first wins and the duplicate is rejected.

Diagram
Redis(key: hookidem:msg-42)Replica BReplica ARedis(key: hookidem:msg-42)Replica BReplica AA crashes before record?claim expires after the lease → provider retry re-runsSET NX PX (token=A, inflight lease)OK — claim acquired (admit)SET NX PX (token=B, inflight lease)nil — already claimed (reject duplicate)record (upgrade to “recorded”, 24h TTL)
CallWhat it doesAtomicity
reserve(key)Single SET NX PX — atomic claim + owner token + inflight-lease TTL (default 900s). Rejects a redelivery and a concurrent duplicate on another replica.Native SET NX PX
record(key)Owner-checked upgrade to a long-lived recorded value (TTL 86400s / 24h). A late record from a replica whose lease expired is a no-op when another replica reclaimed the key.Lua EVAL, with a WATCH/MULTI fallback
release(key)Compare-and-del on the owner token, so it never drops another replica’s live claim.Lua EVAL, with a WATCH/MULTI fallback

Fail-Open on Redis Outage

The store never wedges inbound delivery — it degrades to best-effort and surfaces the fact, never a silent green.

Diagram

Yes

No — missing pkg / unreachable

drops later

recovers

⚙️ store_backend: redis

🔍 Redis reachable
at build time?

🔐 RedisIdempotencyStore
cluster-wide exactly-once

🗄️ Durable SQLite fallback
per-replica

⚠️ Runtime outage
admits · best-effort

⚠️ durability:idempotency
degraded fact

Two failure modes, both surfaced under the durability:idempotency degraded owner:

ModeWhenReason stringRecovery
Build-timeThe redis package is missing or Redis is unreachable at construction"redis idempotency backend not available (running per-replica)"Falls back to durable SQLite; clears when Redis is reachable on the next build / hot-reload
RuntimeRedis was reachable at build but dropped later"redis idempotency backend unreachable at runtime (dedup best-effort until it recovers)"Admits (fails open); auto-clears on the next successful Redis op

When to Enable

Diagram

Yes

No

replicas > 1?

⚙️ store_backend: redis
cluster-wide exactly-once

🗄️ default durable sqlite


Configuration Options

The operator selects the backend and the store reuses the gateway’s push RedisConfig — no separate idempotency Redis knob.

OptionWherePurpose
store_backendhooks.idempotency.store_backendSet to redis for cluster-wide exactly-once admission
urlpush.redis.urlRedis connection URL, reused by the store
host / port / db / passwordpush.redis.*Field-wise connection when no url is given
prefixpush.redis.prefixNamespaces keys as {prefix}hookidem:{key} so environments sharing one Redis do not collide
Gateway Turn Lock

The sibling shared-state knob that wires the same RedisConfig.


Best Practices

Enable the turn lock and idempotency together

For a multi-replica gateway, set both turn_lock.backend: redis and hooks.idempotency.store_backend: redis — one serialises a session’s turns, the other admits each webhook exactly once.

Reuse the gateway's push RedisConfig

The store auto-shares push.redis. Configure Redis once; do not duplicate the connection for idempotency.

Watch for durability:idempotency

A durability:idempotency entry in gateway doctor / GET /health means dedup fell back or is best-effort. Alert on it the same way you alert on a degraded channel.

Enable it before raising the replica count

Switch to store_backend: redis before scaling past one replica, so a fanned-out webhook is deduped from the first extra pod.


Related