Skip to main content
Durable Delivery persists every outbound bot message to a SQLite outbox so retries, restarts, and platform outages never drop a reply. For permanently dead targets — bot kicked, chat deleted — see Dead Target Registry, which suppresses doomed sends instead of queuing them.
This page covers outbound durability (DurableDelivery / OutboundQueue). For inbound durability — which is now on by default for all gateway/bot runs — see Inbound Journal and Inbound DLQ.
Durable Delivery vs Outbound Resilience. Durable Delivery is the heavy option — a SQLite outbox that survives crashes and lets you send_durable() explicitly. The lighter Outbound Resilience is now on by default in every bot adapter and retries transient failures without changing any code. Use Outbound Resilience for typical “don’t drop replies on a 429” scenarios; reach for Durable Delivery when you also need to survive process crashes.
The user expects a bot reply; durable delivery queues outbound messages, retries transient failures, and drains after restarts.

Quick Start

1

Easiest path — DurableAdapterMixin

Add three lines to any existing adapter and every send_durable() call is crash-safe:
2

Configure manually with DurableDelivery

For full control, wire up OutboundQueue and DurableDelivery directly:

How It Works

Each message moves through statuses: pendingsendingsent (or failed / permanent_failure).

State Machine

The outbox tracks six statuses: pending, sending, recovered, sent, failed, and permanent_failure. On restart, stale sending entries transition to recovered instead of pending. This preserves the information that the send was in-flight when the crash occurred.
recovered entries are retryable like pending, but when a reconciler is supplied to drain(), they are offered to it first — allowing adapters that can check the platform to confirm delivery and avoid re-sending an already-delivered message (effectively-once). Without a reconciler, recovered entries are re-sent as normal (at-least-once, unchanged behaviour).
Status-update writes (mark_sent, mark_failed, _claim_entry) now call conn.commit(). Without this, a terminal status could be lost on a crash, causing an already-delivered message to be redelivered. This is a reliability fix — no API change is required.

Age-gated dead-lettering (attempts + wall-clock)

permanent_failure is now a two-condition decision, so a brief channel outage no longer silently drops deliverable messages. An entry moves to permanent_failure when either:
  • The error class is known-permanent ("credential" or "permanent_target") — short-circuits immediately regardless of age.
  • Both attempts >= max_attempts and the entry has been in the queue for >= dead_letter_min_age (default 6h).
Otherwise the entry stays retryable and the next drain() reschedules it under the normal capped backoff. The default 6-hour floor is far longer than any realistic channel incident and well under the 7-day retention TTL, so most operators need no config change — transient outages simply keep retrying until the channel recovers. Restoring legacy behaviour — pass dead_letter_min_age=0 if you rely on attempt-count-only quarantine (e.g. poison-message unit tests):
Custom policy — inject any DeadLetterPolicyProtocol implementation for one-off decisions:
Introduced in PraisonAI PR #3521.

Effectively-Once Delivery

Adapters that can confirm whether a prior send actually landed can upgrade durable delivery from at-least-once to effectively-once.
1

Declare the capability

Set reconciles_unknown_send=True on PlatformCapabilities:
2

Implement was_delivered

Add an async method that checks the platform for a prior send:
3

Call drain_pending() at startup

DurableDelivery auto-wires the reconciler from the adapter’s capability — no extra configuration required:

Choosing the Right Primitive


Configuration Options

OutboundQueue

SQLite-backed outbox. All parameters after path are keyword-only.
On first open, older outbox.sqlite files are auto-migrated to add a lane_key column, backfilled to target. No user action required — existing outbox files continue to work.

OutboundQueue.drain() signature

OutboundQueue.enqueue() signature

enqueue() accepts an optional keyword-only lane_key for per-conversation grouping. It defaults to target, so messages to the same chat share a lane under ordering="strict".

OutboundQueue.status_for() signature

status_for() returns the current status of the entry for an idempotency key.
Returns — the current entry status ("pending", "sending", "recovered", "sent", "failed", or "permanent_failure"), or None if no entry exists for that key. Use it to tell an already-delivered duplicate (status "sent") apart from a genuine delivery failure after a drain — so a deduplicated re-fire is reported as a suppressed success rather than a spurious failure.

DurableDelivery

Wraps an OutboundQueue and an adapter to provide a simple .send() / .drain_pending() API.

DurableAdapterMixin.setup_durable_delivery()

Call once in your adapter’s __init__ to wire up the outbox.

deliver_with_retry()

Bounded retry without persistence — on a recoverable failure, the delay is the server-mandated wait (server_retry_after(err)) when present, otherwise compute_backoff(policy, attempt).

server_retry_after()

Extracts a server-mandated wait (seconds) from an error or response. Used internally by deliver_with_retry, ConnectionMonitor.record_error, and OutboundQueue.drain to honour explicit throttle signals over the policy backoff.
Returns None when no hint is present — callers fall back to the policy backoff.

deliver_chunked()

Splits a long message at paragraph boundaries and sends each chunk separately. Returns the number of chunks sent.

BackoffPolicy

Controls retry timing for both deliver_with_retry and OutboundQueue.drain.

Idempotency & When the Outbox Drains

Idempotency Keys

Every message has an idempotency_key — a UUID generated automatically if you omit it. Reusing the same key for the same logical message prevents double-sends across retries.
If the webhook is redelivered and send() is called again with the same key, the outbox skips the enqueue (SQLite UNIQUE constraint) and marks the existing row sent.

When the Outbox Drains

Held replies go out on three triggers, so operators can reason about when a queued message is actually re-attempted:
  1. Adapter startup — replays anything queued before the last crash (drain_pending() / drain_outbox()).
  2. Channel recovery (new — Issue #4043) — when ChannelSupervisor observes a channel come back after a recoverable outage (network blip, transient API error, stale socket), it fires drain_outbox() in the background so held replies go out promptly. Applications do not have to call anything.
  3. Lazy on next inbound turn — the next chat() turn also opportunistically drains.
Call drain_pending() once at adapter startup to replay anything that was queued before the last crash:
The drain replays oldest messages first and skips messages that have exceeded max_attempts. The retry gate is max(compute_backoff(policy, attempts + 1), server_retry_after(stored_err)) — the platform’s mandated wait survives across process restarts.
The hint is recovered from the stored error string, not the live exception. Only hints that survive str(err) (python-telegram-bot RetryAfter repr, HTTP headers folded into the error message, text-form “retry after N”) are honoured on drain.

Recovery-triggered re-drain (Issue #4043)

Why did my held reply just go out on its own? When a channel drops on a transient outage, the outbox deliberately holds deliverable replies instead of dropping them. Before, those replies sat undelivered until the next inbound chat() turn or a process restart. Now ChannelSupervisor re-drains that platform’s outbox the moment the channel reconnects — see Channel supervision → Recovery-triggered outbox re-drain.
The re-drain fires only when the channel comes back from at least one recoverable failure (monitor.attempt > 0). A cold, clean first start does not trigger it — the startup drain still covers that path.
It launches as an asyncio.create_task(...) and is not awaited by the supervision loop, so a slow or large re-drain can never delay the channel coming back online.
The supervisor resolves drain_outbox on the supervised object first, then falls back to bot.adapter.drain_outbox. No hook → silent no-op. An exception is logged at WARNING and swallowed — the outbox’s own attempt-and-age dead-letter policy still governs those messages, so a failed re-drain never wedges supervision.
The re-drain is bounded by the outbox’s existing attempt-and-age policy and is safe to call more than once because the outbox itself dedupes on idempotency_key.
Log lines from the merged code:
Introduced in PraisonAI PR #4046 (fixes #4043).

Best Practices

A flaky was_delivered that returns False for an already-delivered message will cause a duplicate send. A reconciler that raises falls back to at-least-once re-send (safe, but logged at WARNING). Only opt in when your platform provides a reliable message-status API.
supports_idempotency_token=True is informational only — the outbox does not currently forward the token on resend. Adapters relying on provider-side deduplication should also set reconciles_unknown_send=True to get effectively-once delivery.
The is_recoverable_error() function checks platform-specific patterns (e.g., Telegram’s HTTP 409 conflict, rate-limit “retry after” responses) when a platform name is provided. Without it, only generic patterns are checked and some transient errors may be misclassified as permanent.
When bridging a webhook to an outbound reply, derive the key from the inbound message ID. This ensures webhook redeliveries don’t produce duplicate outbound sends.
Store the outbox on a persistent, local filesystem path — not /tmp and not a Docker tmpfs. The default suggestion is ~/.praisonai/state/outbox.sqlite.
Multiple concurrent drainers fight over the same rows via SQLite’s status = 'sending' claim mechanism. A 5-minute claim timeout releases stale claims, but concurrent drainers still produce redundant work and log noise.
The 6-hour default is calibrated so a routine channel outage (Telegram 429 storm, Discord Cloudflare hiccup, WhatsApp API blip) never permanently loses a deliverable message, while a genuinely poisoned payload still dead-letters within the day. Only lower it when you have a specific need — e.g. dead_letter_min_age=0 inside a test suite where you want the pre-#3521 attempt-only behaviour, or dead_letter_min_age=60 in a synthetic burn-in that can’t wait 6 hours to observe the terminal state.
The praisonai-bot package’s dependency floor (praisonaiagents>=1.6.152) admits core releases that predate the shared AttemptAndAgeDeadLetterPolicy (first shipped 1.6.161). On those installs the queues fall back to a bundled LocalDeadLetterPolicy with identical semantics, so the age gate holds regardless of which core version is installed. If you customise the policy, prefer importing AttemptAndAgeDeadLetterPolicy from praisonaiagents.gateway — the import will succeed on any core ≥ 1.6.161.

Proactive Path

Direct BotOS.deliver(...) shares the reply-path rate limiter but is fire-and-forget. When the gateway itself performs the scheduled/proactive send, it runs through a durable outbox instead.
Use delivery.send(...) when you need durability across restarts, workers, or deduplication on the reply path. The gateway’s own scheduled/proactive deliveries now get durability automatically — no extra API required. Reach for direct BotOS.deliver(...) (agent-initiated fire-and-forget, no gateway) only when rate-limiting is enough and durability across restart is not required.

Per-Adapter Delivery Guarantee

Only adapters that declare reconciles_unknown_send=True and implement was_delivered reach effectively-once; the rest stay at-least-once by design. Slack is the first built-in adapter to implement the reconciliation seam end-to-end:
  • SlackBot declares reconciles_unknown_send=True on its platform_capabilities.
  • send_message(target, text, ..., idempotency_key=None) stamps the key into the Slack message metadata with event_type="praisonai_outbound".
  • was_delivered(target, idempotency_key) reads back recent history (conversations.history, or conversations.replies for a threaded send) and matches the key in metadata.event_payload. It returns False on any lookup failure, falling back to at-least-once.
Channels whose platform cannot confirm delivery remain at-least-once by design — that is now an explicit per-channel fact, not a silent default. See Bot Platform Capabilities.
For those at-least-once channels you can opt into visible labelling of crash-recovered re-sends with mark_recovered=True. The recipient sees a ♻️ Recovered reply — the gateway restarted during delivery, so this may be a duplicate. prefix instead of a silent possible-duplicate. See Labelling the at-least-once fallback.

Outbound Ordering

Per-conversation FIFO ordering on this outbox — keep messages to a chat in order under retries

Dead Target Registry

Suppress permanently-dead channels — the permanent-failure complement to durable retry

Inbound Journal

Inbound counterpart — deduplicate webhook redeliveries and recover in-flight messages

Inbound DLQ

Dead-letter queue for failed inbound message processing

Delivery Config

Configure outbound resilience for all six bot channels

Bot Streaming Replies

Live-edit streaming UX for bot responses

Messaging Bots

Top-level guide to building bots with PraisonAI