DurableDelivery / OutboundQueue). For inbound durability — which is now on by default for all gateway/bot runs — see Inbound Journal and Inbound DLQ.send_durable() explicitly. The lighter Outbound Resilience is now on by default in every bot adapter and retries transient failures without changing any code. Use Outbound Resilience for typical “don’t drop replies on a 429” scenarios; reach for Durable Delivery when you also need to survive process crashes.Quick Start
Easiest path — DurableAdapterMixin
send_durable() call is crash-safe:Configure manually with DurableDelivery
OutboundQueue and DurableDelivery directly:How It Works
Each message moves through statuses:pending → sending → sent (or failed / permanent_failure).
State Machine
The outbox tracks six statuses:pending, sending, recovered, sent, failed, and permanent_failure.
On restart, stale sending entries transition to recovered instead of pending. This preserves the information that the send was in-flight when the crash occurred.
recovered entries are retryable like pending, but when a reconciler is supplied to drain(), they are offered to it first — allowing adapters that can check the platform to confirm delivery and avoid re-sending an already-delivered message (effectively-once). Without a reconciler, recovered entries are re-sent as normal (at-least-once, unchanged behaviour).
mark_sent, mark_failed, _claim_entry) now call conn.commit(). Without this, a terminal status could be lost on a crash, causing an already-delivered message to be redelivered. This is a reliability fix — no API change is required.Age-gated dead-lettering (attempts + wall-clock)
permanent_failure is now a two-condition decision, so a brief channel outage no longer silently drops deliverable messages.
An entry moves to permanent_failure when either:
- The error class is known-permanent (
"credential"or"permanent_target") — short-circuits immediately regardless of age. - Both
attempts >= max_attemptsand the entry has been in the queue for>= dead_letter_min_age(default 6h).
drain() reschedules it under the normal capped backoff.
The default 6-hour floor is far longer than any realistic channel incident and well under the 7-day retention TTL, so most operators need no config change — transient outages simply keep retrying until the channel recovers.
Restoring legacy behaviour — pass dead_letter_min_age=0 if you rely on attempt-count-only quarantine (e.g. poison-message unit tests):
DeadLetterPolicyProtocol implementation for one-off decisions:
Effectively-Once Delivery
Adapters that can confirm whether a prior send actually landed can upgrade durable delivery from at-least-once to effectively-once.Declare the capability
reconciles_unknown_send=True on PlatformCapabilities:Implement was_delivered
Call drain_pending() at startup
DurableDelivery auto-wires the reconciler from the adapter’s capability — no extra configuration required:Choosing the Right Primitive
Configuration Options
OutboundQueue
SQLite-backed outbox. All parameters after path are keyword-only.
outbox.sqlite files are auto-migrated to add a lane_key column, backfilled to target. No user action required — existing outbox files continue to work.OutboundQueue.drain() signature
OutboundQueue.enqueue() signature
enqueue() accepts an optional keyword-only lane_key for per-conversation grouping. It defaults to target, so messages to the same chat share a lane under ordering="strict".
OutboundQueue.status_for() signature
status_for() returns the current status of the entry for an idempotency key.
"pending", "sending", "recovered", "sent", "failed", or "permanent_failure"), or None if no entry exists for that key.
Use it to tell an already-delivered duplicate (status "sent") apart from a genuine delivery failure after a drain — so a deduplicated re-fire is reported as a suppressed success rather than a spurious failure.
DurableDelivery
Wraps an OutboundQueue and an adapter to provide a simple .send() / .drain_pending() API.
DurableAdapterMixin.setup_durable_delivery()
Call once in your adapter’s __init__ to wire up the outbox.
deliver_with_retry()
Bounded retry without persistence — on a recoverable failure, the delay is the server-mandated wait (server_retry_after(err)) when present, otherwise compute_backoff(policy, attempt).
server_retry_after()
Extracts a server-mandated wait (seconds) from an error or response. Used internally by deliver_with_retry, ConnectionMonitor.record_error, and OutboundQueue.drain to honour explicit throttle signals over the policy backoff.
None when no hint is present — callers fall back to the policy backoff.
deliver_chunked()
Splits a long message at paragraph boundaries and sends each chunk separately. Returns the number of chunks sent.
BackoffPolicy
Controls retry timing for both deliver_with_retry and OutboundQueue.drain.
Idempotency & When the Outbox Drains
Idempotency Keys
Every message has anidempotency_key — a UUID generated automatically if you omit it. Reusing the same key for the same logical message prevents double-sends across retries.
send() is called again with the same key, the outbox skips the enqueue (SQLite UNIQUE constraint) and marks the existing row sent.
When the Outbox Drains
Held replies go out on three triggers, so operators can reason about when a queued message is actually re-attempted:- Adapter startup — replays anything queued before the last crash (
drain_pending()/drain_outbox()). - Channel recovery (new — Issue #4043) — when
ChannelSupervisorobserves a channel come back after a recoverable outage (network blip, transient API error, stale socket), it firesdrain_outbox()in the background so held replies go out promptly. Applications do not have to call anything. - Lazy on next inbound turn — the next
chat()turn also opportunistically drains.
drain_pending() once at adapter startup to replay anything that was queued before the last crash:
max_attempts. The retry gate is max(compute_backoff(policy, attempts + 1), server_retry_after(stored_err)) — the platform’s mandated wait survives across process restarts.
str(err) (python-telegram-bot RetryAfter repr, HTTP headers folded into the error message, text-form “retry after N”) are honoured on drain.Recovery-triggered re-drain (Issue #4043)
Why did my held reply just go out on its own? When a channel drops on a transient outage, the outbox deliberately holds deliverable replies instead of dropping them. Before, those replies sat undelivered until the next inboundchat() turn or a process restart. Now ChannelSupervisor re-drains that platform’s outbox the moment the channel reconnects — see Channel supervision → Recovery-triggered outbox re-drain.
Recovery only — not a clean start
Recovery only — not a clean start
monitor.attempt > 0). A cold, clean first start does not trigger it — the startup drain still covers that path.Background-scheduled — never blocks recovery
Background-scheduled — never blocks recovery
asyncio.create_task(...) and is not awaited by the supervision loop, so a slow or large re-drain can never delay the channel coming back online.Duck-typed and best-effort
Duck-typed and best-effort
drain_outbox on the supervised object first, then falls back to bot.adapter.drain_outbox. No hook → silent no-op. An exception is logged at WARNING and swallowed — the outbox’s own attempt-and-age dead-letter policy still governs those messages, so a failed re-drain never wedges supervision.Bounded and idempotent
Bounded and idempotent
idempotency_key.Best Practices
Set reconciles_unknown_send=True only if you can answer the question reliably
Set reconciles_unknown_send=True only if you can answer the question reliably
was_delivered that returns False for an already-delivered message will cause a duplicate send. A reconciler that raises falls back to at-least-once re-send (safe, but logged at WARNING). Only opt in when your platform provides a reliable message-status API.Don't rely on supports_idempotency_token alone for dedupe
Don't rely on supports_idempotency_token alone for dedupe
supports_idempotency_token=True is informational only — the outbox does not currently forward the token on resend. Adapters relying on provider-side deduplication should also set reconciles_unknown_send=True to get effectively-once delivery.Always set platform= for accurate error classification
Always set platform= for accurate error classification
is_recoverable_error() function checks platform-specific patterns (e.g., Telegram’s HTTP 409 conflict, rate-limit “retry after” responses) when a platform name is provided. Without it, only generic patterns are checked and some transient errors may be misclassified as permanent.Use a stable idempotency_key derived from the inbound message
Use a stable idempotency_key derived from the inbound message
Keep the outbox on persistent local disk
Keep the outbox on persistent local disk
/tmp and not a Docker tmpfs. The default suggestion is ~/.praisonai/state/outbox.sqlite.Call drain_pending() exactly once per adapter start
Call drain_pending() exactly once per adapter start
status = 'sending' claim mechanism. A 5-minute claim timeout releases stale claims, but concurrent drainers still produce redundant work and log noise.Only lower dead_letter_min_age with a specific reason
Only lower dead_letter_min_age with a specific reason
dead_letter_min_age=0 inside a test suite where you want the pre-#3521 attempt-only behaviour, or dead_letter_min_age=60 in a synthetic burn-in that can’t wait 6 hours to observe the terminal state.Import AttemptAndAgeDeadLetterPolicy from praisonaiagents.gateway
Import AttemptAndAgeDeadLetterPolicy from praisonaiagents.gateway
praisonai-bot package’s dependency floor (praisonaiagents>=1.6.152) admits core releases that predate the shared AttemptAndAgeDeadLetterPolicy (first shipped 1.6.161). On those installs the queues fall back to a bundled LocalDeadLetterPolicy with identical semantics, so the age gate holds regardless of which core version is installed. If you customise the policy, prefer importing AttemptAndAgeDeadLetterPolicy from praisonaiagents.gateway — the import will succeed on any core ≥ 1.6.161.Proactive Path
DirectBotOS.deliver(...) shares the reply-path rate limiter but is fire-and-forget. When the gateway itself performs the scheduled/proactive send, it runs through a durable outbox instead.
delivery.send(...) when you need durability across restarts, workers, or deduplication on the reply path. The gateway’s own scheduled/proactive deliveries now get durability automatically — no extra API required. Reach for direct BotOS.deliver(...) (agent-initiated fire-and-forget, no gateway) only when rate-limiting is enough and durability across restart is not required.Per-Adapter Delivery Guarantee
Only adapters that declarereconciles_unknown_send=True and implement was_delivered reach effectively-once; the rest stay at-least-once by design.
SlackBotdeclaresreconciles_unknown_send=Trueon itsplatform_capabilities.send_message(target, text, ..., idempotency_key=None)stamps the key into the Slack messagemetadatawithevent_type="praisonai_outbound".was_delivered(target, idempotency_key)reads back recent history (conversations.history, orconversations.repliesfor a threaded send) and matches the key inmetadata.event_payload. It returnsFalseon any lookup failure, falling back to at-least-once.
mark_recovered=True. The recipient sees a ♻️ Recovered reply — the gateway restarted during delivery, so this may be a duplicate. prefix instead of a silent possible-duplicate. See Labelling the at-least-once fallback.
