> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Durable Outbound Delivery
> Persist outbound bot messages with retry, idempotency, and crash-safe drain on restart
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
agent = Agent(name="bot-agent", instructions="Reply on messaging channels.")
agent.start("Send the user a confirmation once the job completes.")
```
Durable Delivery persists every outbound bot message to a SQLite outbox so retries, restarts, and platform outages never drop a reply. For **permanently** dead targets — bot kicked, chat deleted — see [Dead Target Registry](/docs/features/dead-target-registry), which suppresses doomed sends instead of queuing them.
This page covers **outbound** durability (`DurableDelivery` / `OutboundQueue`). For **inbound** durability — which is now on by default for all gateway/bot runs — see [Inbound Journal](/docs/features/inbound-journal) and [Inbound DLQ](/docs/features/inbound-dlq).
For an **end-to-end** guarantee, pair outbound durable delivery with `BotSessionManager(defer_journal_completion=True)` — the inbound entry is settled only after the outbound send is durable, so a crash mid-send re-drives the turn instead of losing the reply. See [Inbound Journal → Defer completion until reply is delivered](/docs/features/inbound-journal#defer-completion-until-reply-is-delivered-opt-in).
**Durable Delivery vs Outbound Resilience.** Durable Delivery is the heavy option — a SQLite outbox that survives crashes and lets you `send_durable()` explicitly. The lighter [Outbound Resilience](/docs/features/outbound-resilience) is now on by default in every bot adapter and retries transient failures without changing any code. Use Outbound Resilience for typical "don't drop replies on a 429" scenarios; reach for Durable Delivery when you also need to survive process crashes.
The user expects a bot reply; durable delivery queues outbound messages, retries transient failures, and drains after restarts.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
subgraph "Durable Outbound Delivery"
Agent[🤖 Agent] --> Send[📤 send_durable]
Send --> Queue[💾 OutboundQueue]
Queue --> Retry[🔁 deliver_with_retry]
Retry -->|success| Sent[✅ mark_sent]
Retry -->|transient fail| Later[⏳ drain_pending]
Later --> Retry
end
classDef agent fill:#8B0000,stroke:#7C90A0,color:#fff
classDef process fill:#189AB4,stroke:#7C90A0,color:#fff
classDef storage fill:#6366F1,stroke:#7C90A0,color:#fff
classDef success fill:#10B981,stroke:#7C90A0,color:#fff
classDef pending fill:#F59E0B,stroke:#7C90A0,color:#fff
class Agent agent
class Send,Retry process
class Queue storage
class Sent success
class Later pending
```
## Quick Start
Add three lines to any existing adapter and every `send_durable()` call is crash-safe:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
import os
from praisonaiagents import Agent
from praisonai.bots import TelegramBot
# TelegramBot already supports durable delivery via DurableAdapterMixin
agent = Agent(name="assistant", instructions="Help users")
bot = TelegramBot(
token=os.getenv("TELEGRAM_BOT_TOKEN"),
agent=agent,
)
# Replay anything queued before the last crash.
# Also fires automatically on channel recovery — no need to wire it yourself.
# Manual call is only needed for adapters not driven by ChannelSupervisor.
await bot.drain_outbox()
# Send with durability
await bot.send_durable(
channel_id="12345",
content="Hello! I'm crash-safe.",
idempotency_key="welcome-msg-12345",
)
```
For full control, wire up `OutboundQueue` and `DurableDelivery` directly:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
import os
from praisonaiagents import Agent
from praisonai.bots import OutboundQueue, DurableDelivery, TelegramBot
outbox = OutboundQueue(path="~/.praisonai/state/outbox.sqlite")
adapter = TelegramBot(
token=os.getenv("TELEGRAM_BOT_TOKEN"),
agent=Agent(name="assistant", instructions="Help users"),
)
delivery = DurableDelivery(
outbox, adapter, platform="telegram",
# mark_recovered=True, # opt-in: prefix crash-recovered re-sends with a "possible duplicate" marker
)
# On startup: replay anything queued before the last crash
succeeded, failed = await delivery.drain_pending()
# Send with durability — idempotent if you reuse the key
success = await delivery.send(
channel_id="12345",
content="Hello, world!",
idempotency_key="msg-123",
)
```
***
## How It Works
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
participant Bot as 🤖 Bot Adapter
participant Queue as 💾 OutboundQueue
participant Platform as 📱 Platform API
Bot->>Queue: enqueue(key, channel, payload)
Queue-->>Bot: tracking key
Bot->>Platform: attempt delivery
alt success
Platform-->>Bot: 200 OK
Bot->>Queue: mark_sent(key)
else transient failure
Platform-->>Bot: 429 (retry_after: 30)
Bot->>Queue: mark_failed(key, permanent=False)
Note over Queue: status = 'failed'
Bot->>Queue: drain_pending()
Note over Queue: drain gates by max(backoff, server_retry_after)
Queue->>Platform: retry after 30s
Platform-->>Queue: 200 OK
Queue->>Queue: mark_sent(key)
else permanent failure
Platform-->>Bot: 400 Bad Request
Bot->>Queue: mark_failed(key, permanent=True)
Note over Queue: status = 'permanent_failure'
end
```
Each message moves through statuses: `pending` → `sending` → `sent` (or `failed` / `permanent_failure`).
### State Machine
The outbox tracks six statuses: `pending`, `sending`, `recovered`, `sent`, `failed`, and `permanent_failure`.
To inspect, retry, or drop individual entries by status, use the [Outbox Operator CLI](/docs/features/outbox-operator-cli) — `praisonai gateway outbox list|stats|retry|purge`.
On restart, stale `sending` entries transition to **`recovered`** instead of `pending`. This preserves the information that the send was in-flight when the crash occurred.
```
status = 'sending' (crash) ──► status = 'recovered' (on restart)
```
`recovered` entries are retryable like `pending`, but when a reconciler is supplied to `drain()`, they are offered to it first — allowing adapters that can check the platform to confirm delivery and avoid re-sending an already-delivered message (**effectively-once**). Without a reconciler, `recovered` entries are re-sent as normal (at-least-once, unchanged behaviour).
**Surfaced as `SendResult.status`.** When an adapter routes a send through the outbox, the [`SendResult`](/docs/features/bot-platform-adapter#branching-on-delivery-outcome) it hands back carries `status == "queued"` — so a caller can tell "will land later" from "landed now" without inspecting private outbox state. After a crash-recovered re-drain that re-sends without positive reconciliation, the surfaced status is `"duplicate"` — an honest at-least-once signal. `DurableDelivery` itself still returns its historical value; only the adapter's `SendResult` gains the typed status.
Status-update writes (`mark_sent`, `mark_failed`, `_claim_entry`) now call `conn.commit()`. Without this, a terminal status could be lost on a crash, causing an already-delivered message to be redelivered. This is a reliability fix — no API change is required.
### Age-gated dead-lettering (attempts + wall-clock)
`permanent_failure` is now a two-condition decision, so a brief channel outage no longer silently drops deliverable messages.
An entry moves to `permanent_failure` when **either**:
* The error class is **known-permanent** (`"credential"` or `"permanent_target"`) — short-circuits immediately regardless of age.
* **Both** `attempts >= max_attempts` **and** the entry has been in the queue for `>= dead_letter_min_age` (default 6h).
Otherwise the entry stays retryable and the next `drain()` reschedules it under the normal capped backoff.
The default 6-hour floor is far longer than any realistic channel incident and well under the 7-day retention TTL, so most operators need no config change — transient outages simply keep retrying until the channel recovers.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
Start([drain sees exhausted entry]) --> ErrCheck{Known-permanent
error class?}
ErrCheck -->|credential / permanent_target| DL[💀 permanent_failure]
ErrCheck -->|recoverable| AgeCheck{Age ≥ dead_letter_min_age?}
AgeCheck -->|Yes| DL
AgeCheck -->|No| Retry[🔁 Reschedule under backoff]
classDef start fill:#6366F1,stroke:#7C90A0,color:#fff
classDef check fill:#F59E0B,stroke:#7C90A0,color:#fff
classDef terminal fill:#8B0000,stroke:#7C90A0,color:#fff
classDef retry fill:#10B981,stroke:#7C90A0,color:#fff
class Start start
class ErrCheck,AgeCheck check
class DL terminal
class Retry retry
```
**Restoring legacy behaviour** — pass `dead_letter_min_age=0` if you rely on attempt-count-only quarantine (e.g. poison-message unit tests):
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonai.bots import OutboundQueue
# Attempt-only dead-lettering (pre-#3521 behaviour)
outbox = OutboundQueue(
path="~/.praisonai/state/outbox.sqlite",
max_attempts=5,
dead_letter_min_age=0,
)
```
**Custom policy** — inject any `DeadLetterPolicyProtocol` implementation for one-off decisions:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.gateway import AttemptAndAgeDeadLetterPolicy
from praisonai.bots import OutboundQueue
outbox = OutboundQueue(
path="~/.praisonai/state/outbox.sqlite",
dead_letter_policy=AttemptAndAgeDeadLetterPolicy(
max_attempts=10,
min_age_seconds=24 * 3600, # allow a full day of retries
),
)
```
Introduced in [PraisonAI PR #3521](https://github.com/MervinPraison/PraisonAI/pull/3521).
***
## Effectively-Once Delivery
Adapters that can confirm whether a prior send actually landed can upgrade durable delivery from at-least-once to effectively-once.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
participant Adapter as 🤖 Adapter
participant Queue as 💾 OutboundQueue
participant Provider as 📱 Platform
Note over Queue: status = 'sending'
Adapter--xProvider: 💥 crash mid-send
Note over Queue: on restart → 'recovered'
Adapter->>Queue: drain_pending() (with reconciler)
Queue->>Adapter: reconciler(entry)
Adapter->>Provider: was_delivered(idempotency_key)
alt Provider says delivered
Provider-->>Adapter: True
Adapter-->>Queue: True → mark_sent (NO resend)
else Provider says not delivered
Provider-->>Adapter: False
Queue->>Provider: resend
end
```
Set `reconciles_unknown_send=True` on `PlatformCapabilities`:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.bots import PlatformCapabilities
class MyAdapter(DurableAdapterMixin):
platform_capabilities = PlatformCapabilities(
reconciles_unknown_send=True,
)
```
Add an async method that checks the platform for a prior send:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
async def was_delivered(self, idempotency_key: str) -> bool:
status = await my_provider.get_message_status(idempotency_key)
return status == "sent"
```
`DurableDelivery` auto-wires the reconciler from the adapter's capability — no extra configuration required:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.bots import PlatformCapabilities
from praisonai.bots._adapter_base import DurableAdapterMixin
class AcmeAdapter(DurableAdapterMixin):
platform_capabilities = PlatformCapabilities(
reconciles_unknown_send=True,
)
async def was_delivered(self, idempotency_key: str) -> bool:
status = await acme_client.messages.lookup(idempotency_key)
return status.delivered
async def send_message(self, channel_id: str, content: str) -> None:
await acme_client.messages.send(channel_id, content)
# DurableDelivery picks up the reconciler automatically.
succeeded, failed = await adapter.drain_outbox()
```
***
## Choosing the Right Primitive
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
Start([What do I need?]) --> Q1{Survive crashes\nand restarts?}
Q1 -->|Yes, I have an adapter| Mixin[DurableAdapterMixin]
Q1 -->|Yes, full control| DD[DurableDelivery]
Q1 -->|No persistence needed| Q2{Message too\nlong for platform?}
Q2 -->|Yes| Chunk[deliver_chunked]
Q2 -->|No| Q3{Need retry on\ntransient 429/503?}
Q3 -->|Yes, on by default| Resilience[OutboundResilienceMixin]
Q3 -->|Manual control| Retry[deliver_with_retry]
classDef decision fill:#F59E0B,stroke:#7C90A0,color:#fff
classDef result fill:#10B981,stroke:#7C90A0,color:#fff
class Q1,Q2,Q3 decision
class Mixin,DD,Chunk,Resilience,Retry result
```
| I need to… | Use |
| - | - |
| Auto-retry transient failures — no config, works everywhere | [`OutboundResilienceMixin`](/docs/features/outbound-resilience) (on by default in every adapter) |
| Manual retry with explicit backoff policy | `deliver_with_retry()` |
| Send a message that may exceed platform length limits | `deliver_chunked()` |
| Survive process crashes / platform outages with replay | `DurableDelivery` or `DurableAdapterMixin` |
| Build a brand-new custom adapter with durability built in | `DurableAdapterMixin` |
***
## Configuration Options
### `OutboundQueue`
SQLite-backed outbox. All parameters after `path` are keyword-only.
| Parameter | Type | Default | Description |
| - | - | - | - |
| `path` | `str \| Path` | *(required)* | SQLite file path. Parent dirs are created automatically. |
| `max_size` | `int` | `50_000` | Max entries kept. Oldest sent entries are evicted when exceeded. |
| `ttl_seconds` | `int` | `604800` (7 days) | Sent entries older than this are evicted. |
| `max_attempts` | `int` | `5` | Max delivery attempts before a message is eligible for permanent failure. Since [PR #3521](https://github.com/MervinPraison/PraisonAI/pull/3521), exhausting attempts alone is no longer enough to dead-letter a *transient* failure — the entry must also satisfy `dead_letter_min_age`. A known-permanent error (credentials, permanently-invalid target) still short-circuits immediately. |
| `dead_letter_min_age` | `float` (seconds) | `21600` (6h) | Minimum wall-clock age since first receipt before an attempt-exhausted transient failure is marked `permanent_failure`. Set to `0` to restore the pre-#3521 attempt-only behaviour. |
| `dead_letter_policy` | `DeadLetterPolicyProtocol \| None` | `None` | Optional custom `praisonaiagents.gateway.DeadLetterPolicyProtocol` implementation. When supplied, overrides `max_attempts` + `dead_letter_min_age`. |
| `backoff` | `BackoffPolicy` | `BackoffPolicy()` | Retry backoff configuration. |
| `ordering` | `"strict" \| "best_effort"` | `"best_effort"` | Per-conversation FIFO gate. `strict` holds later same-lane messages until the head reaches `sent` or `permanent_failure`. See [Outbound Ordering](/docs/features/outbound-ordering). |
On first open, older `outbox.sqlite` files are auto-migrated to add a `lane_key` column, backfilled to `target`. No user action required — existing outbox files continue to work.
#### `OutboundQueue.drain()` signature
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
async def drain(
self,
sender: Callable[[str, Dict[str, Any]], Awaitable[bool]],
limit: Optional[int] = None,
*,
reconciler: Optional[Callable[[OutboundEntry], Awaitable[bool]]] = None,
) -> Tuple[int, int]:
```
| Parameter | Type | Default | Description |
| - | - | - | - |
| `sender` | `async callable` | *(required)* | Async callable that sends the message. |
| `limit` | `int \| None` | `None` | Max entries to drain in this call. |
| `reconciler` | `async callable \| None` | `None` | Called only for `recovered` entries. Returns `True` → mark sent without re-dispatch (effectively-once). Returns `False` → re-send as normal. Raises → falls back to re-send (logged at WARNING). |
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonai.bots import OutboundQueue
outbox = OutboundQueue(
path="~/.praisonai/state/outbox.sqlite",
max_size=10_000,
ttl_seconds=3 * 86400, # 3 days
max_attempts=3,
)
```
#### `OutboundQueue.enqueue()` signature
`enqueue()` accepts an optional keyword-only `lane_key` for per-conversation grouping. It defaults to `target`, so messages to the same chat share a lane under `ordering="strict"`.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
async def enqueue(
self,
idempotency_key: str,
target: str,
payload: Dict[str, Any],
metadata: Optional[Dict[str, Any]] = None,
*,
lane_key: Optional[str] = None, # defaults to target
) -> str:
...
```
| Parameter | Type | Default | Description |
| - | - | - | - |
| `lane_key` | `str \| None` | `None` (= `target`) | Custom conversation grouping. Pass a thread id or user id to order sends within a shared channel. See [Outbound Ordering](/docs/features/outbound-ordering). |
#### `OutboundQueue.status_for()` signature
`status_for()` returns the current status of the entry for an idempotency key.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
def status_for(self, idempotency_key: str) -> Optional[str]:
...
```
| Parameter | Type | Description |
| - | - | - |
| `idempotency_key` | `str` | The stable key used at `enqueue()` time. |
**Returns** — the current entry status (`"pending"`, `"sending"`, `"recovered"`, `"sent"`, `"failed"`, or `"permanent_failure"`), or `None` if no entry exists for that key.
Use it to tell an already-delivered duplicate (status `"sent"`) apart from a genuine delivery failure after a drain — so a deduplicated re-fire is reported as a suppressed success rather than a spurious failure.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonai.bots import OutboundQueue
outbox = OutboundQueue(path="~/.praisonai/state/gateway_outbox.sqlite")
await outbox.enqueue(
idempotency_key="daily-brief-2026-07-20",
target="telegram:-100123",
payload={"text": "Morning brief ready."},
)
await outbox.drain(sender)
if outbox.status_for("daily-brief-2026-07-20") == "sent":
print("Delivered (or suppressed as a duplicate of a prior send).")
```
### `DurableDelivery`
Wraps an `OutboundQueue` and an adapter to provide a simple `.send()` / `.drain_pending()` API.
| Parameter | Type | Default | Description |
| - | - | - | - |
| `outbox` | `OutboundQueue` | *(required)* | The outbound queue to persist messages. |
| `adapter` | adapter instance | *(required)* | Bot adapter with a `send_message(channel_id, content)` method. |
| `platform` | `str` | `""` | Platform name for platform-aware error classification. |
| `backoff` | `BackoffPolicy` | `BackoffPolicy()` | Retry backoff configuration. |
| `max_attempts` | `int` | `3` | Max delivery attempts. |
| `mark_recovered` | `bool` | `False` | When `True`, a crash-recovered message re-sent without positive reconciliation is prefixed with a visible "possible duplicate after restart" marker instead of being re-delivered silently. See [Labelling the at-least-once fallback](/docs/docs/features/bot-platform-capabilities#labelling-the-at-least-once-fallback). |
### `DurableAdapterMixin.setup_durable_delivery()`
Call once in your adapter's `__init__` to wire up the outbox.
| Parameter | Type | Default | Description |
| - | - | - | - |
| `outbox_path` | `str \| None` | `None` | Path to SQLite outbox. `None` disables durability. |
| `platform` | `str` | `""` | Platform name for error classification. |
| `max_attempts` | `int` | `3` | Max delivery attempts per message. |
| `max_size` | `int` | `50_000` | Max messages in outbox. |
| `ttl_seconds` | `int` | `604800` (7 days) | TTL for sent messages. |
| `mark_recovered` | `bool` | `False` | When `True`, label crash-recovered re-sends with a "possible duplicate after restart" marker instead of re-delivering silently. See [Labelling the at-least-once fallback](/docs/docs/features/bot-platform-capabilities#labelling-the-at-least-once-fallback). |
### `deliver_with_retry()`
Bounded retry without persistence — on a recoverable failure, the delay is the **server-mandated wait** (`server_retry_after(err)`) when present, otherwise `compute_backoff(policy, attempt)`.
| Parameter | Type | Default | Description |
| - | - | - | - |
| `send_func` | `async callable` | *(required)* | Async callable to execute (the send operation). |
| `policy` | `BackoffPolicy` | `BackoffPolicy(max_attempts=3)` | Retry backoff configuration. |
| `is_recoverable` | `callable \| None` | `None` | Function to classify errors as transient. Defaults to `is_recoverable_error(e, platform)`. |
| `platform` | `str` | `""` | Platform name for platform-specific error rules. |
| `rate_limiter` | `Optional[RateLimiter]` | `None` | When supplied, a server-mandated wait (Telegram `retry_after`, HTTP `Retry-After`) triggers `await rate_limiter.penalise(channel_id, delay)` so subsequent sends to the same channel hold off until the platform's window elapses. |
| `parked_store` | `Any \| None` | `None` | Optional DLQ for failed sends. |
| `reply_data` | `dict \| None` | `None` | Optional metadata for DLQ storage. |
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonai.bots._delivery import deliver_with_retry
from praisonai.bots._rate_limit import RateLimiter
from praisonai.bots._resilience import BackoffPolicy
limiter = RateLimiter.for_platform("telegram")
ok, err = await deliver_with_retry(
send_fn, channel_id="telegram-chat-12345",
rate_limiter=limiter,
policy=BackoffPolicy(initial_ms=2000, max_ms=30000, max_attempts=5),
platform="telegram",
)
```
### `server_retry_after()`
Extracts a server-mandated wait (seconds) from an error or response. Used internally by `deliver_with_retry`, `ConnectionMonitor.record_error`, and `OutboundQueue.drain` to honour explicit throttle signals over the policy backoff.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonai.bots._resilience import server_retry_after
# Telegram RetryAfter
wait = server_retry_after(err) # -> 30.0
# HTTP Retry-After header (seconds or HTTP-date)
wait = server_retry_after(http_err) # -> 5.0 or seconds-until-date
# No hint present
wait = server_retry_after(generic_err) # -> None
```
| Source | Where it's read | Notes |
| - | - | - |
| Telegram `RetryAfter` | `err.retry_after` attribute | python-telegram-bot exposes this directly. |
| Telegram raw API 429 | `err.parameters["retry_after"]` | Raw API shape. |
| HTTP `Retry-After` | `err.headers`, `err.response.headers`, or `err.resp.headers` | Integer seconds **or** HTTP-date (parsed via `email.utils.parsedate_to_datetime`; missing tz treated as UTC; clamped `>= 0`). |
| Text fallback | regex on `str(err)` | Matches `retry after ` / `retry_after: `. |
Returns `None` when no hint is present — callers fall back to the policy backoff.
### `deliver_chunked()`
Splits a long message at paragraph boundaries and sends each chunk separately. Returns the number of chunks sent.
| Parameter | Type | Default | Description |
| - | - | - | - |
| `adapter` | adapter instance | *(required)* | Bot adapter with `send_message(channel_id, content)`. |
| `channel_id` | `str` | *(required)* | Target channel. |
| `content` | `str` | *(required)* | Message text to split and send. |
| `max_length` | `int` | `4096` | Max characters per chunk (Telegram limit is 4096). |
| `preserve_fences` | `bool` | `True` | Keep code fence blocks intact even if they exceed `max_length`. |
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonai.bots._chunk import chunk_message
# chunk_message is the underlying splitter
chunks = chunk_message(long_text, max_length=4096, preserve_fences=True)
for chunk in chunks:
await adapter.send_message(channel_id, chunk)
```
### `BackoffPolicy`
Controls retry timing for both `deliver_with_retry` and `OutboundQueue.drain`.
| Attribute | Type | Default | Description |
| - | - | - | - |
| `initial_ms` | `float` | `2000.0` | Initial delay in milliseconds. |
| `max_ms` | `float` | `30000.0` | Maximum delay in milliseconds. |
| `factor` | `float` | `1.8` | Multiplicative factor per attempt. |
| `jitter` | `float` | `0.25` | Random jitter fraction (0.0–1.0). |
| `max_attempts` | `int` | `0` | Max retry attempts. `0` = unlimited. |
***
## Idempotency & When the Outbox Drains
### Idempotency Keys
Every message has an `idempotency_key` — a UUID generated automatically if you omit it. Reusing the same key for the same logical message prevents double-sends across retries.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Webhook-triggered reply: derive key from inbound message ID
inbound_id = webhook_payload["message_id"]
await delivery.send(
channel_id=chat_id,
content=reply_text,
idempotency_key=f"reply-{inbound_id}",
)
```
If the webhook is redelivered and `send()` is called again with the same key, the outbox skips the enqueue (SQLite `UNIQUE` constraint) and marks the existing row sent.
### When the Outbox Drains
Held replies go out on **three** triggers, so operators can reason about *when* a queued message is actually re-attempted:
1. **Adapter startup** — replays anything queued before the last crash (`drain_pending()` / `drain_outbox()`).
2. **Channel recovery** *(new — Issue #4043)* — when `ChannelSupervisor` observes a channel come back after a recoverable outage (network blip, transient API error, stale socket), it fires `drain_outbox()` in the background so held replies go out promptly. Applications do not have to call anything.
3. **Lazy on next inbound turn** — the next `chat()` turn also opportunistically drains.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
subgraph "When the outbox drains"
Start[🚀 Adapter startup] --> Drain[💾 drain_outbox]
Recover[🔁 Channel recovers] --> Drain
Chat[💬 Next chat turn] --> Drain
Drain --> Send[📤 Held replies go out]
end
classDef trigger fill:#6366F1,stroke:#7C90A0,color:#fff
classDef process fill:#F59E0B,stroke:#7C90A0,color:#fff
classDef result fill:#10B981,stroke:#7C90A0,color:#fff
class Start,Recover,Chat trigger
class Drain process
class Send result
```
Call `drain_pending()` once at adapter startup to replay anything that was queued before the last crash:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# On adapter startup
succeeded, failed = await delivery.drain_pending()
# Log output: "Drained outbox: 3 sent, 0 failed"
# Or with the mixin:
succeeded, failed = await adapter.drain_outbox()
```
The drain replays oldest messages first and skips messages that have exceeded `max_attempts`. The retry gate is `max(compute_backoff(policy, attempts + 1), server_retry_after(stored_err))` — the platform's mandated wait survives across process restarts.
The hint is recovered from the **stored error string**, not the live exception. Only hints that survive `str(err)` (python-telegram-bot `RetryAfter` repr, HTTP headers folded into the error message, text-form "retry after N") are honoured on drain.
### Recovery-triggered re-drain (Issue #4043)
*Why did my held reply just go out on its own?* When a channel drops on a transient outage, the outbox deliberately **holds** deliverable replies instead of dropping them. Before, those replies sat undelivered until the next inbound `chat()` turn or a process restart. Now `ChannelSupervisor` re-drains that platform's outbox the moment the channel reconnects — see [Channel supervision → Recovery-triggered outbox re-drain](/docs/features/gateway-channel-supervision#recovery-triggered-outbox-re-drain).
The re-drain fires only when the channel comes back from **at least one** recoverable failure (`monitor.attempt > 0`). A cold, clean first start does **not** trigger it — the startup drain still covers that path.
It launches as an `asyncio.create_task(...)` and is not awaited by the supervision loop, so a slow or large re-drain can never delay the channel coming back online.
The supervisor resolves `drain_outbox` on the supervised object first, then falls back to `bot.adapter.drain_outbox`. No hook → silent no-op. An exception is logged at `WARNING` and swallowed — the outbox's own attempt-and-age dead-letter policy still governs those messages, so a failed re-drain never wedges supervision.
The re-drain is bounded by the outbox's existing attempt-and-age policy and is safe to call more than once because the outbox itself dedupes on `idempotency_key`.
Log lines from the merged code:
```
# success (only when something moved):
Channel '' recovered: re-drained outbox ( sent, failed)
# failure (swallowed — retried on the next drain):
Channel '' outbox re-drain on recovery failed (will retry on next drain):
```
Introduced in [PraisonAI PR #4046](https://github.com/MervinPraison/PraisonAI/pull/4046) (fixes [#4043](https://github.com/MervinPraison/PraisonAI/issues/4043)).
***
## Best Practices
A flaky `was_delivered` that returns `False` for an already-delivered message will cause a duplicate send. A reconciler that raises falls back to at-least-once re-send (safe, but logged at WARNING). Only opt in when your platform provides a reliable message-status API.
`supports_idempotency_token=True` is informational only — the outbox does not forward the token on resend for most adapters. Adapters relying on provider-side deduplication should also set `reconciles_unknown_send=True` to get effectively-once delivery.
**Exception — AgentMail.** `AgentMailBot.send_message` declares an explicit `idempotency_key` parameter, so the durable layer forwards the persisted outbox key on every attempt (including a post-crash resend). AgentMail deduplicates that resend at the provider for 24h, giving AgentMail provider-level dedup even without a `was_delivered` reconciler. See [Duplicate protection on retries](/docs/features/email-bot#duplicate-protection-on-retries).
The `is_recoverable_error()` function checks platform-specific patterns (e.g., Telegram's HTTP 409 conflict, rate-limit "retry after" responses) when a platform name is provided. Without it, only generic patterns are checked and some transient errors may be misclassified as permanent.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
delivery = DurableDelivery(outbox, adapter, platform="telegram")
```
When bridging a webhook to an outbound reply, derive the key from the inbound message ID. This ensures webhook redeliveries don't produce duplicate outbound sends.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# ✅ Good: stable key tied to the inbound event
await delivery.send(
channel_id=chat_id,
content=reply,
idempotency_key=f"reply-{inbound_message_id}",
)
# ❌ Bad: new UUID every call — no deduplication across retries
import uuid
await delivery.send(channel_id=chat_id, content=reply, idempotency_key=str(uuid.uuid4()))
```
Store the outbox on a persistent, local filesystem path — not `/tmp` and not a Docker tmpfs. The default suggestion is `~/.praisonai/state/outbox.sqlite`.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# ✅ Good: persistent path
outbox = OutboundQueue(path="~/.praisonai/state/outbox.sqlite")
# ❌ Bad: lost on restart or container rebuild
outbox = OutboundQueue(path="/tmp/outbox.sqlite")
```
Multiple concurrent drainers fight over the same rows via SQLite's `status = 'sending'` claim mechanism. A 5-minute claim timeout releases stale claims, but concurrent drainers still produce redundant work and log noise.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# ✅ Good: single drain at startup
async def on_start():
succeeded, failed = await delivery.drain_pending()
# ❌ Bad: two drainers in separate tasks
asyncio.create_task(delivery.drain_pending())
asyncio.create_task(delivery.drain_pending())
```
The 6-hour default is calibrated so a routine channel outage (Telegram 429 storm, Discord Cloudflare hiccup, WhatsApp API blip) never permanently loses a deliverable message, while a genuinely poisoned payload still dead-letters within the day. Only lower it when you have a specific need — e.g. `dead_letter_min_age=0` inside a test suite where you want the pre-#3521 attempt-only behaviour, or `dead_letter_min_age=60` in a synthetic burn-in that can't wait 6 hours to observe the terminal state.
The `praisonai-bot` package's dependency floor (`praisonaiagents>=1.6.152`) admits core releases that predate the shared `AttemptAndAgeDeadLetterPolicy` (first shipped 1.6.161). On those installs the queues fall back to a bundled `LocalDeadLetterPolicy` with identical semantics, so the age gate holds regardless of which core version is installed. If you customise the policy, prefer importing `AttemptAndAgeDeadLetterPolicy` from `praisonaiagents.gateway` — the import will succeed on any core ≥ 1.6.161.
***
## Proactive Path
Direct `BotOS.deliver(...)` shares the reply-path rate limiter but is fire-and-forget. When the gateway itself performs the scheduled/proactive send, it runs through a durable outbox instead.
| Property | Reply-path outbox (`delivery.send`) | Direct `BotOS.deliver(...)` | Gateway-hosted scheduled / proactive |
| - | - | - | - |
| Persistence | SQLite outbox, survives restart | Not persisted — fire-and-forget | SQLite outbox at `~/.praisonai/state/gateway_outbox.sqlite`, survives restart |
| Cross-worker dedup | Yes (`UNIQUE` on `idempotency_key`) | ❌ not supported | ✅ Yes (`UNIQUE` on the derived idem key) |
| Idempotency key | ✅ | ❌ not a parameter | ✅ derived from `sched::::` |
| Rate limiter shared | ✅ same bucket | ✅ same bucket (post PraisonAI #2578) | ✅ same bucket |
| Adapter `was_delivered` reconciliation | ✅ | ❌ not applicable | Available if the adapter opts in (same `OutboundQueue.drain(reconciler=…)` API) |
Use `delivery.send(...)` when you need durability across restarts, workers, or deduplication on the reply path. The gateway's own scheduled/proactive deliveries now get durability automatically — no extra API required. Reach for direct `BotOS.deliver(...)` (agent-initiated fire-and-forget, no gateway) only when rate-limiting is enough and durability across restart is not required.
***
## Per-Adapter Delivery Guarantee
Only adapters that declare `reconciles_unknown_send=True` and implement `was_delivered` reach effectively-once; the rest stay at-least-once by design.
| Adapter | `reconciles_unknown_send` | Delivery guarantee |
| - | :-: | - |
| Slack | ✅ | Effectively-once (via `metadata.event_type`) |
| AgentMail | ❌ | At-least-once + provider-level dedup² |
| Telegram | ❌ | At-least-once |
| Discord | ❌ | At-least-once |
| Webhook | ❌ | At-least-once |
| HTTP | ❌ | At-least-once |
² `AgentMailBot.send_message(..., idempotency_key=…)` forwards the outbox key as an AgentMail `Idempotency-Key` header (1–256 chars, 24h TTL). The outbox key survives crashes, so a re-send of the same outbox entry is deduplicated at the AgentMail provider — even without a `was_delivered` reconciler. See [Duplicate protection on retries](/docs/features/email-bot#duplicate-protection-on-retries).
Slack is the first built-in adapter to implement the reconciliation seam end-to-end:
* `SlackBot` declares `reconciles_unknown_send=True` on its `platform_capabilities`.
* `send_message(target, text, ..., idempotency_key=None)` stamps the key into the Slack message `metadata` with `event_type="praisonai_outbound"`.
* `was_delivered(target, idempotency_key)` reads back recent history (`conversations.history`, or `conversations.replies` for a threaded send) and matches the key in `metadata.event_payload`. It returns `False` on any lookup failure, falling back to at-least-once.
```
Before: sending → (restart) → recovered → re-dispatched blindly → duplicate
After: sending → (restart) → recovered → reconciler confirms → marked sent (no duplicate)
```
Channels whose platform cannot confirm delivery remain at-least-once by design — that is now an explicit per-channel fact, not a silent default. See [Bot Platform Capabilities](/docs/features/bot-platform-capabilities).
For those at-least-once channels you can opt into visible labelling of crash-recovered re-sends with `mark_recovered=True`. The recipient sees a `♻️ Recovered reply — the gateway restarted during delivery, so this may be a duplicate.` prefix instead of a silent possible-duplicate. See [Labelling the at-least-once fallback](/docs/docs/features/bot-platform-capabilities#labelling-the-at-least-once-fallback).
***
## Related
Per-conversation FIFO ordering on this outbox — keep messages to a chat in order under retries
Suppress permanently-dead channels — the permanent-failure complement to durable retry
Inbound counterpart — deduplicate webhook redeliveries and recover in-flight messages
Dead-letter queue for failed inbound message processing
Inspect, retry, and purge individual outbox entries — `praisonai gateway outbox`
Configure outbound resilience for all six bot channels
Live-edit streaming UX for bot responses
Top-level guide to building bots with PraisonAI
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.