This page covers task/workflow-level reliability (retry jitter,
workflow_timeout, fail_on_callback_error). If you are looking for the gateway-level reliability preset (reliability="production" on BotOS / gateway YAML / CLI), see Gateway Reliability Preset — it is a separate feature that composes drain and admission control.This page covers task/workflow reliability (retry, timeouts, failure policies). For gateway reliability (graceful drain + inbound admission control presets), see Gateway Reliability.
Quick Start
How It Works
| Component | Purpose |
|---|---|
| Retry jitter | Desynchronises multi-agent retries on rate limits |
| Workflow timeout | Hard kill after specified seconds (sync and async) |
| Failure policies | Surface or swallow callback and memory exceptions |
Configuration Options
| Param | Type | Default | Effect when True |
|---|---|---|---|
fail_on_callback_error | bool | False | Re-raises exceptions inside task.callback |
fail_on_memory_error | bool | False | Re-raises memory-store failures |
workflow_timeout | int | None | Seconds before workflow cancellation |
Retry categories
| Error category | Behaviour | Cap |
|---|---|---|
RATE_LIMIT | exp backoff ×3 + full jitter | 60s |
TRANSIENT | exp backoff ×2 + full jitter | 30s |
CONTEXT_LIMIT | deterministic 0.5s | 0.5s |
AUTH / INVALID_REQUEST / PERMANENT | no retry | — |
Common Patterns
Strict CI mode
Lenient production mode
Multi-agent fan-out
Best Practices
Set workflow_timeout for external API calls
Set workflow_timeout for external API calls
Network calls can hang indefinitely. Use 60s for quick tasks, 300s for multi-step workflows.
Use fail_on_callback_error=True in tests
Use fail_on_callback_error=True in tests
Tests should surface bugs immediately; production should log and continue unless the callback is critical.
Do not manually retry rate limits
Do not manually retry rate limits
The SDK already applies exponential backoff with jitter. Catching
RateLimitError and sleeping duplicates that work.Track non_fatal_errors in monitoring
Track non_fatal_errors in monitoring
Non-fatal errors indicate latent issues — increment metrics and log them even when the workflow completes.
Related
Task Retry Policy
Per-task retry with exponential backoff
Workflow Error Recovery
Recover from workflow failures gracefully

