Quick Start
1
Install
2
Auto-detect (Default)
3
Force Streaming
Choosing the Right Method
Common Patterns
Terminal Streaming
App Integration with iter_stream()
Best for integrating into your own application — yields raw chunks with no display overhead.
praisonai chat / praisonai code) consumes iter_stream() directly since PR #2906 — every token you see in the terminal is a real model delta, not a post-hoc word replay. If the provider does not stream, the CLI falls back to a single non-streamed chat() call and prints the completed answer as one block. See Interactive TUI.
Streaming with Callbacks
Hook into every streaming event for fine-grained control.FastAPI SSE Integration
Pipe streaming tokens directly to a web client using Server-Sent Events.Async Streaming
Streaming with Tools
When your agent uses tools, streaming happens in two phases: the initial response that decides to call tools, and a follow-up response that synthesizes the tool results. Tools can also emit incremental progress while they run usingemit_tool_progress() — these arrive as TOOL_PROGRESS events in your stream callback before the tool returns its result. See Tool Progress Streaming.
Error Handling in the Stream
If the LLM call fails after retries, the stream ends with a visible error sentence instead of silently dropping. You may receive this exact sentinel string:
Detect the error sentinel in your stream consumer:
The initial LLM call and the follow-up LLM call (after tool execution) now share the same retry and rate-limiting behavior — users no longer need to add their own retry wrapper around streaming + tools.
StreamEvent Protocol
Every streaming chunk emits aStreamEvent with full context.
TODO_UPDATED fires on every todo_add / todo_update — subscribe to render a live checklist. See Live Todo Streaming.
MODEL_FALLBACK fires the moment the runtime swaps models. Subscribe to render a “switched to backup” notice, or pair it with the HookEvent.MODEL_FALLBACK hook for programmatic reactions. See Model Fallback → Observing the Switch.
Reacting to Retries
When the agent hits a rate limit or transient error, it emits aRETRY event before it waits, then retries. The signal fires on both the sync and async retry loops, and reaches both add_callback(sync_fn) and add_async_callback(async_fn) consumers. Emission is guarded so there is zero overhead when nothing is listening.
The RETRY event carries its details in event.metadata:
- Sync callback
- Async callback
RETRY events reach every registered callback, sync or async. You can mix add_callback(fn) and add_async_callback(afn) on the same emitter — both fire for every retry. Requires PraisonAI 2026-07-23 or later (PR #3325); earlier versions dispatched async retries via the sync path only and skipped async-only consumers.Same signal, two consumers — the
ON_RETRY hook is for programmatic control, while the RETRY stream event feeds live UIs and stream-json pipelines. The run.retry NDJSON event is the CLI-facing form of this same event.Metrics
Track Time To First Token (TTFT) and throughput.Key Concepts
Time To First Token (TTFT)
Streaming vs Non-Streaming
Sync vs Async Adapters: Async methods (
achat, astart, _execute_unified_achat_completion) still default to stream=True because async adapters universally support streaming. Sync methods (chat, start, run) use the new smart-fallback default. Some adapters (e.g., sync OpenAI/Deepseek adapter) currently do NOT support sync streaming and will trigger the fallback.Multi-agent teams (AgentTeam, PraisonAIAgents) default to stream=True for "verbose" and "minimal" presets. Use output=MultiAgentOutputConfig(stream=False) or output=["verbose", {"stream": False}] to opt out for sync-only providers.Streaming and guardrails
Output guardrails validate the full response before it is returned to the caller. Token-level streaming yields tokens as the model produces them — there is no full response to validate until streaming completes, and buffering tokens until then would break the streaming contract. Since PraisonAI PR #3632, the SDK emits a clear warning as soon as streaming begins on an agent with a guardrail attached:
See Guardrails → Streaming bypass for the guardrail-side view.
Managed-backend streaming (backend=)
When Agent(backend=<ManagedBackendProtocol>) is used, start(stream=True) delegates to backend.stream(prompt) and yields each chunk as the backend emits it — even from ordinary sync code with no running event loop.
Prior to PraisonAI PR #3908, the sync (“no running event loop”) branch buffered internally, so the generator yielded only after the whole response was produced; since #3908 it is truly incremental, mirroring the running-loop branch.
Closing the generator early signals the producer to stop. The internal queue is bounded (
maxsize=64), so an abandoned consumer applies back-pressure instead of leaking memory, and the producer thread is a daemon — a stalled backend cannot block interpreter shutdown.start(stream=True) call — see Choosing the Right Method and Managed Runtime Protocol for the backend contract.
CLI Usage
Best Practices
Let the SDK pick streaming mode
Let the SDK pick streaming mode
Omit the
stream argument (or pass stream=None) and the SDK will choose streaming where supported and silently fall back where it isn’t. Only override when you have a specific reason.Use iter_stream() for app integration
Use iter_stream() for app integration
iter_stream() yields raw chunks with zero display overhead — ideal for piping into FastAPI, WebSocket, or custom UIs.Use start(stream=True) for terminal
Use start(stream=True) for terminal
start() handles display automatically. Pass stream=True for real-time token output in interactive sessions.Monitor TTFT for performance
Monitor TTFT for performance
High TTFT indicates model or network issues. Use
StreamMetrics to track and optimize.Handle errors in callbacks
Handle errors in callbacks
Two layers of error handling. Callback exceptions are still caught by the emitter to avoid breaking the stream — log them inside your callback. LLM call failures, however, are now retried automatically and, on persistent failure, surface as a visible
[Error: ... (ref: ...)] sentence at the end of the stream — check for this sentinel when consuming iter_stream().Do not stream when a guardrail must run
Do not stream when a guardrail must run
iter_stream() and start(stream=True) bypass output guardrails — the SDK logs a warning when both are combined. Use chat() (or the non-streaming path of start() / astart()) when you need validated output. See Streaming and guardrails.Troubleshooting
”Streaming seems to buffer before showing anything”
This is TTFT, not buffering. The model is generating the first token. Check:- Model complexity (larger models have higher TTFT)
- Prompt length (longer prompts take longer to process)
- Network latency to the API
”Tokens appear in chunks, not one at a time”
Normal. Providers may batch tokens for efficiency.”Stream ends with [Error: Failed to generate final response after tool execution (ref: followup-...)]”
The follow-up LLM call (the one that synthesizes tool results into a final answer) failed after the built-in retries. Common causes:
- Persistent rate limit — pair streaming with a Rate Limiter at higher RPM, or back off the caller.
- Context-length overflow — reduce conversation history or tool-result size.
- Provider outage — include the
ref:ID when reporting. The internal log line (ref=..., model=..., error=...) makes it searchable.
”Streaming is not supported in sync OpenAIAdapter” / Deepseek multi-agent crash
Fixed with single-agent smart-fallback in PR #1734. For multi-agent teams that use sync-only providers, explicitly disable streaming withoutput=["verbose", {"stream": False}] or similar. See Multi-Agent Output for configuration options.
Related
Bot Streaming Replies
Live draft replies on messaging platforms using streaming events
Multi-Agent Output
Configure streaming and display for agent teams
Output & Display
Single-agent output formatting options
Async
Async agent execution
Rate Limiter
Control request rates across initial and follow-up LLM calls
Guardrails
Why streaming bypasses output guardrails, and how to opt in to validation
Live Todo Streaming
Render a live agent checklist from
TODO_UPDATED eventsManaged Runtime Protocol
backend= streams each chunk incrementally from sync code
