Skip to main content
Stream AI responses token-by-token as they’re generated, instead of waiting for the complete response.
The user sends a prompt; tokens stream back incrementally instead of waiting for the full reply.

Quick Start

1

Install

2

Auto-detect (Default)

By default the SDK tries streaming first and silently falls back to non-streaming if your provider’s sync client doesn’t support it — multi-agent workflows on providers like Deepseek now Just Work.
3

Force Streaming


Choosing the Right Method


Common Patterns

Terminal Streaming

App Integration with iter_stream()

Best for integrating into your own application — yields raw chunks with no display overhead.
The interactive CLI (praisonai chat / praisonai code) consumes iter_stream() directly since PR #2906 — every token you see in the terminal is a real model delta, not a post-hoc word replay. If the provider does not stream, the CLI falls back to a single non-streamed chat() call and prints the completed answer as one block. See Interactive TUI.

Streaming with Callbacks

Hook into every streaming event for fine-grained control.

FastAPI SSE Integration

Pipe streaming tokens directly to a web client using Server-Sent Events.

Async Streaming


Streaming with Tools

When your agent uses tools, streaming happens in two phases: the initial response that decides to call tools, and a follow-up response that synthesizes the tool results. Tools can also emit incremental progress while they run using emit_tool_progress() — these arrive as TOOL_PROGRESS events in your stream callback before the tool returns its result. See Tool Progress Streaming.
Both phases go through the same retry-wrapped LLM path, so transient rate-limit or network errors are retried automatically without any caller intervention.

Error Handling in the Stream

If the LLM call fails after retries, the stream ends with a visible error sentence instead of silently dropping. You may receive this exact sentinel string:
Detect the error sentinel in your stream consumer:
The initial LLM call and the follow-up LLM call (after tool execution) now share the same retry and rate-limiting behavior — users no longer need to add their own retry wrapper around streaming + tools.

StreamEvent Protocol

Every streaming chunk emits a StreamEvent with full context. TODO_UPDATED fires on every todo_add / todo_update — subscribe to render a live checklist. See Live Todo Streaming. MODEL_FALLBACK fires the moment the runtime swaps models. Subscribe to render a “switched to backup” notice, or pair it with the HookEvent.MODEL_FALLBACK hook for programmatic reactions. See Model Fallback → Observing the Switch.

Reacting to Retries

When the agent hits a rate limit or transient error, it emits a RETRY event before it waits, then retries. The signal fires on both the sync and async retry loops, and reaches both add_callback(sync_fn) and add_async_callback(async_fn) consumers. Emission is guarded so there is zero overhead when nothing is listening. The RETRY event carries its details in event.metadata:
RETRY events reach every registered callback, sync or async. You can mix add_callback(fn) and add_async_callback(afn) on the same emitter — both fire for every retry. Requires PraisonAI 2026-07-23 or later (PR #3325); earlier versions dispatched async retries via the sync path only and skipped async-only consumers.
Same signal, two consumers — the ON_RETRY hook is for programmatic control, while the RETRY stream event feeds live UIs and stream-json pipelines. The run.retry NDJSON event is the CLI-facing form of this same event.

Metrics

Track Time To First Token (TTFT) and throughput.

Key Concepts

Time To First Token (TTFT)

TTFT is the time before the first token arrives. This is provider latency — the model must process your prompt before generating. Streaming does NOT reduce TTFT, but it shows progress immediately.

Streaming vs Non-Streaming

Sync vs Async Adapters: Async methods (achat, astart, _execute_unified_achat_completion) still default to stream=True because async adapters universally support streaming. Sync methods (chat, start, run) use the new smart-fallback default. Some adapters (e.g., sync OpenAI/Deepseek adapter) currently do NOT support sync streaming and will trigger the fallback.Multi-agent teams (AgentTeam, PraisonAIAgents) default to stream=True for "verbose" and "minimal" presets. Use output=MultiAgentOutputConfig(stream=False) or output=["verbose", {"stream": False}] to opt out for sync-only providers.

Streaming and guardrails

Output guardrails validate the full response before it is returned to the caller. Token-level streaming yields tokens as the model produces them — there is no full response to validate until streaming completes, and buffering tokens until then would break the streaming contract. Since PraisonAI PR #3632, the SDK emits a clear warning as soon as streaming begins on an agent with a guardrail attached:
If your production path relies on guardrail validation, do not consume streamed output as if it were validated. Grep for output guardrail is not applied to streamed responses in your logs after upgrading — every match is a caller that thought streaming was validated when it wasn’t.
See Guardrails → Streaming bypass for the guardrail-side view.

Managed-backend streaming (backend=)

When Agent(backend=<ManagedBackendProtocol>) is used, start(stream=True) delegates to backend.stream(prompt) and yields each chunk as the backend emits it — even from ordinary sync code with no running event loop. Prior to PraisonAI PR #3908, the sync (“no running event loop”) branch buffered internally, so the generator yielded only after the whole response was produced; since #3908 it is truly incremental, mirroring the running-loop branch.
Closing the generator early signals the producer to stop. The internal queue is bounded (maxsize=64), so an abandoned consumer applies back-pressure instead of leaking memory, and the producer thread is a daemon — a stalled backend cannot block interpreter shutdown.
This path streams like any other start(stream=True) call — see Choosing the Right Method and Managed Runtime Protocol for the backend contract.

CLI Usage


Best Practices

Omit the stream argument (or pass stream=None) and the SDK will choose streaming where supported and silently fall back where it isn’t. Only override when you have a specific reason.
iter_stream() yields raw chunks with zero display overhead — ideal for piping into FastAPI, WebSocket, or custom UIs.
start() handles display automatically. Pass stream=True for real-time token output in interactive sessions.
High TTFT indicates model or network issues. Use StreamMetrics to track and optimize.
Two layers of error handling. Callback exceptions are still caught by the emitter to avoid breaking the stream — log them inside your callback. LLM call failures, however, are now retried automatically and, on persistent failure, surface as a visible [Error: ... (ref: ...)] sentence at the end of the stream — check for this sentinel when consuming iter_stream().
iter_stream() and start(stream=True) bypass output guardrails — the SDK logs a warning when both are combined. Use chat() (or the non-streaming path of start() / astart()) when you need validated output. See Streaming and guardrails.

Troubleshooting

”Streaming seems to buffer before showing anything”

This is TTFT, not buffering. The model is generating the first token. Check:
  • Model complexity (larger models have higher TTFT)
  • Prompt length (longer prompts take longer to process)
  • Network latency to the API

”Tokens appear in chunks, not one at a time”

Normal. Providers may batch tokens for efficiency.

”Stream ends with [Error: Failed to generate final response after tool execution (ref: followup-...)]

The follow-up LLM call (the one that synthesizes tool results into a final answer) failed after the built-in retries. Common causes:
  • Persistent rate limit — pair streaming with a Rate Limiter at higher RPM, or back off the caller.
  • Context-length overflow — reduce conversation history or tool-result size.
  • Provider outage — include the ref: ID when reporting. The internal log line (ref=..., model=..., error=...) makes it searchable.

”Streaming is not supported in sync OpenAIAdapter” / Deepseek multi-agent crash

Fixed with single-agent smart-fallback in PR #1734. For multi-agent teams that use sync-only providers, explicitly disable streaming with output=["verbose", {"stream": False}] or similar. See Multi-Agent Output for configuration options.

Bot Streaming Replies

Live draft replies on messaging platforms using streaming events

Multi-Agent Output

Configure streaming and display for agent teams

Output & Display

Single-agent output formatting options

Async

Async agent execution

Rate Limiter

Control request rates across initial and follow-up LLM calls

Guardrails

Why streaming bypasses output guardrails, and how to opt in to validation

Live Todo Streaming

Render a live agent checklist from TODO_UPDATED events

Managed Runtime Protocol

backend= streams each chunk incrementally from sync code