Skip to main content
Stream AI responses token-by-token as they’re generated, instead of waiting for the complete response.
The user sends a prompt; tokens stream back incrementally instead of waiting for the full reply.

Quick Start

1

Install

2

Auto-detect (Default)

By default the SDK tries streaming first and silently falls back to non-streaming if your provider’s sync client doesn’t support it — multi-agent workflows on providers like Deepseek now Just Work.
3

Force Streaming


Choosing the Right Method

MethodStreamsDisplayBest For
start() (auto-detect)🎯 Auto✅ AutoRecommended — works everywhere
start(stream=True)✅ Yes✅ AutoForce streaming, interactive chat
iter_stream()✅ Always❌ NoApp integration, custom UIs
run()❌ No❌ NoProduction, batch processing
chat(stream=True)ConfigurableConfigurableLow-level control

Common Patterns

Terminal Streaming

App Integration with iter_stream()

Best for integrating into your own application — yields raw chunks with no display overhead.
The interactive CLI (praisonai chat / praisonai code) consumes iter_stream() directly since PR #2906 — every token you see in the terminal is a real model delta, not a post-hoc word replay. If the provider does not stream, the CLI falls back to a single non-streamed chat() call and prints the completed answer as one block. See Interactive TUI.

Streaming with Callbacks

Hook into every streaming event for fine-grained control.

FastAPI SSE Integration

Pipe streaming tokens directly to a web client using Server-Sent Events.

Async Streaming


Streaming with Tools

When your agent uses tools, streaming happens in two phases: the initial response that decides to call tools, and a follow-up response that synthesizes the tool results. Tools can also emit incremental progress while they run using emit_tool_progress() — these arrive as TOOL_PROGRESS events in your stream callback before the tool returns its result. See Tool Progress Streaming.
Both phases go through the same retry-wrapped LLM path, so transient rate-limit or network errors are retried automatically without any caller intervention.

Error Handling in the Stream

If the LLM call fails after retries, the stream ends with a visible error sentence instead of silently dropping. You may receive this exact sentinel string:
PartMeaning
ref: followup-<timestamp>Correlation ID logged server-side — share this when reporting issues
Please retryRetries already ran internally; another attempt may succeed if the root cause was transient
reducing prompt sizeCommon root cause is context-length or provider capacity errors
Detect the error sentinel in your stream consumer:
The initial LLM call and the follow-up LLM call (after tool execution) now share the same retry and rate-limiting behavior — users no longer need to add their own retry wrapper around streaming + tools.

StreamEvent Protocol

Every streaming chunk emits a StreamEvent with full context.
EventWhen
REQUEST_STARTBefore API call
HEADERS_RECEIVEDHTTP 200 arrives
FIRST_TOKENFirst content delta (TTFT marker)
DELTA_TEXTEach text chunk
DELTA_TOOL_CALLTool call streaming
LAST_TOKENFinal content delta
STREAM_ENDStream completed

Metrics

Track Time To First Token (TTFT) and throughput.
MetricDescription
TTFTTime from request to first token (provider latency)
Stream DurationFrom first to last token
Total TimeEnd-to-end request time
Tokens/sToken generation rate

Key Concepts

Time To First Token (TTFT)

TTFT is the time before the first token arrives. This is provider latency — the model must process your prompt before generating. Streaming does NOT reduce TTFT, but it shows progress immediately.

Streaming vs Non-Streaming

ModeBehaviorUse Case
stream=None (default)Try streaming, fall back to non-streaming if unsupportedRecommended — works across all providers
stream=TrueForce streaming (errors on sync adapters that don’t support it)When you definitely want tokens
stream=FalseForce non-streamingBatch jobs, structured output, sync providers
Multi-agent output=None (default)stream=True — auto streamingDefault streaming behavior
Multi-agent output="verbose" / "minimal"stream=True by defaultDisplay with streaming
Multi-agent output=MultiAgentOutputConfig(stream=False)Disable streaming for all team agentsOpt-out for sync-only providers
Sync vs Async Adapters: Async methods (achat, astart, _execute_unified_achat_completion) still default to stream=True because async adapters universally support streaming. Sync methods (chat, start, run) use the new smart-fallback default. Some adapters (e.g., sync OpenAI/Deepseek adapter) currently do NOT support sync streaming and will trigger the fallback.Multi-agent teams (AgentTeam, PraisonAIAgents) default to stream=True for "verbose" and "minimal" presets. Use output=MultiAgentOutputConfig(stream=False) or output=["verbose", {"stream": False}] to opt out for sync-only providers.

CLI Usage


Best Practices

Omit the stream argument (or pass stream=None) and the SDK will choose streaming where supported and silently fall back where it isn’t. Only override when you have a specific reason.
iter_stream() yields raw chunks with zero display overhead — ideal for piping into FastAPI, WebSocket, or custom UIs.
start() handles display automatically. Pass stream=True for real-time token output in interactive sessions.
High TTFT indicates model or network issues. Use StreamMetrics to track and optimize.
Two layers of error handling. Callback exceptions are still caught by the emitter to avoid breaking the stream — log them inside your callback. LLM call failures, however, are now retried automatically and, on persistent failure, surface as a visible [Error: ... (ref: ...)] sentence at the end of the stream — check for this sentinel when consuming iter_stream().

Troubleshooting

”Streaming seems to buffer before showing anything”

This is TTFT, not buffering. The model is generating the first token. Check:
  • Model complexity (larger models have higher TTFT)
  • Prompt length (longer prompts take longer to process)
  • Network latency to the API

”Tokens appear in chunks, not one at a time”

Normal. Providers may batch tokens for efficiency.

”Stream ends with [Error: Failed to generate final response after tool execution (ref: followup-...)]

The follow-up LLM call (the one that synthesizes tool results into a final answer) failed after the built-in retries. Common causes:
  • Persistent rate limit — pair streaming with a Rate Limiter at higher RPM, or back off the caller.
  • Context-length overflow — reduce conversation history or tool-result size.
  • Provider outage — include the ref: ID when reporting. The internal log line (ref=..., model=..., error=...) makes it searchable.

”Streaming is not supported in sync OpenAIAdapter” / Deepseek multi-agent crash

Fixed with single-agent smart-fallback in PR #1734. For multi-agent teams that use sync-only providers, explicitly disable streaming with output=["verbose", {"stream": False}] or similar. See Multi-Agent Output for configuration options.

Bot Streaming Replies

Live draft replies on messaging platforms using streaming events

Multi-Agent Output

Configure streaming and display for agent teams

Output & Display

Single-agent output formatting options

Async

Async agent execution

Rate Limiter

Control request rates across initial and follow-up LLM calls