Quick Start
Unlock 100+ Providers with LiteLLM
Install LiteLLM to use Anthropic, Google, Azure, Bedrock, Ollama, and more:Switch to Anthropic Claude:Switch to Google Gemini:Switch to a local Ollama model:
Programmatic Use (Sync & Async)
Both sync and async paths honour the LiteLLM → OpenAI fallback:Async version:
Calling from an existing event loop (FastAPI, Jupyter)
AutoGenerator.generate() can now be called from inside a running event loop — it offloads to a worker thread automatically instead of raising RuntimeError: This event loop is already running.await generator.agenerate() when you are already in an async context — it avoids the thread-offload overhead. The sync .generate() form is provided as a safe fallback.How It Works
| Step | What happens |
|---|---|
| 1. LiteLLM attempt | Calls litellm.completion(model=..., response_format=PydanticModel) |
| 2. Runtime fallback | If LiteLLM raises (auth error, timeout, unsupported model), logs a warning and continues |
| 3. OpenAI fallback | Calls client.beta.chat.completions.parse(...) using the OpenAI SDK |
| 4. No providers | Raises ImportError with install instructions |
The fallback warning message is:
LiteLLM structured completion failed (...); falling back to OpenAI SDK.
Watch your logs for this — it means LiteLLM is degrading silently.Which Provider Should I Use?
Install Matrix
| Installed | Auto mode behaviour |
|---|---|
openai only | OpenAI SDK used directly via beta.chat.completions.parse |
litellm only | LiteLLM used; if it raises, the call fails (no fallback) |
| both | LiteLLM first; on runtime error, falls back to OpenAI with a warning |
| neither | ImportError: Structured output requires either litellm or openai... |
Configuration
| Environment variable | What it does |
|---|---|
OPENAI_API_KEY | API key used by both LiteLLM and OpenAI paths |
OPENAI_MODEL_NAME | Model passed to whichever provider runs. For LiteLLM use prefixed names: anthropic/claude-sonnet-4, gemini/gemini-2.0-flash, ollama/llama3 |
OPENAI_API_BASE | Forwarded as base_url to the OpenAI client (overrides default endpoint) |
ANTHROPIC_API_KEY | Required when OPENAI_MODEL_NAME=anthropic/... |
GEMINI_API_KEY | Required when OPENAI_MODEL_NAME=gemini/... |
Auto Module SDK Reference
Full API reference for AutoGenerator, WorkflowAutoGenerator, and BaseAutoGenerator
Common Patterns
Switch from OpenAI to Anthropic:Best Practices
Install both litellm and openai in production
Install both litellm and openai in production
Install both packages so transient LiteLLM failures (rate limits, network blips, provider outages) don’t break agent generation:LiteLLM runs first; OpenAI silently catches any runtime errors.
Watch logs for the fallback warning
Watch logs for the fallback warning
The warning
LiteLLM structured completion failed (...); falling back to OpenAI SDK. means LiteLLM silently degraded. Check:- Is your provider API key set correctly?
- Does the model support
response_format(structured output)? - Is the provider having an outage?
Pin model names in config_list for concurrent generators
Pin model names in config_list for concurrent generators
If your code runs multiple
AutoGenerator instances in parallel, set model names in config_list rather than OPENAI_MODEL_NAME to avoid environment variable races:Do not import litellm at module scope
Do not import litellm at module scope
Lazy loading is on by default —
litellm is never imported at module load time. Do not add import litellm at the top of your files. Let AutoGenerator pull it in on demand. This keeps startup fast and avoids import-time side effects.Related
Auto Mode CLI
Command-line reference for
praisonai --autoAutoAgents
Automatically create and run agents from a prompt
Structured Output
Control agent output format with Pydantic models
YAML Workflows
The agents.yaml format generated by auto mode

