Model Fallback

Keep your agent answering when a model is overloaded by automatically falling back to alternates

Model Fallback keeps your agent answering by automatically retrying on alternate models when the primary model is overloaded or unavailable.

from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig

agent = Agent(
    name="assistant",
    instructions="You are a helpful assistant",
    model=LLMConfig(
        model="gpt-4o",
        fallback_models=["anthropic/claude-3-5-sonnet", "gpt-4o-mini"],
    ),
)
agent.start("Answer even when the primary model is overloaded")

The user sends a prompt; the agent retries on fallback models when the primary returns errors.

Quick Start

1. One-line resilience

   from praisonaiagents import Agent
   from praisonaiagents.config import LLMConfig
   
   agent = Agent(
       instructions="You are a helpful assistant",
       model=LLMConfig(
           model="gpt-4o",
           fallback_models=["claude-3-5-sonnet", "gpt-4o-mini"],
       ),
   )
   agent.start("Summarise today's news")

2. Cross-provider chain

   Use LiteLLM-style prefixes when mixing providers:

   from praisonaiagents import Agent
   from praisonaiagents.config import LLMConfig
   
   agent = Agent(
       model=LLMConfig(
           model="openai/gpt-4o",
           fallback_models=["anthropic/claude-3-5-sonnet", "openai/gpt-4o-mini"],
       ),
   )

3. Notice the switch

   Subscribe to HookEvent.MODEL_FALLBACK to react the moment the primary model is swapped out:

   from praisonaiagents import Agent
   from praisonaiagents.config import LLMConfig
   from praisonaiagents.hooks import HookRegistry, HookEvent, HookResult
   
   registry = HookRegistry()
   
   @registry.on(HookEvent.MODEL_FALLBACK)
   def notice(evt):
       print(f"[fallback] {evt.from_model} → {evt.to_model} ({evt.reason_category})")
       return HookResult.allow()
   
   agent = Agent(
       instructions="You are a helpful assistant",
       model=LLMConfig(
           model="gpt-4o",
           fallback_models=["claude-3-5-sonnet", "gpt-4o-mini"],
       ),
       hooks=registry,
   )
   agent.start("Answer even during a provider outage")

How It Works

On transient errors (503, timeout, model overloaded), the agent retries the same turn against the next model in fallback_models. Successful calls stay on the primary model.

Failover fires on retryable errors classified by the LLM error classifier and covers every turn shape — non-streaming, streaming, tool-iteration turns, reflection turns, and their async equivalents. A 503 on a streaming chunk pushes the same turn to the next model in fallback_models; the user sees continuous output, not a failure.

Observing the Switch

Subscribe to HookEvent.MODEL_FALLBACK (or the MODEL_FALLBACK stream event) to react the moment the primary model is swapped out — for alerts, metrics, a UI notice, or a per-user notification.

ModelFallbackInput fields

Hook subscription

from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig
from praisonaiagents.hooks import HookRegistry, HookEvent, HookResult

registry = HookRegistry()

@registry.on(HookEvent.MODEL_FALLBACK)
def on_fallback(evt):
    print(f"[fallback] {evt.from_model} → {evt.to_model}")
    print(f"           reason={evt.reason_category} index={evt.fallback_index}")
    return HookResult.allow()

agent = Agent(
    instructions="You are a helpful assistant",
    model=LLMConfig(
        model="gpt-4o",
        fallback_models=["claude-3-5-sonnet", "gpt-4o-mini"],
    ),
    hooks=registry,
)
agent.start("Answer even during a provider outage")

Stream event

When a stream callback is active, the same swap emits a StreamEventType.MODEL_FALLBACK event carrying the four fields in metadata, plus agent_id, session_id, and run_id.

from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig
from praisonaiagents.streaming import StreamEvent, StreamEventType

agent = Agent(
    instructions="You are a helpful assistant",
    model=LLMConfig(
        model="gpt-4o",
        fallback_models=["claude-3-5-sonnet", "gpt-4o-mini"],
    ),
    stream=True,
)

def on_event(event: StreamEvent):
    if event.type == StreamEventType.MODEL_FALLBACK:
        m = event.metadata
        print(f"[stream] switched {m['from_model']} → {m['to_model']} ({m['reason_category']})")

agent.stream_emitter.add_callback(on_event)
agent.start("Answer even during a provider outage")

Sync and async paths

Both call sites emit — the sync _chat_completion path and the async _handle_async_llm_error path — so the hook fires whether you run agent.start(...) or await agent.astart(...).

@registry.on(HookEvent.MODEL_FALLBACK)
def on_fallback(evt):
    print(f"[fallback] {evt.from_model} → {evt.to_model}")
    return HookResult.allow()

agent.start("Answer even during a provider outage")

@registry.on(HookEvent.MODEL_FALLBACK)
async def on_fallback(evt):
    print(f"[fallback] {evt.from_model} → {evt.to_model}")
    return HookResult.allow()

await agent.astart("Answer even during a provider outage")

Shared-Agent async safety

On a shared Agent, a fallback on one turn does not affect the model, cost, or observability attribution of concurrent turns.

Many users share a single Agent across asyncio.gather (see Thread-Safe Agent State → Chat History). Before this fix, a mid-fallback shared-state mutation could silently flip a concurrent turn to the fallback model — wrong cost, wrong logs, wrong hook payloads. Both paths now keep each turn on its own model.

import asyncio
from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig

agent = Agent(
    instructions="You are helpful.",
    model=LLMConfig(
        model="gpt-4o",
        fallback_models=["claude-3-5-sonnet"],
    ),
)

async def main():
    # Same Agent, two concurrent turns.
    # If turn A hits a 503 and falls back to claude-3-5-sonnet,
    # turn B still dispatches on gpt-4o with correct cost/observability.
    await asyncio.gather(
        agent.astart("Summarise today's news"),
        agent.astart("Draft a follow-up email"),
    )

asyncio.run(main())

Async path stays clean. The fallback retry runs against a throwaway per-call dispatcher bound to the fallback model; the shared model state is never mutated, so a concurrent turn keeps dispatching on its original model even while this turn is suspended on an await. No lock is taken on the async path for ordinary turns — non-fallback turns stay fully concurrent.

Sync path stays clean. The sync fallback still recurses through the full chat wrapper (which reads the current model for BEFORE_LLM hook attribution and budget), so it temporarily swaps the model. That mutate/restore window is now serialised by a lazily-created per-instance lock; only fallback turns contend on it — non-fallback turns never acquire it.

Custom LLMs & provider/model strings

Fallback works for every way of passing an LLM — provider/model strings, llm={...} dicts, agents built with a base_url, and model objects — on sync, async, and streaming turns.

from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig

agent = Agent(
    instructions="You are a helpful assistant",
    llm=LLMConfig(
        model="anthropic/claude-3-5-sonnet",
        fallback_models=["openai/gpt-4o-mini"],
    ),
)
agent.start("Answer even during an Anthropic outage")

from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig

agent = Agent(
    instructions="You are a helpful assistant",
    llm=LLMConfig(
        model="model-a",
        fallback_models=["model-b"],
        base_url="http://localhost:11434/v1",  # kept for bare-name fallbacks
        api_key="ollama",
    ),
)
agent.start("Hello from the custom server")

from praisonaiagents import Agent
from praisonaiagents.llm import LLM

agent = Agent(instructions="You are a helpful assistant", llm=LLM(model="ollama/llama3"))
agent.fallback_models = ["openai/gpt-4o-mini"]  # set after construction for model objects
agent.start("Answer even if the primary is overloaded")

Configuration Options

Model Fallback is configured through LLMConfig — set model for the primary and fallback_models for the ordered backup chain.

from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig

agent = Agent(
    instructions="You are a helpful assistant",
    model=LLMConfig(
        model="gpt-4o",
        fallback_models=["claude-3-5-sonnet", "gpt-4o-mini"],
    ),
)
agent.start("Hello!")

For model-object agents (llm=my_llm), assign agent.fallback_models = [...] after construction. Entries can be model-id strings or ready-made LLM instances. A string entry becomes a fresh LLM(model=...) that reads credentials from the provider’s environment variable; pass an LLM(...) instance instead when the fallback needs its own base_url or api_key.

The decision diagram below shows which chain shape fits which scenario.

Full LLMConfig surface is covered by the auto-generated SDK reference — see Hook Events for the paired MODEL_FALLBACK hook.

Best Practices

Related