Failover

Automatic fallback between LLM providers for reliability and cost optimization

Failover automatically switches between LLM providers when one fails, keeping agents running during API outages or rate limits.

import os
from praisonaiagents import Agent, AuthProfile, FailoverConfig, FailoverManager

manager = FailoverManager(FailoverConfig(max_retries=3))
manager.add_profile(AuthProfile(
    name="openai", provider="openai",
    api_key=os.getenv("OPENAI_API_KEY"), priority=1,
))
manager.add_profile(AuthProfile(
    name="anthropic", provider="anthropic",
    api_key=os.getenv("ANTHROPIC_API_KEY"), priority=2,
))

agent = Agent(name="assistant", llm={"model": "gpt-4o-mini", "failover_manager": manager})
agent.start("Hello!")

import { FailoverManager } from 'praisonai';

const manager = new FailoverManager({
  providers: ['openai', 'anthropic'],
  strategy: 'priority',
  maxRetries: 3,
});

let provider = manager.getNextProvider(); // 'openai' under the priority strategy

// Rotate on a retryable failure (e.g. a 401 that a fresh key may fix):
manager.markFailed(provider!);
provider = manager.getNextProvider(); // now falls through to 'anthropic'

manager.markHealthy('openai', 120); // back online, 120ms latency
console.log(manager.getStatuses());

The user sends a message; if the primary LLM profile fails, the agent tries the next profile in the failover chain.

Quick Start

1. Run an agent with failover

   import os
   from praisonaiagents import Agent, AuthProfile, FailoverConfig, FailoverManager
   
   manager = FailoverManager(FailoverConfig(max_retries=3, exponential_backoff=True))
   manager.add_profile(AuthProfile(
       name="openai", provider="openai",
       api_key=os.getenv("OPENAI_API_KEY"), priority=1,
   ))
   manager.add_profile(AuthProfile(
       name="anthropic", provider="anthropic",
       api_key=os.getenv("ANTHROPIC_API_KEY"), priority=2,
   ))
   
   agent = Agent(
       name="assistant",
       llm={"model": "gpt-4o-mini", "failover_manager": manager},
   )
   agent.start("Hello!")

2. Add more profiles

   import os
   from praisonaiagents import AuthProfile, FailoverManager
   
   manager = FailoverManager()
   manager.add_profile(AuthProfile(
       name="groq", provider="groq",
       api_key=os.getenv("GROQ_API_KEY"), priority=3,
   ))

3. Monitor provider health

   status = manager.status()
   for name, info in status.items():
       print(f"{name}: {info['status']}")

How failover activates during retries

Failover now drives LLM retries through direct integration with the retry mechanism:

• On every LLM call, the system first gets the current profile via get_next_profile() and applies its api_key, base_url, and model settings
• On success, mark_success(profile) is called to track the working provider
• On failure, mark_failure(profile, error, is_rate_limit=...) marks the provider as failed, then get_next_profile() fetches the next available provider
• Profile switching overrides non-retryable classification—one extra attempt is always granted after switching providers
• The LLM automatically updates request parameters (api_key, base_url, model) when switching between profiles

How It Works

Configuration Options

from praisonaiagents import FailoverConfig

config = FailoverConfig(
    max_retries=3,
    retry_delay=1.0,
    exponential_backoff=True,
    max_retry_delay=60.0,
)

Load spreading across equal-priority credentials

If you have two API keys of the same priority — say, two OpenAI keys for redundancy — turn on rotate_on_success and PraisonAI will spread traffic across them one call at a time instead of hammering one until it hits a rate limit.

When you have multiple credentials of the same priority — e.g. two OpenAI keys — set rotate_on_success=True so each successful call moves to the next key in the tier.

import os
from praisonaiagents import Agent, AuthProfile, FailoverConfig, FailoverManager

manager = FailoverManager(FailoverConfig(rotate_on_success=True))

# Two equal-priority keys share the top tier and get round-robin traffic
manager.add_profile(AuthProfile(name="key-a", provider="openai",
    api_key=os.getenv("OPENAI_KEY_A"), priority=0))
manager.add_profile(AuthProfile(name="key-b", provider="openai",
    api_key=os.getenv("OPENAI_KEY_B"), priority=0))

# A lower-priority key is used only when the top tier is exhausted
manager.add_profile(AuthProfile(name="backup", provider="anthropic",
    api_key=os.getenv("ANTHROPIC_KEY"), priority=1))

agent = Agent(name="assistant", llm={"model": "gpt-4o-mini", "failover_manager": manager})
for _ in range(4):
    agent.start("Hi")   # alternates key-a → key-b → key-a → key-b

How rotation is scoped: Rotation only cycles within the top available priority. If key-a gets rate-limited into cooldown, both remaining calls go to key-b until key-a recovers, then rotation resumes.

Auth Profiles

Configure credentials for each provider:

import os
from praisonaiagents import AuthProfile

profile = AuthProfile(
    name="openai-primary",
    provider="openai",
    api_key=os.getenv("OPENAI_API_KEY"),
    priority=1,
    rate_limit_rpm=100,
)

Common Patterns

from praisonaiagents import AuthProfile, FailoverManager

manager = FailoverManager()

# Add multiple providers
manager.add_profile(AuthProfile(
    name="openai", provider="openai",
    api_key=os.getenv("OPENAI_API_KEY"), priority=1,
))

manager.add_profile(AuthProfile(
    name="anthropic", provider="anthropic",
    api_key=os.getenv("ANTHROPIC_API_KEY"), priority=2,
))

manager.add_profile(AuthProfile(
    name="groq", provider="groq",
    api_key=os.getenv("GROQ_API_KEY"), priority=3,
))

import os
from praisonaiagents import AuthProfile, FailoverManager

manager = FailoverManager()
manager.add_profile(AuthProfile(
    name="gpt-4o-mini", provider="openai",
    api_key=os.getenv("OPENAI_API_KEY"), priority=1,
    model="gpt-4o-mini",
))
manager.add_profile(AuthProfile(
    name="gpt-4o", provider="openai",
    api_key=os.getenv("OPENAI_API_KEY"), priority=2,
    model="gpt-4o",
))

import os
from praisonaiagents import AuthProfile, FailoverManager

manager = FailoverManager()
manager.add_profile(AuthProfile(
    name="openai-us", provider="openai",
    api_key=os.getenv("OPENAI_API_KEY"),
    base_url="https://api.openai.com/v1", priority=1,
))
manager.add_profile(AuthProfile(
    name="azure-eu", provider="azure",
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    base_url=os.getenv("AZURE_OPENAI_ENDPOINT"), priority=2,
))

Failover Callbacks

React to failover events:

from praisonaiagents import FailoverManager, FailoverConfig

def on_failover(from_profile, to_profile, error):
    print(f"Failing over from {from_profile} to {to_profile}")
    print(f"Reason: {error}")
    # Log to monitoring system

config = FailoverConfig(
    on_failover=on_failover
)

manager = FailoverManager(config)

Failover callbacks run outside the pool lock

Your on_failover callbacks are invoked after FailoverManager releases its internal lock (PraisonAI PR #3574). This means:

• Blocking work in a callback (network I/O, logging, metrics) does not stall other agents queuing on the pool.
• A callback that acquires an external lock will not deadlock against the pool’s lock.
• Callbacks still run in the thread that triggered the failover — if you care about ordering, that is preserved.

Provider Status

Monitor provider health:

from praisonaiagents import FailoverManager

manager = FailoverManager()

# Get status of all providers
status = manager.status()
for name, info in status.items():
    print(f"{name}: {info['status']}")
    print(f"  Failures: {info['failure_count']}")
    print(f"  Last success: {info['last_success']}")

# Reset a provider after recovery
manager.mark_success("openai")

# Reset all profiles
manager.reset_all()

Resetting all profiles

reset_all() forces every profile back to AVAILABLE and clears both local cooldowns and the shared coordinator bench.

import os
from praisonaiagents import Agent, AuthProfile, FailoverManager

manager = FailoverManager()
manager.add_profile(AuthProfile(
    name="openai", provider="openai",
    api_key=os.getenv("OPENAI_API_KEY"), priority=1,
))

agent = Agent(name="assistant", llm={"model": "gpt-4o-mini", "failover_manager": manager})
agent.start("Hello!")

# After rotating a key or clearing a known incident,
# force every profile back to AVAILABLE across local + coordinator state.
manager.reset_all()

Call it after rotating an API key, finishing a maintenance window, or resolving a known fleet-wide incident.

Thread safety

FailoverManager is safe to share across concurrent agents and threads — it is the process-wide credential pool (PraisonAI PR #3574). Concurrent agents hitting a rate limit at the same time cannot observe torn reads or corrupt the pool: every read and mutation of the profile list and each profile’s status is serialised internally.

Wire one FailoverManager into many agents and let them share the same credential pool — no external locking required. See Thread Safety for how this fits with the rest of the SDK’s concurrency model.

Cross-Replica Coordination

FailoverManager accepts an optional coordinator= argument (implementing QuotaCoordinatorProtocol) that shares credential cooldowns across replicas. When omitted, a private in-memory coordinator is used and behaviour is exactly as before.

from praisonaiagents import FailoverManager
from praisonaiagents.llm import LocalQuotaCoordinator

# Share cooldowns across managers in the same process.
shared = LocalQuotaCoordinator()
manager = FailoverManager(coordinator=shared)

Benches key on AuthProfile.credential_id — a stable, non-secret hash of provider + base_url + api_key — so two profiles named "default" with different keys never collide. See Quota Coordinator.

Best Practices

Related