LLM Error Classification

Typed error categories, failover decisions, and idle-timeout protection for LLM calls

PraisonAI classifies every LLM failure into one of 11 typed categories so retries, failovers, and circuit breakers can react correctly without parsing error strings.

from praisonaiagents import Agent

agent = Agent(
    name="Smart Agent",
    instructions="Process user requests with automatic error handling",
)
result = agent.start("Generate a summary")

The user sends a prompt; transient LLM errors trigger retries or profile rotation instead of opaque failures.

Quick Start

1. Simple Usage

   Error classification happens automatically when using agents:

   from praisonaiagents import Agent
   
   agent = Agent(
       name="Smart Agent",
       instructions="Process user requests with automatic error handling"
   )
   
   # Classification happens transparently
   result = agent.start("Generate a summary")

2. Read Error Category

   Access the typed error category in error hooks:

   from praisonaiagents import Agent
   from praisonaiagents.errors import AgentErrorKind
   
   def handle_error(error):
       # Access typed error category instead of parsing strings
       if error.error_category == "billing":
           print("💳 Quota exceeded - contact billing")
       elif error.error_category == "rate_limit":
           print("⏸️ Rate limited - will auto-retry")
       elif error.error_category == "auth_permanent":
           print("🔑 Invalid API key - check configuration")
   
   agent = Agent(
       name="Smart Agent",
       instructions="Handle requests with error monitoring",
       on_error=handle_error
   )

3. Custom Idle-Timeout Protection

   Tune the idle-timeout circuit breaker for different scenarios:

   from praisonaiagents import Agent
   from praisonaiagents.errors import IdleTimeoutBreaker
   from praisonaiagents.llm import LLM
   
   # Custom breaker for slow self-hosted models
   breaker = IdleTimeoutBreaker(max_consecutive=5)
   
   llm = LLM(
       model="self-hosted/slow-model",
       idle_timeout_breaker=breaker,
       max_iter=15
   )
   
   agent = Agent(
       name="Patient Agent", 
       instructions="Work with slow models",
       llm=llm
   )

How It Works

The classification system converts every LLM exception into a structured FailoverDecision that tells retry logic exactly what to do.

Classification (is_retryable, should_fallback_model) is consulted from every turn shape — non-streaming, streaming, tool-iteration, reflection, sync and async — so an LLMError category maps to the same retry/failover decision regardless of how the turn is being executed.

Error Categories

All LLM failures are classified into these 11 typed categories:

As of PR #1898, billing errors are explicitly added to the non-retryable list and are classified before rate-limit (so quota exceeded no longer double-matches as a rate-limit error). This means a quota-exhausted call now surfaces immediately instead of retrying with backoff.

Classification Ordering

The order of checks in classify_error_kind matters: billing is checked before rate-limit so phrases like quota exceeded route to billing (non-retryable) instead of rate_limit (retryable). Auth is checked before billing so auth_permanent takes precedence on invalid API keys.

An auth error with no failover manager configured now surfaces immediately (surface_error) instead of falling through the generic retry ladder. Introduced in PR #4958 and confirmed on both the sync and async paths by PR #5359, this stops such an error from burning the whole retry budget on a credential that would never recover; with a failover manager registered, auth still triggers rotate_profile as before.

Status-code-first classification

As of PraisonAI PR #5841, classify_error_kind decides in two steps:

1. Category from the status code or exception type. The HTTP status code (401, 404, 429, 503, …) or the exception class picks the category — this is authoritative.
2. Kind refined by message text. The message text can only pick a kind that is allowed for that category. A 400 can become format_error, context_overflow, billing and so on, but never overloaded.

If the error has no status code and no known exception type, the message-text matching described above runs as before. Text matching is now the fallback, not the main path.

If you upgraded, these errors now route differently:

The billing keyword was narrowed from the bare word credit to credit balance / insufficient credit, so text such as credit card no longer counts as a billing error.

Retry-After Header

When a provider sends a Retry-After response header, the retry loop waits that long instead of the 1 s default. resolve_failover_decision passes the exception itself to the delay parser, so the header is read straight from the response.

For example, a 429 with Retry-After: 37 gives FailoverDecision.backoff_ms == 37000 — the agent waits 37 s, not 1 s. The wait is still capped (300 s by default, or the rate limiter’s max_retry_delay).

from praisonaiagents import Agent

agent = Agent(name="Polite Agent", instructions="Summarise the report")
agent.start("Summarise this quarter's sales")  # waits as long as the provider asks on a 429

FailoverDecision Structure

Every classification produces a FailoverDecision with these fields:

Idle-Timeout Circuit Breaker

The idle-timeout circuit breaker is separate from the per-tool circuit breaker. It protects against LLM provider stalls:

• Default: Stops after 3 consecutive idle_timeout failures
• Auto-resets: On any successful LLM call
• Only triggered by: idle_timeout error kind (not other timeouts)

Choosing Between Options

Common Patterns

Log Every Classified Failure

from praisonaiagents import Agent
from praisonaiagents import LLMError

def error_logger(error: LLMError):
    print(f"⚠️ LLM {error.error_category}: {error.message}")

    # Route specific error types
    if error.error_category == "billing":
        # Alert ops team
        send_alert("billing", error.message)
    elif error.error_category == "auth_permanent":
        # Alert dev team
        send_alert("config", error.message)

agent = Agent(
    name="Monitored Agent",
    instructions="Track all LLM failures",
    on_error=error_logger
)

Custom Breaker for Slow Models

from praisonaiagents import Agent
from praisonaiagents.errors import IdleTimeoutBreaker
from praisonaiagents.llm import LLM

# More patient with self-hosted models
slow_model_breaker = IdleTimeoutBreaker(max_consecutive=8)

agent = Agent(
    name="Self-Hosted Agent",
    llm=LLM(
        model="ollama/custom-model",
        idle_timeout_breaker=slow_model_breaker,
        timeout=120  # 2 minute timeout
    )
)

Gate Alerts by Error Type

from praisonaiagents import Agent

def smart_alerting(error):
    # Only alert on errors that need human intervention
    alert_worthy = [
        "auth_permanent", "model_not_found", 
        "context_overflow", "billing"
    ]

    if error.error_category in alert_worthy:
        send_slack_alert(f"🚨 {error.error_category}: {error.message}")
    else:
        # Just log retryable errors
        logger.info(f"Retryable {error.error_category} - auto-handling")

agent = Agent(
    name="Smart Alerting Agent",
    instructions="Only alert on actionable errors",
    on_error=smart_alerting
)

Legacy Migration

Example migration:

# OLD (deprecated - emits warning)
if error.error_category == "budget":
    handle_budget_error()

# NEW (recommended)  
if error.error_category == "billing":
    handle_billing_error()

Migrating from _classify_error_and_should_retry

If you previously subclassed LLM to customise retry classification via _classify_error_and_should_retry, override resolve_failover_decision instead. It returns a typed FailoverDecision in place of the old (category, retry, delay) tuple.

# OLD — no longer works after praisonaiagents 1.6.161
from praisonaiagents import LLM

class MyLLM(LLM):
    def _classify_error_and_should_retry(self, error, attempt=1):
        # Silently ignored — nothing calls this method any more
        return "rate_limit", True, 5.0

# NEW — override the live decision point
from praisonaiagents import LLM
from praisonaiagents.errors import FailoverDecision

class MyLLM(LLM):
    def resolve_failover_decision(self, error, attempt_state):
        return FailoverDecision(
            action="retry", reason="rate_limit",
            backoff_ms=5000, is_retryable=True,
        )

If you called llm._classify_error_and_should_retry(...) directly (rare — the underscore prefix marked it as private), the equivalent public path is llm.resolve_failover_decision(error, {"attempt": n, "max_retries": llm._max_retries}).

Best Practices

permanent_errors = ["auth_permanent", "model_not_found", "format_error"]
if error.error_category in permanent_errors:
    # Log as config issue, not service degradation
    config_logger.error(f"Config issue: {error.error_category}")

# Fast cloud model (default)
fast_breaker = IdleTimeoutBreaker()  # max_consecutive=3

# Slow self-hosted model  
slow_breaker = IdleTimeoutBreaker(max_consecutive=8)

# AVOID: String parsing
if "quota" in str(exception):
    handle_quota()

# PREFER: Typed classification
if error.error_category == "billing":
    handle_billing_issue()

from praisonaiagents import FailoverManager

failover = FailoverManager([
    AuthProfile(provider="openai", api_key="..."),
    AuthProfile(provider="anthropic", api_key="...")
])

agent = Agent(
    name="Resilient Agent",
    llm={"model": "gpt-4o", "failover_manager": failover}
)
# Auth errors will automatically rotate to Anthropic

Related