> ## Documentation Index > Fetch the complete documentation index at: https://praison.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Tool Circuit Breaker > Automatic protection against repeatedly-failing tools Every tool call is automatically protected by a circuit breaker that stops repeated failures from wasting time. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent, tool @tool def flaky_api(query: str) -> str: """Call an external API.""" return f"Results for: {query}" agent = Agent(name="Researcher", tools=[flaky_api]) agent.start("Search quantum computing") # circuit breaker protects automatically ``` **Sync and Async Parity**: Circuit breaker protection applies uniformly to both sync and async tool execution paths. Parity was first delivered in [MervinPraison/PraisonAI#4469](https://github.com/MervinPraison/PraisonAI/pull/4469), which wraps each async invocation through `CircuitBreaker.acall(...)` — event-loop safe, no `time.sleep`. [MervinPraison/PraisonAI#4533](https://github.com/MervinPraison/PraisonAI/pull/4533) then added the `_circuit_breaker_precheck(...)` / `_circuit_breaker_record(...)` helpers for the pre-check and outcome recording. Both a **raised exception** inside the tool and a **result dict carrying `error`** count as breaker failures — but approval / permission / policy / guardrail denials do not. Earlier releases skipped the breaker on `achat()`/`astart()`. [MervinPraison/PraisonAI#4969](https://github.com/MervinPraison/PraisonAI/pull/4969) then removed a double-record on the async path and started recording `wait_for` cancellations so async timeouts count toward the threshold like sync failures. This tool-level circuit breaker is separate from the new [LLM idle-timeout circuit breaker](/docs/features/llm-error-classification#idle-timeout-circuit-breaker), which protects against LLM provider stalls during model calls. **Per-agent scoping:** breakers are now keyed per agent instance (`tool_{id(self)}_{function_name}`), so two agents that expose same-named tools (e.g. `search`) no longer share one breaker — one agent's failures can't degrade the other. Per-agent breakers are auto-pruned from the registry when the `Agent` is garbage-collected (via `weakref.finalize`), so a reused instance id can't inherit a stale OPEN breaker — `agent.close()` / `aclose()` merely reclaims that space earlier. **Retrieving a breaker by tool name alone no longer returns the runtime instance.** Because the registry key is now `tool_{id(agent)}_{function_name}`, calling `get_circuit_breaker("tool_my_tool")` builds a **fresh, disconnected** breaker — not the one protecting live calls. Use `get_circuit_breaker(f"tool_{id(agent)}_my_tool")`, or enumerate this agent's breakers with the [Observability](#common-patterns) pattern below. Cleanup is covered in [Agent Lifecycle Cleanup](/docs/features/agent-lifecycle-cleanup). The user triggers a flaky tool; repeated failures open the breaker so later calls fail fast instead of looping on errors. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph LR subgraph "Circuit Breaker Flow" A[🛠️ Tool Call] --> B{🛡️ Breaker} B -->|Closed| C[⚡ Execute] B -->|Open| D[🚫 Skip] C --> E[✅ Success] D --> F[❌ Error Dict] end classDef tool fill:#8B0000,stroke:#7C90A0,color:#fff classDef breaker fill:#F59E0B,stroke:#7C90A0,color:#fff classDef execute fill:#189AB4,stroke:#7C90A0,color:#fff classDef skip fill:#6366F1,stroke:#7C90A0,color:#fff classDef success fill:#10B981,stroke:#7C90A0,color:#fff classDef error fill:#6366F1,stroke:#7C90A0,color:#fff class A tool class B breaker class C execute class D skip class E success class F error ``` ## Quick Start Circuit breaker protection is automatically enabled for every tool call with zero configuration needed. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent, tool @tool def flaky_api(query: str) -> str: """Call an external API.""" return f"Results for: {query}" agent = Agent( name="Researcher", instructions="Research the topic", tools=[flaky_api], ) agent.start("Research quantum computing") ``` When a tool fails 5 times consecutively, subsequent calls return an error dictionary instead of calling the tool. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # If my_tool fails 5 times in a row, subsequent calls return: # { # "error": "Tool 'my_tool' circuit breaker open - too many recent failures", # "circuit_open": True, # "agent_name": "Researcher", # "session_id": "...", # "remediation": "Wait for recovery_timeout (60s) or investigate recent tool failures." # } ``` Customize circuit breaker behavior or reset all breakers between test runs. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.tools.circuit_breaker import ( get_circuit_breaker, CircuitBreakerConfig, reset_all_circuit_breakers ) # Register config for a specific tool BEFORE the first call. # Breakers are keyed per agent instance, so build the key from id(agent). breaker = get_circuit_breaker( f"tool_{id(agent)}_my_tool", CircuitBreakerConfig( failure_threshold=3, # Open after 3 failures instead of 5 recovery_timeout=30.0, # Try again after 30s instead of 60s ) ) # Reset all circuit breakers reset_all_circuit_breakers() ``` *** ## How It Works ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant User participant Agent participant CircuitBreaker participant Tool User->>Agent: Request Agent->>CircuitBreaker: Check state CircuitBreaker->>Tool: Call (1st-4th failure) Tool--xCircuitBreaker: Failure CircuitBreaker->>CircuitBreaker: Count failure (5th) CircuitBreaker->>CircuitBreaker: State → OPEN Agent->>CircuitBreaker: Next call CircuitBreaker-->>Agent: {"circuit_open": true} Agent-->>User: Error response Note over CircuitBreaker: Wait 60s CircuitBreaker->>CircuitBreaker: State → HALF_OPEN Agent->>CircuitBreaker: Probe call CircuitBreaker->>Tool: Call (probe) Tool-->>CircuitBreaker: Success CircuitBreaker->>CircuitBreaker: Success count (2nd success) CircuitBreaker->>CircuitBreaker: State → CLOSED ``` ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} stateDiagram-v2 [*] --> CLOSED CLOSED --> OPEN : 5 failures OPEN --> HALF_OPEN : 60s elapsed HALF_OPEN --> CLOSED : 2 successes HALF_OPEN --> OPEN : failure OPEN --> OPEN : rejected calls ``` | State | Behavior | | - | - | | **CLOSED** | Normal operation - all calls pass through | | **OPEN** | Tool calls blocked - returns error dict | | **HALF\_OPEN** | Recovery mode - limited probe calls allowed | The `OPEN → HALF_OPEN` transition is driven by `recovery_timeout` **or** by an `enable_health_check=True` probe reporting healthy — whichever fires first. Since [PR #5330](https://github.com/MervinPraison/PraisonAI/pull/5330), the probe runs on both sync and async paths. The async path (`achat()` / `astart()`) performs the same OPEN → HALF\_OPEN → CLOSED transitions through the shared helpers — `_circuit_breaker_precheck` runs the pre-check, `_circuit_breaker_record` records the outcome — so this diagram applies uniformly to `chat`/`start` and `achat`/`astart`. **Fixed in [PR #4969](https://github.com/MervinPraison/PraisonAI/pull/4969)** (async parity): earlier releases counted each async outcome twice, so an async breaker actually opened at half the configured `failure_threshold` and closed at half the configured `success_threshold`. If you tuned these values to compensate on async workloads, you can now restore them to the values you'd use on sync. A further double-record in `agent/execution_mixin.py` — where `breaker.acall(...)` records the outcome and two sites below it re-recorded the same failure — was closed in [PR #4958](https://github.com/MervinPraison/PraisonAI/pull/4958). Users who still saw async breakers opening at 3 failures despite `failure_threshold=5` after #4969 will now see the configured threshold honoured. The mechanism is identical to sync, but delivered through `CircuitBreaker.acall(...)` rather than `CircuitBreaker.call(...)` so it never blocks the event loop ([PR #4469](https://github.com/MervinPraison/PraisonAI/pull/4469)). An **error-dict result** counts as a failure just like a raised exception — the async path surfaces it to the breaker through a `_ToolFailure` sentinel, mirroring the sync `_ToolFailure` wrapper — so five error-dict results open the breaker exactly as five raises would. Async workflows using `asyncio.gather(...)` get the same parity: one failing tool opens only its own per-agent breaker and short-circuits, so it won't consume retries across every gathered task. *** ## Configuration Options ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph TB subgraph "Configuration Decision Tree" A[Tool Reliability Issue?] --> B{Slow Tool?} A --> C{Flaky Tool?} A --> D{Long Outages?} B -->|Yes| E[Raise timeout] C -->|Yes| F[Raise failure_threshold] D -->|Yes| G[Raise recovery_timeout] B -->|No| H[Use defaults] C -->|No| H D -->|No| H end classDef issue fill:#F59E0B,stroke:#7C90A0,color:#fff classDef decision fill:#6366F1,stroke:#7C90A0,color:#fff classDef action fill:#10B981,stroke:#7C90A0,color:#fff class A issue class B,C,D decision class E,F,G,H action ``` | Option | Type | Default | Description | | - | - | - | - | | `failure_threshold` | `int` | `5` | Failures before opening | | `recovery_timeout` | `float` | `60.0` | Seconds before half-open probe | | `success_threshold` | `int` | `2` | Successes in half-open to close | | `timeout` | `float` | `30.0` | Per-call timeout | | `monitor_window` | `float` | `300.0` | Failure-rate window | | `enable_health_check` | `bool` | `True` | Enable periodic health probes. Triggered automatically when the circuit opens (works on **both** sync `call()` and async `acall()` paths, from [PR #5330](https://github.com/MervinPraison/PraisonAI/pull/5330)). Requires a running event loop; pure-sync callers with no loop fall back to `recovery_timeout`. | | `health_check_interval` | `float` | `30.0` | Health-check interval | | `graceful_degradation` | `bool` | `True` | Return error dict instead of raising | *** ## Custom Health Checks Circuit breakers close themselves as soon as the backing service is healthy again — you just tell the breaker how to check. Provide any object with a `health_check()` method (sync) or `ahealth_check()` method (async), and the breaker will probe it on the interval you configured. It now works the same way whether your agent calls tools via `agent.start()` or `await agent.astart()`. Pass a health check that implements `HealthCheckProtocol` and the breaker (or `HealthMonitor`) probes it on the configured interval. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from typing import Protocol, runtime_checkable @runtime_checkable class HealthCheckProtocol(Protocol): def health_check(self) -> bool: ... # sync — return True if healthy async def ahealth_check(self) -> bool: ... # async — return True if healthy ``` The protocol is exported from `praisonaiagents.tools.circuit_breaker`. Attach an implementation when creating a breaker: ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.tools.circuit_breaker import CircuitBreaker class DatabaseHealth: def health_check(self) -> bool: return ping_database() breaker = CircuitBreaker("database", health_check=DatabaseHealth()) ``` * Canonical method names are `health_check` / `ahealth_check`. * Legacy names `check_health` / `acheck_health` remain supported — the monitor tries the canonical names first, then falls back. * A plain sync callable (e.g. `lambda: True`) also works — it is treated as the sync health check. As of [PR #5050](https://github.com/MervinPraison/PraisonAI/pull/5050), `HealthMonitor` dispatches on the canonical `health_check` / `ahealth_check` names. Classes written against the public protocol are no longer silently reported unhealthy. **Health checks now recover the circuit on both sync and async paths ([PR #5330](https://github.com/MervinPraison/PraisonAI/pull/5330)).** Earlier releases only wired the probe into `acall()`, and it self-disabled after the first async call by pinning a completed Task forever — so a caller relying on `health_check` for fast recovery silently fell back to the full `recovery_timeout` wall-clock wait, no matter how the config was set. The probe is now started when the circuit flips OPEN (from `_on_failure()`), restarts cleanly on each new OPEN episode, and — for a synchronous callable — runs off the event loop so a blocking probe does not stall unrelated coroutines. Pure-sync callers with no running loop keep the pre-#5330 behaviour: recovery is driven by `recovery_timeout` alone. *** ## What triggers health-check recovery A health probe recovers an open circuit as soon as it reports healthy, on both sync and async paths. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph LR Open[🚫 Circuit OPEN] --> Probe[🔧 Probe every health_check_interval] Probe --> Healthy{✅ Healthy?} Healthy -->|No| Probe Healthy -->|Yes| Half[⚙️ HALF_OPEN] Half --> Met{success_threshold met?} Met -->|Yes| Closed[✅ CLOSED] Met -->|No| Half classDef open fill:#8B0000,stroke:#7C90A0,color:#fff classDef probe fill:#189AB4,stroke:#7C90A0,color:#fff classDef decision fill:#6366F1,stroke:#7C90A0,color:#fff classDef half fill:#F59E0B,stroke:#7C90A0,color:#fff classDef closed fill:#10B981,stroke:#7C90A0,color:#fff class Open open class Probe probe class Healthy,Met decision class Half half class Closed closed ``` *** ## What DOES Trip the Breaker **Async tool timeouts count.** When an async tool is cancelled by the per-call `tool_timeout`, the timeout is recorded on the breaker just like a raised exception or an error-dict result. Five consecutive timeouts open the circuit; the sixth call short-circuits with `circuit_open: True`. *** ## What Does NOT Trip the Breaker Circuit breakers ignore certain error types to avoid false positives: * `approval_denied` — user rejected the tool call * `permission_denied` — access control failure * `approval_error` — approval workflow error * `policy_denied` — policy engine deny * `guardrail_denied` — guardrail-blocked tool call These exclusions apply to **both the sync and async** execution paths. On the async path they're checked as keys on the result dict returned by the tool, matching the sync `_ToolFailure` wrapper's exclusions — so a denial never counts toward opening the breaker on either path. *** ## Lifecycle Per-agent breakers are pruned from the global registry automatically when the `Agent` is garbage-collected — a `weakref.finalize` callback is registered when each breaker is created. This closes the CPython `id()`-reuse window without requiring `agent.close()` / `agent.aclose()` to be called explicitly. Calling `agent.close()` / `agent.aclose()` triggers `_cleanup_circuit_breakers()`, which removes every `tool_{id(agent)}_*` entry from the registry earlier and deterministically — keeping it bounded and preventing a reused id from inheriting a stale `OPEN` breaker. See [Agent Lifecycle Cleanup](/docs/features/agent-lifecycle-cleanup) for the full teardown story. *** ## Async tools A user calling an unreliable async tool via `await agent.achat(...)` sees the breaker open after five raised exceptions (or five error-dict results); further calls short-circuit with a `circuit_open: True` error dict, terminating the async retry loop immediately. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import asyncio from praisonaiagents import Agent, tool @tool async def flaky_upstream(query: str) -> str: """Async tool that may raise on a bad upstream.""" raise RuntimeError("upstream 503") agent = Agent(name="Researcher", tools=[flaky_upstream]) async def main(): # After 5 raised failures the breaker opens; the 6th call # short-circuits with {"circuit_open": True} instead of retrying. await agent.achat("Search quantum computing") asyncio.run(main()) ``` An async tool that hangs past `tool_timeout` counts the same way — the timeout is recorded on the breaker, so five consecutive timeouts open the circuit. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import asyncio from praisonaiagents import Agent, tool from praisonaiagents.config import ToolConfig @tool async def slow_upstream(query: str) -> str: """Async tool that hangs past the tool timeout.""" await asyncio.sleep(60) return "never" agent = Agent( name="Researcher", tools=[slow_upstream], tool_config=ToolConfig(timeout=2), ) async def main(): # After 5 consecutive timeouts, the 6th call short-circuits. for _ in range(6): await agent.achat("Search quantum computing") asyncio.run(main()) ``` Breakers are per Agent instance (`tool_{id(self)}_{function_name}`), so two agents sharing a tool name never share a breaker — an OPEN breaker on `agent_a` never blocks `agent_b`. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import asyncio from praisonaiagents import Agent, tool @tool async def shared_search(query: str) -> str: """Same tool name, two independent agents.""" return f"Results for: {query}" agent_a = Agent(name="A", tools=[shared_search]) agent_b = Agent(name="B", tools=[shared_search]) async def main(): # agent_a's breaker opening on shared_search never blocks agent_b. await asyncio.gather( agent_a.achat("Search topic A"), agent_b.achat("Search topic B"), ) asyncio.run(main()) ``` *** ## Common Patterns Enumerate the breakers that belong to a specific agent instance. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.tools.circuit_breaker import _get_global_registry registry = _get_global_registry() prefix = f"tool_{id(agent)}_" for name in registry.list_services(): if name.startswith(prefix): breaker = registry.get(name) print(name, breaker.state, breaker.stats.failure_count) ``` Retrieving a single breaker by name works too — just build the per-agent key: ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.tools.circuit_breaker import get_circuit_breaker breaker = get_circuit_breaker(f"tool_{id(agent)}_my_tool") print(f"State: {breaker.state}") print(f"Failure count: {breaker.stats.failure_count}") print(f"Total requests: {breaker.stats.total_requests}") print(f"Rejected requests: {breaker.stats.rejected_requests}") ``` Apply different settings per tool by registering the config **before** the agent runs the tool for the first time. The breaker is lazily created on first call using the config passed to `get_circuit_breaker`, so pre-registration under the per-agent key wins. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.tools.circuit_breaker import get_circuit_breaker, CircuitBreakerConfig # Strict config for unreliable external APIs strict_config = CircuitBreakerConfig( failure_threshold=2, # Open quickly recovery_timeout=120.0, # Wait longer before retry ) api_breaker = get_circuit_breaker(f"tool_{id(agent)}_external_api", strict_config) # Lenient config for internal tools lenient_config = CircuitBreakerConfig( failure_threshold=10, # Allow more failures recovery_timeout=30.0, # Recover quickly ) internal_breaker = get_circuit_breaker(f"tool_{id(agent)}_internal_db", lenient_config) ``` Reset all circuit breakers after deployments or system maintenance. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.tools.circuit_breaker import reset_all_circuit_breakers # Reset all breakers (useful for test cleanup or after deployment) reset_all_circuit_breakers() # Or reset individual breakers (use the per-agent key) from praisonaiagents.tools.circuit_breaker import get_circuit_breaker breaker = get_circuit_breaker(f"tool_{id(agent)}_my_tool") breaker.reset() ``` *** ## Best Practices Circuit breakers prevent cascading failures and protect system stability. Keep them enabled in production environments to ensure reliable agent operation. Track circuit breaker statistics in your monitoring systems. Frequent openings indicate underlying tool reliability issues that need attention. Per-agent breakers are auto-pruned when the `Agent` is collected (via `weakref.finalize`). Explicit `agent.close()` reclaims registry space immediately. `reset_all_circuit_breakers()` remains the sledgehammer for global tests that need a clean slate regardless of GC timing. When handling `circuit_open: true` responses, provide clear user feedback about temporary tool unavailability and suggest retry timeframes or alternative approaches. An async tool that raises (e.g. `RuntimeError("upstream 503")`) counts as a breaker failure just like an error-dict result — the async path records the failure inside its `except` block. Five raised failures in a row open the breaker so the sixth call short-circuits with `circuit_open: True` instead of hammering the flaky tool again. *** ## Related Automatic LLM provider switching Comprehensive error handling strategies This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.