Durable Tool Runs

Survive a crash mid-tool-loop without re-running side-effecting tools or re-billing the LLM

Turn on durability with one flag: the agent journals every model and tool boundary, so a crash mid-loop resumes from the exact step it left off — recorded tool results replay instead of re-executing.

Quick Start

1. Enable durable execution

   Set durable=True on ExecutionConfig, then capture agent.last_durable_run_id after the run and persist it alongside your business record.

   from praisonaiagents import Agent, ExecutionConfig
   
   def charge_card(amount: float, customer_id: str) -> dict:
       return payment_gateway.charge(amount, customer_id)
   
   agent = Agent(
       name="Order Processor",
       instructions="Charge the card and ship the order.",
       tools=[charge_card, ship_order, notify_customer],
       execution=ExecutionConfig(
           durable=True,
           journal_path="./journal.db",   # optional; defaults to ~/.praisonai/runs/journal.db
       ),
   )
   
   result = agent.start("Process order #A-1042 for customer C-99")
   
   # Persist alongside your business record so a restart can find it
   run_id = agent.last_durable_run_id

2. Resume after a crash

   Set resume_run_id to the saved id and call start() with the same prompt. Recorded tool results replay from the journal — side-effecting tools do not fire again.

   # SAME agent, SAME prompt
   agent.execution.resume_run_id = run_id
   result = agent.start("Process order #A-1042 for customer C-99")
   
   # charge_card does NOT fire again — the recorded result replays from the journal.
   # resume_run_id auto-clears to None once the turn ends terminally.

3. Enable in the gateway

   Gateway agents journal their turns by default whenever the session store is durable (the shipped default) — no flag needed. Set the flag explicitly only when you want to pin the choice in gateway.yaml:

   gateway:
     durable_runs: true   # already the default — this just makes it explicit

   See Gateway Durable Runs for the full operator surface, including the durable_runs: false and reliability: "off" opt-outs.

How It Works

The agent records one event per boundary. On resume the loop is re-driven from the top; journalled steps return their recorded payload instantly, and only un-journalled steps do real work.

Run Outcomes

Each terminal outcome maps to an auto-finalize state and controls whether resume_run_id clears itself.

A retryable failure leaves the run resumable. A non-retryable failure marks it failed and non-resumable.

This applies to achat() too (PR #6109): a tool that raises on the async path now surfaces as ToolExecutionError instead of being handed to the model, and a restart_safe=False tool that raises surfaces with is_retryable=False so the run is finalized failed. Before the fix, durable runs on achat() could finalize succeeded with the model’s “the operation failed” reply.

A per-tool NotSafelyResumable outcome does not by itself change the run status; the loop continues and the run finalizes normally on the next iteration.

Configuration Options

Three fields on ExecutionConfig control durability. Agent.last_durable_run_id is the read-side companion.

Idempotency Contract for Tools

Any tool whose signature accepts idempotency_key (or **kwargs) automatically receives a stable per-call key on every invocation. Non-declaring tools work unchanged.

def charge_card(amount: float, customer_id: str, idempotency_key: str = "") -> dict:
    return payment_gateway.charge(amount, customer_id, idem_key=idempotency_key)

The key shape is "{run_id}:{seq}:{function_name}" — the same on resume, so a tool that dedups on it stays safe even before journal replay records its result.

restart_safe and idempotency_key are complementary: restart_safe=True says the tool may be replayed at all; the auto-injected idempotency_key gives it a stable per-call key to dedupe on when it is. See Restart-Safety Contract for Tools below.

Restart-Safety Contract for Tools

A tool can declare whether it is safe to re-run after a crash — on resume, an in-flight tool is only re-executed when this declaration says so, otherwise it surfaces a typed outcome for the operator to reconcile.

Declare it on the decorator: True for read-only or idempotent tools, False for anything that sends, writes, charges, or deletes.

from praisonaiagents import Agent, ExecutionConfig, tool

@tool(restart_safe=True)
def fetch_report(report_id: str) -> dict:
    return report_api.get(report_id)

@tool(restart_safe=False)
def send_invoice(customer_id: str, amount: int) -> str:
    return billing.send(customer_id, amount)

agent = Agent(
    name="Billing Agent",
    instructions="Fetch the report, then send the invoice.",
    tools=[fetch_report, send_invoice],
    execution=ExecutionConfig(durable=True),
)
agent.start("Bill customer C-99 for report R-42")

Pick the value by asking whether the tool has an external side effect that is not safe to repeat.

The class attribute form BaseTool.restart_safe is the equivalent for subclass-style tools.

The same declaration also gates the ordinary tool-retry loops: a restart_safe=False tool is executed exactly once even when a transient error would normally be retried. See Tool Retry & Backoff → Retrying Non-Idempotent Tools.

When resume returns NotSafelyResumable

An effectful or undeclared tool that was in-flight at the crash is not re-run — the resumed turn records a typed outcome instead, so the operator can reconcile the external action.

{
    "error": (
        "Tool '<name>' was in-flight when the durable run crashed and is not "
        "declared restart_safe; its outcome could not be confirmed. It was not "
        "re-executed on resume to avoid a duplicate side effect. Reconcile the "
        "external action manually or mark the tool @tool(restart_safe=True) if "
        "it is idempotent."
    ),
    "error_type": "NotSafelyResumable",
    "not_safely_resumable": True,
    "restart_safe": False,
}

Reconcile the external action out of band (or with an idempotency check) and re-drive the loop, or mark the tool restart_safe=True if it is genuinely idempotent.

Discovering Resumable Runs

When your process restarts and you don’t have the run_id handy, ask the journal directly for runs still running.

from praisonaiagents.runtime import RunJournal

journal = RunJournal("./journal.db")

for run_id in journal.interrupted_runs():
    print(f"Resumable: {run_id}")
    # Set agent.execution.resume_run_id = run_id and re-run the original prompt

Advanced

Framework and tool authors integrating a custom executor can reach the durable context directly through praisonaiagents.agent.durable.

from praisonaiagents.agent.durable import get_durable_idempotency_key

def charge_card(amount: float, customer_id: str) -> dict:
    key = get_durable_idempotency_key()   # stable per-call key, or None outside a durable run
    return payment_gateway.charge(amount, customer_id, idem_key=key)

The module also exposes begin_durable_run, abegin_durable_run, end_durable_run, get_durable_run, and DurableRunContext for wiring the journal into a bespoke tool loop.

Common Patterns

Long-running scheduled job that survives a container restart:

from praisonaiagents import Agent, ExecutionConfig
from praisonaiagents.runtime import RunJournal

agent = Agent(
    name="Nightly Reconciler",
    instructions="Reconcile pending orders.",
    tools=[reconcile_order],
    execution=ExecutionConfig(durable=True, journal_path="/data/praisonai/journal.db"),
)

# On startup, resume anything left running by a previous crash
for run_id in RunJournal("/data/praisonai/journal.db").interrupted_runs():
    agent.execution.resume_run_id = run_id
    agent.start("Reconcile pending orders.")

Batch of paid API calls that must never double-charge — the recorded receipt replays instead of re-calling:

agent = Agent(
    name="Billing Runner",
    instructions="Charge each pending invoice once.",
    tools=[charge_card],
    execution=ExecutionConfig(durable=True),
)

result = agent.start("Charge invoice #A-1042")
invoice_run_id = agent.last_durable_run_id   # store next to the invoice row

# After a crash: resume without re-charging
agent.execution.resume_run_id = invoice_run_id
agent.start("Charge invoice #A-1042")

Pause-for-approval flows survive a restart because the approval decision is journalled — see Durable Approvals.

Best Practices

Related