Durable Tool Runs

Turn on durability with one flag: the agent journals every model and tool boundary, so a crash mid-loop resumes from the exact step it left off — recorded tool results replay instead of re-executing.

Quick Start

Enable durable execution

Set durable=True on ExecutionConfig, then capture agent.last_durable_run_id after the run and persist it alongside your business record.

from praisonaiagents import Agent, ExecutionConfig

def charge_card(amount: float, customer_id: str) -> dict:
    return payment_gateway.charge(amount, customer_id)

agent = Agent(
    name="Order Processor",
    instructions="Charge the card and ship the order.",
    tools=[charge_card, ship_order, notify_customer],
    execution=ExecutionConfig(
        durable=True,
        journal_path="./journal.db",   # optional; defaults to ~/.praisonai/runs/journal.db
    ),
)

result = agent.start("Process order #A-1042 for customer C-99")

# Persist alongside your business record so a restart can find it
run_id = agent.last_durable_run_id

Resume after a crash

Set resume_run_id to the saved id and call start() with the same prompt. Recorded tool results replay from the journal — side-effecting tools do not fire again.

# SAME agent, SAME prompt
agent.execution.resume_run_id = run_id
result = agent.start("Process order #A-1042 for customer C-99")

# charge_card does NOT fire again — the recorded result replays from the journal.
# resume_run_id auto-clears to None once the turn ends terminally.

The prompt must match on resume. A different prompt raises ValueError("Resume prompt does not match ..."), an unknown id raises ValueError("Cannot resume unknown durable run ..."), and a terminal run raises ValueError("Cannot resume terminal durable run ... (status=...)").

Enable in the gateway

Gateway agents journal their turns by default whenever the session store is durable (the shipped default) — no flag needed. Set the flag explicitly only when you want to pin the choice in gateway.yaml:

gateway:
  durable_runs: true   # already the default — this just makes it explicit

See Gateway Durable Runs for the full operator surface, including the durable_runs: false and reliability: "off" opt-outs.

How It Works

The agent records one event per boundary. On resume the loop is re-driven from the top; journalled steps return their recorded payload instantly, and only un-journalled steps do real work.

Run Outcomes

Each terminal outcome maps to an auto-finalize state and controls whether resume_run_id clears itself.

Outcome	Auto-finalize	`resume_run_id` auto-clears
Success	`succeeded`	Yes
`ToolExecutionError(is_retryable=False)`	`failed`	Yes
`ToolExecutionError(is_retryable=True)`	(none — stays `running`)	No
`InterruptedError` / `asyncio.CancelledError`	`cancelled`	Yes
Streaming `GeneratorExit`	`cancelled`	Yes

A retryable failure leaves the run resumable. A non-retryable failure marks it failed and non-resumable. A per-tool NotSafelyResumable outcome does not by itself change the run status; the loop continues and the run finalizes normally on the next iteration.

Configuration Options

Three fields on ExecutionConfig control durability. Agent.last_durable_run_id is the read-side companion.

Option	Type	Default	Description
`durable`	`bool`	`False`	Enable journal writes and resume. Zero overhead when `False` — the journal is not constructed.
`journal_path`	`Optional[str]`	`None` → `~/.praisonai/runs/journal.db`	SQLite path for the run journal. Use `":memory:"` in tests, a mounted volume in containers.
`resume_run_id`	`Optional[str]`	`None`	Set to a prior run’s id to resume it. Auto-cleared to `None` after any terminal turn.

Idempotency Contract for Tools

Any tool whose signature accepts idempotency_key (or **kwargs) automatically receives a stable per-call key on every invocation. Non-declaring tools work unchanged.

def charge_card(amount: float, customer_id: str, idempotency_key: str = "") -> dict:
    return payment_gateway.charge(amount, customer_id, idem_key=idempotency_key)

The key shape is "{run_id}:{seq}:{function_name}" — the same on resume, so a tool that dedups on it stays safe even before journal replay records its result. restart_safe and idempotency_key are complementary: restart_safe=True says the tool may be replayed at all; the auto-injected idempotency_key gives it a stable per-call key to dedupe on when it is. See Restart-Safety Contract for Tools below.

Restart-Safety Contract for Tools

A tool can declare whether it is safe to re-run after a crash — on resume, an in-flight tool is only re-executed when this declaration says so, otherwise it surfaces a typed outcome for the operator to reconcile. Declare it on the decorator: True for read-only or idempotent tools, False for anything that sends, writes, charges, or deletes.

from praisonaiagents import Agent, ExecutionConfig, tool

@tool(restart_safe=True)
def fetch_report(report_id: str) -> dict:
    return report_api.get(report_id)

@tool(restart_safe=False)
def send_invoice(customer_id: str, amount: int) -> str:
    return billing.send(customer_id, amount)

agent = Agent(
    name="Billing Agent",
    instructions="Fetch the report, then send the invoice.",
    tools=[fetch_report, send_invoice],
    execution=ExecutionConfig(durable=True),
)
agent.start("Bill customer C-99 for report R-42")

Option	Type	Default	Description
`restart_safe`	`Optional[bool]`	`None`	Declare replay behaviour for durable resume. `True` = safe to re-run (read-only / idempotent). `False` = effectful; must never be silently re-executed on resume. `None` = undeclared — falls back to the read-only name heuristic and fails closed when uncertain.

Pick the value by asking whether the tool has an external side effect that is not safe to repeat. The class attribute form BaseTool.restart_safe is the equivalent for subclass-style tools. The same declaration also gates the ordinary tool-retry loops: a restart_safe=False tool is executed exactly once even when a transient error would normally be retried. See Tool Retry & Backoff → Retrying Non-Idempotent Tools.

When resume returns `NotSafelyResumable`

An effectful or undeclared tool that was in-flight at the crash is not re-run — the resumed turn records a typed outcome instead, so the operator can reconcile the external action.

{
    "error": (
        "Tool '<name>' was in-flight when the durable run crashed and is not "
        "declared restart_safe; its outcome could not be confirmed. It was not "
        "re-executed on resume to avoid a duplicate side effect. Reconcile the "
        "external action manually or mark the tool @tool(restart_safe=True) if "
        "it is idempotent."
    ),
    "error_type": "NotSafelyResumable",
    "not_safely_resumable": True,
    "restart_safe": False,
}

Reconcile the external action out of band (or with an idempotency check) and re-drive the loop, or mark the tool restart_safe=True if it is genuinely idempotent.

Discovering Resumable Runs

When your process restarts and you don’t have the run_id handy, ask the journal directly for runs still running.

from praisonaiagents.runtime import RunJournal

journal = RunJournal("./journal.db")

for run_id in journal.interrupted_runs():
    print(f"Resumable: {run_id}")
    # Set agent.execution.resume_run_id = run_id and re-run the original prompt

Advanced

Framework and tool authors integrating a custom executor can reach the durable context directly through praisonaiagents.agent.durable.

from praisonaiagents.agent.durable import get_durable_idempotency_key

def charge_card(amount: float, customer_id: str) -> dict:
    key = get_durable_idempotency_key()   # stable per-call key, or None outside a durable run
    return payment_gateway.charge(amount, customer_id, idem_key=key)

The module also exposes begin_durable_run, abegin_durable_run, end_durable_run, get_durable_run, and DurableRunContext for wiring the journal into a bespoke tool loop.

Common Patterns

Long-running scheduled job that survives a container restart:

from praisonaiagents import Agent, ExecutionConfig
from praisonaiagents.runtime import RunJournal

agent = Agent(
    name="Nightly Reconciler",
    instructions="Reconcile pending orders.",
    tools=[reconcile_order],
    execution=ExecutionConfig(durable=True, journal_path="/data/praisonai/journal.db"),
)

# On startup, resume anything left running by a previous crash
for run_id in RunJournal("/data/praisonai/journal.db").interrupted_runs():
    agent.execution.resume_run_id = run_id
    agent.start("Reconcile pending orders.")

Batch of paid API calls that must never double-charge — the recorded receipt replays instead of re-calling:

agent = Agent(
    name="Billing Runner",
    instructions="Charge each pending invoice once.",
    tools=[charge_card],
    execution=ExecutionConfig(durable=True),
)

result = agent.start("Charge invoice #A-1042")
invoice_run_id = agent.last_durable_run_id   # store next to the invoice row

# After a crash: resume without re-charging
agent.execution.resume_run_id = invoice_run_id
agent.start("Charge invoice #A-1042")

Pause-for-approval flows survive a restart because the approval decision is journalled — see Durable Approvals.

Best Practices

Persist last_durable_run_id with your business record

Capture agent.last_durable_run_id right after start()/chat() and store it next to the order id or ticket id. That’s the value you set into resume_run_id later.

Point journal_path at a persistent volume in containers

The default ~/.praisonai/runs/journal.db is lost when a container is recreated. Pass an explicit path on a mounted volume so the journal survives restarts.

Keep tools idempotent as belt-and-braces

Opt into the auto-injected idempotency_key so a tool that crashes before recording its result still dedups on retry.

Declare restart_safe on every tool used in a durable run

True for read-only / idempotent tools; False for anything that sends, writes, charges, or deletes. Undeclared tools fall back to a name heuristic and fail closed on uncertainty — safe, but noisier than an explicit declaration.

Handle NotSafelyResumable on resume

If a resumed turn returns error_type: "NotSafelyResumable", the in-flight side effect was not re-driven. Reconcile the external action, then either mark the tool restart_safe=True (if truly idempotent) or continue the turn out of band.

Distinguish retryable vs terminal failures

ToolExecutionError(is_retryable=True) leaves the run resumable; is_retryable=False marks it failed and non-resumable.

Close streams cleanly

Calling stream.close() finalizes the run as cancelled, so aborting a stream never leaves a ghost running row.

Run-State Journal

The SQLite layer under durable runs.

Durable Approvals

Pause-for-approval that survives a restart.

Execution

Agent execution limits and configuration.

Gateway Session Persistence

Persist gateway sessions across restarts.

Enable durable runs for every agent in the WebSocket gateway.

Async Jobs

Persist async jobs and idempotency keys with SqliteJobStore.

Getting Started

Install

Core Concepts

Guides

Features

Models

Databases

Observability

Memory

Telemetry & Analytics

Knowledge

RAG

Persistence

Tools

Other Features

Developers

Configuration

Best Practices

Getting Started (No Code)

Quick Start

How It Works

Run Outcomes

Configuration Options

Idempotency Contract for Tools

Restart-Safety Contract for Tools

When resume returns `NotSafelyResumable`

Discovering Resumable Runs

Advanced

Common Patterns

Best Practices

Run-State Journal

Durable Approvals

Execution

Gateway Session Persistence

Async Jobs

​Quick Start

​How It Works

​Run Outcomes

​Configuration Options

​Idempotency Contract for Tools

​Restart-Safety Contract for Tools

​When resume returns NotSafelyResumable

​Discovering Resumable Runs

​Advanced

​Common Patterns

​Best Practices

​Related

Run-State Journal

Durable Approvals

Execution

Gateway Session Persistence

Async Jobs

Quick Start

How It Works

Run Outcomes

Configuration Options

Idempotency Contract for Tools

Restart-Safety Contract for Tools

When resume returns `NotSafelyResumable`

Discovering Resumable Runs

Advanced

Common Patterns

Best Practices

Related