Durable execution — CheckpointSaver

CheckpointSaver persists per-turn snapshots of a session so a crashed or paused agent loop can resume from the last committed turn instead of re-running every step. Pair it with a stable thread_id and the idempotency_key argument to run_turn / arun_turn for replay-safe multi-turn loops.

Quick example

from techrevati.runtime import (
    AgentSession, SqliteSaver, UsageSnapshot,
)

saver = SqliteSaver("checkpoints.db")
agent = AgentSession(role="writer", phase="draft", saver=saver)

with agent.session(thread_id="user-42:essay") as session:
    draft, usage = session.run_turn(
        lambda: call_model(outline_prompt),
        model="model-a",
        usage=UsageSnapshot(input_tokens=2_000, output_tokens=900),
        idempotency_key="draft:turn-1",
    )
    revision, _ = session.run_turn(
        lambda: call_model(revision_prompt(draft)),
        model="model-a",
        idempotency_key="draft:turn-2",
    )

On a clean run, two checkpoints land in checkpoints.db. If the process crashes between the two turns and a future invocation opens a fresh AgentSession against the same thread_id, the first run_turn returns the cached result for "draft:turn-1" without calling the model again, and execution continues with turn 2.

When to use

When NOT to use

Reference implementations

Both implement the same CheckpointSaver protocol, so a session can swap between them by changing the saver= argument on AgentSession.

Anti-patterns

Ordering and failures

get(thread_id) and list return checkpoints newest-first by insertion, not by created_at, so a backward wall-clock step cannot make an older turn look like the latest. created_at is informational.

A write that fails on a lock raises once and rolls back; the saver stays usable and sees other writers' commits on the next call. Concurrent processes may open a new database at the same time.

Tuning

Knob Default Why touch it
SqliteSaver path required :memory: for tests; a real file for restart durability.
list(..., limit=N) 10 Raise it if you have very long threads and need to reach further back.
_restore_idempotent_turn scan depth 100 Internal cap on how far back an idempotency lookup walks. If your threads exceed 100 turns and you need replay older than that, cache the lookup outside the runtime.
SQLite schema version 1 Managed by the runtime; create a fresh database or migrate explicitly if a future version changes the schema.

Step-level durability (in-tool-call replay)

Checkpoints fire between turns. For an expensive, idempotent step inside a turn (an API call you don't want to repeat on a re-run), both reference savers also implement StepCheckpointSaver: cache the step's result under a caller-chosen step_key and skip it next time.

def run_step(saver, thread_id):
    cached = saver.get_step(thread_id, "turn3:fetch-rates")
    if cached is not None:
        return cached.state                  # skip the expensive work
    result = fetch_rates()                    # idempotent, costly
    saver.put_step(thread_id, "turn3:fetch-rates", result)
    return result

put_step overwrites by step_key; list_steps(thread_id) returns records in (created_at, step_key) order; delete(thread_id) clears steps too.

This is opportunistic, caller-keyed memoization — not full deterministic workflow replay. There is no recorded event history and no automatic determinism enforcement; you choose the keys and what is safe to cache.

Tool calls

run_tool / arun_tool use the same step store through idempotency_key, so a re-run does not repeat a side effect that already completed:

with agent.session(thread_id="order-7") as session:
    session.run_tool("charge", charge, idempotency_key="charge")

The result is recorded after the tool passes its post-checks. A failed tool is not recorded, so a retry runs it again. The guarantee is at-least-once: a crash between the tool returning and the record being written re-executes it. For payments and similar calls, also send the key to the downstream API.

A non-JSON-serializable result logs a warning and is not recorded. A custom saver that does not implement StepCheckpointSaver logs a warning and runs the tool normally.

What a replayed turn counts towards

A replayed turn adds its recorded usage to the new session's tracker and governance state, so a restart cannot reset a budget. Sinks are skipped because they saw the original turn. A turn that crosses a budget is checkpointed before BudgetExceededError is raised, because it was already paid for: the retry replays it and raises the same error without paying again.

See also