packages/agent/docs/tool-durability.md
This document specifies the durable tool-call lifecycle and the minimal harness-specific progress-checkpoint extension to the current tool API. It does not otherwise finalize the harness-native public tool interface; that interface must provide the capabilities required here without exposing raw session storage.
The design has two independent additions:
outcome_ready state between external-effect settlement and source-ordered conversation placement;onUpdate snapshots.Parallel tool effects finish in completion order, while tool-result entries must enter the conversation in assistant source order.
Without an intermediate durable state:
calls: A, B, C
B finishes
C finishes
A is still running
process crashes
B and C exist only in process memory because A prevented source-ordered placement. Recovery treats them as unresolved and may replay or interrupt effects that already completed.
The solution separates two orders:
A complete finalized result becomes durable immediately in pi.pending.entry; the call becomes outcome_ready; placement happens later when every earlier source position is complete or ready.
replay: "safe" | "never" contract.step.do.step.do call.Session, SessionMutator, or bound storage-address access.Each call already reserves its result entry ID before execution. Use that as the stable public invocation identity:
invocationId = resultEntryId
It is session-unique, survives safe replay, and is distinct from the provider's batch-local toolCallId, which a later assistant message may reuse.
The surrounding operation state continues to provide:
operationId;turnId/generation step ID;sourceIndex;operationToolArgs(operationId, turnId, sourceIndex)
// Effective validated arguments, persisted before effect admission.
pendingEntry(resultEntryId)
// Complete finalized ToolResultMessage while outcome_ready awaits placement.
Define the operation-owned address constructor in session/values.ts:
export const operationToolMemo = (
operationId: string,
invocationId: string,
memoName: string,
) => value<JsonValue>(
"pi.op.tool_memo",
`${operationId}:${invocationId}:${memoName}`,
);
memoName must be non-empty and contain no :. Names may use dots or slashes for application-local grouping. setMemo(name, undefined) deletes the exact bound value.
scanValues(operationToolMemoPrefix(operationId)) permits defensive operation cleanup. scanValues(operationToolMemoPrefix(operationId, invocationId)) permits atomic invocation cleanup when one outcome becomes ready. Core cleanup uses these owner-defined prefix constructors rather than repeating raw reserved namespace/key grammar.
The durable recovery value is the latest complete bounded progress snapshot selected by the tool. That is total current state, so use one bound value address:
export const pendingToolOutput = (
operationId: string,
invocationId: string,
) => value<AgentToolResult<unknown>>(
"pi.pending.tool_output",
`${operationId}:${invocationId}`,
);
This is auxiliary observation data. It never proves the effect succeeded or completed. The stored value has exactly the same content/details/usage shape as the live partialResult; recovery does not need a tool-specific progress codec.
The tool owns snapshot bounding, checkpoint cadence, and duplicate suppression. The harness owns synchronous enqueue, promise tracking, invocation fencing, and cleanup. The first API has no generic byte cap and does not truncate or reinterpret typed tool data; in-process tools are trusted to honor the bounded-snapshot contract. There is no tool-visible flush() method and no durable list of progress updates.
A crash may lose live updates newer than the latest committed checkpoint. JSONL physical growth is proportional to the size and frequency of distinct requested checkpoints until compaction; Memory and SQLite retain one current value. At 50 KiB every two seconds, JSONL's uncompacted worst case is approximately 15 MiB per ten minutes of continuously changing output.
Keep the existing full-snapshot update callback and add harness-specific options:
export interface AgentHarnessToolUpdateOptions {
/** Request replacement of this invocation's durable recovery checkpoint. */
checkpoint?: true;
}
export type AgentHarnessToolUpdateCallback<TDetails> = (
partialResult: AgentToolResult<TDetails>,
options?: AgentHarnessToolUpdateOptions,
) => void;
AgentHarnessTool uses this callback instead of the legacy AgentToolUpdateCallback. The harness always supplies it, even without a live listener, because a tool may request persistence through it. The legacy AgentTool and old agent loop remain unchanged because they cannot honor durable checkpoints.
Every callback invocation remains an immediate live update. The callback is synchronous and returns void; checkpoint: true additionally requests persistence of that complete snapshot. It is not a durability acknowledgement. Internally, the harness retains the latest events.emit(tool_update) promise so existing listener delivery completes before after_tool; tools do not await or receive that promise.
Every checkpoint:true call synchronously enqueues one scalar replacement on the Session mutation line, attaches the ordinary harness-fault observer to that promise, and replaces the process-local latestCheckpointWrite reference. Writes themselves are neither dropped nor coalesced, and replacing the reference never leaves an earlier rejection unobserved:
effect_pending;after_tool;A tool that requests checkpoints faster than storage can queue work in memory. Under the trusted-tool contract, cadence is the tool's responsibility. The built-in bash policy bounds this queue in ordinary use.
Tools should compare against their last requested checkpoint when duplicate suppression matters. Storage does not read and deep-compare the current scalar as part of every checkpoint.
The built-in bash tool keeps its current 100 ms live update cadence. It requests a checkpoint at most once every two seconds and only when the complete bounded snapshot differs from its last requested checkpoint:
const BASH_UPDATE_THROTTLE_MS = 100;
const BASH_CHECKPOINT_INTERVAL_MS = 2_000;
The current ShellCaptureProgress already supplies a bounded snapshot: the last 2,000 lines or 50 KiB, plus truncation metadata and the overflow-file path. The initial empty update is live-only. Output volume never accelerates checkpoint frequency. A short tool may settle without writing any checkpoint because its complete final result commits instead.
Extend the call union with outcome_ready:
type ToolCall =
| {
status: "planned";
sourceIndex: number;
resultEntryId: string;
}
| {
status: "effect_pending";
sourceIndex: number;
resultEntryId: string;
replay: "never" | "safe";
}
| {
status: "outcome_ready";
sourceIndex: number;
resultEntryId: string;
terminate: boolean;
}
| {
status: "completed";
sourceIndex: number;
resultEntryId: string;
terminate: boolean;
};
outcome_ready means:
after_tool have finished, or the harness has constructed a final synthetic result;ToolResultMessage exists at pendingEntry(resultEntryId);The exact union may later carry small settlement metadata, but it must not duplicate the finalized result payload.
planned
├─ real effect cleared → effect_pending
└─ immediate/synthetic → outcome_ready
effect_pending
├─ live effect settles → outcome_ready
├─ safe orphan replay settles→ outcome_ready
└─ unsafe orphan synthesis → outcome_ready
outcome_ready
└─ source position eligible → completed
An implementation may fuse outcome_ready → completed into the same transaction that finalizes an immediately placeable head call, but the semantic checks and tests must still cover durable outcome_ready for out-of-order calls. Prefer implementing the explicit two-transaction form first.
Unchanged effect sandwich:
planned
→ prepare arguments, run before_tool, validate replacements
→ TX[
set pi.op.tool_args,
set call = effect_pending(replay)
]
→ post-commit tool_start
→ admit tool execution
The invocation-scoped capability becomes active only for this durable effect_pending call.
Every onUpdate(partialResult, options) publishes the live update through the existing event/snapshot path. When options.checkpoint === true, the harness additionally requests a scalar replacement:
TX[
setValue(pendingToolOutput(operationId, invocationId), partialResult)
]
The mutation verifies the same operation, turn, source position, and invocation remain effect_pending. It does not rewrite pi.op.state. A late checkpoint after settlement returns without committing.
The tool must checkpoint bounded complete snapshots, not growing unbounded values. Bash uses the same bounded ShellCaptureProgress snapshot it already sends live. Clients may render and locally retain live updates newer than the durable checkpoint, but those updates are explicitly process-local.
outcome_readyWhen the tool promise settles:
tool_update delivery and latest checkpoint-write promise; each implies completion of its preceding queue;after_tool when this is a real fresh or safely replayed result and cancellation did not prevent the hook;ToolResultMessage;outcome_ready;tool_end from the committed staging transition.setMemo() returns a promise and tools must await it; step.do always does. An unawaited pre-return mutation is still enqueued before staging and is deleted by staging. Calls begun after capability expiry reject. No separate invocation-write drain exists.
Transaction:
TX[
setValue(
pendingEntry(resultEntryId),
{ type: "message", payload: finalizedToolResultMessage },
),
deleteValue(pendingToolOutput(operationId, invocationId)),
deleteValue(memo.address) for every memo returned before commit by
scanValues(operationToolMemoPrefix(operationId, invocationId)),
setValue(operationState(operationId), call = outcome_ready(terminate))
]
The transaction is the linearization point after which the invocation can never replay. Its post-commit tool_end is therefore durable evidence that the finalized outcome is ready; it is no longer a pre-commit effect observation.
The staged message contains the final:
isError;terminate remains orchestration state because it controls the batch continuation and is copied to the immutable entry's terminate field at placement. Staged addedToolNames do not affect the active tool set until that result materializes in the transcript.
After any call becomes outcome_ready, find the contiguous ready prefix beginning at the first non-completed source position.
Example:
[completed, outcome_ready, outcome_ready, effect_pending]
└──────── ready prefix ────────┘
Before the placement transaction, emit and await each finalized result's message_start and message_end in source order. Materialize the prefix in one transaction when practical, then emit entry_added and reported usage events in the same source order:
TX[
insert result entry i from pendingEntry(i),
deleteValue(pendingEntry(i)),
insert tool usage row i if reported,
insert result entry i+1 with parent = result i,
deleteValue(pendingEntry(i+1)),
insert tool usage row i+1 if reported,
setValue(branchTip(lane), newest result),
setValue(operationState(operationId),
calls i..i+1 = completed and, when complete, next checkpoint)
]
Each inserted entry uses its already-reserved resultEntryId. Writes construct the parent chain in source order inside the transaction.
Tool-reported usage remains durable in the staged message until placement. The initial implementation writes its ledger row atomically with entry materialization, matching the current entry/usage ordering and avoiding a ledger row that references an entry not yet present. No usage ID reservation is needed because a failed placement transaction writes neither row nor completed state.
When the final call materializes, the same transaction calls scanValues(operationToolArgsPrefix(operationId, turnId)) and deletes every returned address, and transitions to the correct checkpoint:
may_finish, no final assistant required;need_assistant(false).Outcome staging follows actual completion order. Entry materialization follows source order.
A, B, C start
B finishes → B outcome_ready
C finishes → C outcome_ready
A finishes → A outcome_ready
materialize A, B, C
Crash after B and C stage:
A effect_pending
B outcome_ready
C outcome_ready
Recovery applies unknown-outcome policy only to A. B and C require no tool registration or hook execution to become entries.
The durable invariant changes from “completed calls form a source-ordered prefix” to:
planned, effect_pending, or outcome_ready in any mixture;Sequential execution constructs at most one non-planned call after the completed prefix. Committed call state is trusted on restore; the owning procedure enforces this shape while creating and consuming transitions rather than through a broad restore audit.
For orphaned effect_pending with replay: "never":
pendingToolOutput(operationId, invocationId) when present;ToolResultMessage with isError: true;outcome_ready and clean invocation state.The marker must state that output is partial and the external outcome is unknown. isError: true describes the result delivered to the model; it does not assert that the external effect failed.
Example final text suffix:
[Tool execution was interrupted. The preceding output is the latest durable progress snapshot; newer live output may be missing, and the external outcome is unknown.]
Rules:
after_tool for this synthetic result;usage when present, but ignore checkpoint addedToolNames and terminate because progress never has final-result authority;terminate: false and add no tools;For orphaned effect_pending where both the stored and current declarations are replay: "safe":
pendingToolOutput(operationId, invocationId);invocationId;step.do calls return their memoized values;outcome_ready path.Deleting old progress prevents duplicate chunks when replayed code emits progress again. A crash after the delete but before replay admission remains effect_pending; the next recovery repeats the same safe procedure.
If the current tool declaration is missing or no longer safe, use unsafe interruption recovery rather than suspending.
The harness-native tool call receives a purpose-built invocation capability conceptually equivalent to:
interface AgentHarnessToolInvocation {
readonly invocationId: string;
readonly operationId: string;
readonly turnId: string;
getMemo(key: string): Promise<JsonValue | undefined>;
setMemo(key: string, value: JsonValue | undefined): Promise<void>;
}
Each operation:
effect_pending call when that job executes;operationToolMemo(operationId, invocationId, name);The durable check matters for authorized external finalization. In ordinary execution, a memo mutation initiated before the tool returns is FIFO-ordered before outcome staging; one initiated afterward fails the expired-capability check.
A late zombie callback can neither recreate memos after outcome_ready nor write into a later operation.
Memos are immediate durable replay state, not application-visible settlement state. They survive close/crash while the call remains effect_pending and are deleted when any real or synthetic outcome becomes ready.
Terminal cleanup defensively scans and deletes the operation-owned families:
scanValues(operationToolMemoPrefix(operationId))
scanValues(pendingToolOutputPrefix(operationId))
in addition to other operation-owned addresses. Each returned StoredValue supplies its exact bound address for deleteValue; no later operation receives a raw key.
step.doBuild step.do over invocation memos; it does not need its own harness state union:
interface ToolSteps {
do<T extends JsonValue>(
name: string,
effect: () => T | Promise<T>,
): Promise<T>;
}
Algorithm:
validate deterministic unique name
→ getMemo("step/" + name)
→ present: return stored value
→ absent: run effect
→ setMemo("step/" + name, value)
→ await durability
→ return value
Crash behavior:
before/during effect → effect may run on replay
effect returned, memo not committed → effect may run on replay
memo committed → replay returns memo
step A memoized, step B interrupted → rerun tool; A skips, B runs
This is exactly-once recorded and at-least-once executed. It does not make arbitrary external effects exactly once. An application may derive a stable external idempotency key from (invocationId, stepName) when the external API supports one.
Errors are not memoized. A thrown effect either contributes to the current tool result or runs again after safe whole-tool recovery.
Do not add per-step replay: "never" in this slice. Supporting it correctly requires a nested planned → effect_pending → completed state and an explicit unknown-outcome policy. Whole-tool replay policy is sufficient for the Flue use case already discussed.
Within one live execution, calling the same step name twice is an invariant error. Names must be deterministic across safe replay.
Flue-style application state is distinct from invocation memos.
Invocation memo:
step completed → memo becomes visible immediately
Application state:
tool stages state change
→ crash before outcome_ready: state must not appear committed
→ outcome_ready: state and finalized result become visible together
Do not implement application state by calling Session.setValue() directly during execution.
Two valid implementation stages:
invocationId.outcome_ready transaction.The second option is required only if Flue's usePersistentState moves into harness-owned session values. Its public typing and conflict semantics remain an open design item for the harness-native tool discussion.
Cancellation reconciliation never replays a restored tool.
planned calls receive a synthetic aborted result and become outcome_ready;outcome_ready with terminate: false;effect_pending calls use an interrupted synthetic result, optionally including partial output, regardless of safe replay declaration;outcome_ready calls are preserved and materialized in source order;before_tool or after_tool starts during restored synthetic reconciliation.The aborted terminal transaction runs only after every call outcome has materialized and accepted deferred writes have drained under the existing cancellation rules.
Close is still a controlled crash:
effect_pending or outcome_ready.External finalization deletes operation-owned arguments, invocation memos, partial output, staged pending outcomes, and other pending entries in its terminal transaction. A live task that later tries to stage an outcome fails the ownership fence and stops through OperationEnded.
Base restore constructs the trusted lane/operation projection from required owner values. It does not hydrate or semantically audit tool arguments, invocation memos, progress checkpoints, staged outcomes, completed entries, completed-prefix shape, or captured execution-mode relationships.
The procedure responsible for the current typed state performs only its exact consumption-time reads:
plannedClearance needs no auxiliary restore read. It prepares the call and writes arguments before effect admission.
effect_pendingActivation reads operationToolArgs(operationId, turnId, sourceIndex) and optionally reads pendingToolOutput(operationId, invocationId). Missing required arguments are an invariant defect at consumption. Invocation memos are read only through the scoped capability. Safe replay, unsafe interruption, and snapshots use no broad prefix scan.
outcome_readyMaterialization reads pendingEntry(resultEntryId). Its absence or wrong trusted message relationship is an invariant defect when materialization consumes it. No tool identity or effect recovery is needed.
completedOrdinary dispatch performs no restore-time entry audit. Context/tree reads later consume the immutable entry through their normal typed paths.
Every live mutation still verifies current operation, turn, source position, invocation, and status on the Session line. Those checks fence concurrent settlement, cancellation, and external finalization; they are not historical restore validation. Terminal prefix cleanup remains defensive and does not make orphan scans part of restore.
A reconnecting client may see:
outcome_ready calls as settled rows in runningTools until source-ordered materialization;LaneSnapshot.operation.runningTools is a discriminated union. An effect-pending tool has status: "running" and an optional result containing the latest complete progress snapshot, falling back to the durable checkpoint after reopen. An outcome-ready call has status: "settled", its required complete final result, and isError; it remains there until its immutable result entry's entry_added removes the row and places the same presentation in the transcript. Planned and completed calls are omitted.
tool_start begins public processing presentation for a fresh call; it is emitted from the commit that establishes effect intent or a synthetic staged outcome and does not by itself prove an external effect started. It carries effective arguments for an intended effect and source arguments for an immediate synthetic result.tool_update delivery before after_tool, preserving the existing listener ordering without making onUpdate async.tool_end carries the complete finalized result after its outcome_ready staging commit, in completion order. It is durable settlement evidence and does not repeat the arguments from tool_start.tool_start followed by tool_end; these paths still run no tool effect or post-effect hook. Cancellation after effect intent uses the earlier intent-bound start and a staging-bound end.tool_end when interruption synthesis stages.entry_added occur when the staged result materializes, not when it first becomes outcome_ready; entry_added removes only that settled row.Instrumented-storage tests assert intent commit → tool_start → tool_update* → outcome staging → tool_end → source-ordered placement for execution and outcome staging → tool_start → tool_end → source-ordered placement for fresh synthetic results. Historical events are not replayed; a safely replayed execution emits recovery tool_start from its checkpoint-clear commit and tool_end from outcome staging.
| Race | Required result |
|---|---|
| checkpoint vs tool settlement | every accepted checkpoint was enqueued first; settlement awaits the latest promise, then staging deletes the checkpoint value; a late update is ignored |
memo write vs outcome_ready | an awaited or pre-return-enqueued write precedes staging and is then deleted; a post-return call rejects; external finalization first causes the durable ownership check to reject |
| B outcome vs earlier A settlement | B stages independently; placement waits for A |
| crash after outcome staging | tool never replays; pending result later materializes |
| crash during source-prefix placement | transaction exposes either none or all of that placement prefix |
| safe replay vs old partial output | the old bound checkpoint value is deleted before replay emits new progress |
| cancellation vs real settlement | Session mutation order chooses real cancelled-control result or synthetic reconciliation; at most one outcome stages |
| terminal finalization vs late result | terminal ownership wins or outcome stages first; late task never recreates operation data |
| external finalization vs memo/checkpoint mutation | mutation first is removed by terminal cleanup; finalization first makes the mutation's durable ownership check reject |
invocationId equals the reserved result entry ID and is stable across safe replay.outcome_ready or completed never executes again.outcome_ready call has exactly one complete matching pi.pending.entry value.effect_pending.step.do values are memoized only after their memo write commits; effects remain at-least-once.[completed, outcome_ready, planned, effect_pending, outcome_ready] state;after_tool;step.docheckpoint: true writes the complete bounded snapshot;after_tool, outcome-ready staging, post-commit tool_end, source-ordered message lifecycle, and materialization order;outcome_ready;Expected runtime areas:
packages/agent/src/harness/session/types.ts;Concrete built-in address constructors in session/values.ts:
operationToolMemo(operationId, invocationId, name) → value("pi.op.tool_memo", ...)
pendingToolOutput(operationId, invocationId) → value("pi.pending.tool_output", ...)
pendingEntry(resultEntryId) → value("pi.pending.entry", ...)
Implement outcome_ready and invocation memos before progress checkpoints. The state solves incorrect parallel replay by itself; checkpoints improve reconnect observation and unsafe interruption diagnostics without becoming completion authority.