packages/agent/docs/mobile-handoff/01-harness/02-scopes/implementation-handoff.md
Status: actionable, not implemented.
Prepared against: c4b0e35ab (dev). Re-audit named source files if the implementation baseline changes materially.
This handoff implements storage scopes only. Read this file before scopes.md. It supersedes conflicting details there, especially per-scope sequence spaces, operation-derived orphan detection, one-use scope IDs, raw-string scope arguments, sidecars ending in .jsonl, the proposed interned retirement tuple, and bundling JSONL delta/address encoding into the same change. The broader motivation and measurements in scopes.md remain design evidence; its signatures and implementation order are not current where this file differs.
Step 2—Chord-encoded JSONL values, address interning, compact tuple records, and their measurements—is a separate package. Do not begin Step 2 until Step 1 has been tested, reviewed, explicitly approved by the user, committed, and pushed.
This package has a mandatory user checkpoint:
npm run check, and ./test.sh.anthropic and model claude-fable-5.Do not combine the Step 1 and Step 2 commits. Do not commit or push before approval.
Read completely before editing:
packages/agent/docs/harness.md §§0.2–0.9, 1.1–1.7, 2.7–2.8, 3.7–3.8, 3.13, 4.4–4.8, 5.4, Part 7, Part 9.packages/agent/docs/values.md.packages/agent/docs/mobile-handoff/README.md.scopes.md, using this handoff where they conflict.../01-delta/delta.md for vocabulary ownership only; Step 1 does not store Chord ops.../04-tool-output/harness-tools.md §§7.1–7.7 for later-consumer context; do not implement its output redesign here.../05-assistant-output/message-update.md §7 for later-consumer context; do not implement it here.packages/agent/src/harness/session/{types,values,commit,in-memory-storage-state,memory,session,index}.ts.packages/agent/src/harness/session/jsonl/ and the FileSystem declaration/implementation in packages/agent/src/harness/{types,env/nodejs}.ts.packages/agent/src/harness/runtime/{progress,lane}.ts, runtime/drive/{terminal,response,deferred,tools,reconcile,structural,boundary}.ts, and every caller of operationCleanupWrites. Inspect tool-placement.ts through the grep audit, but current source has no scoped delete there.packages/session-backends/sqlite-node/src/{index,sqlite/index}.ts, sqlite/{storage,repo}.ts, migrations/001_initial.sql, and session/{values,session-sequences}.ts.scopes.variance.ts for the covariant-address/invariant-write phantom mechanism only. Its current no-ID EphemeralScope and retireScope(id: string) signatures are superseded by §2 and must be updated in Slice A.Do not use dist/ as implementation input. Format 4 remains WIP; no R11 migration is part of this package.
A scope ID is a reusable logical lifetime name, not a globally one-use token:
export type SessionScope = { readonly kind: "session" };
export interface EphemeralScope {
readonly kind: "ephemeral";
readonly id: string;
}
export type Scope = SessionScope | EphemeralScope;
export function ephemeralScope(id: string): EphemeralScope;
ephemeralScope(id) requires a non-empty well-formed Unicode string and returns a frozen value. Equal IDs identify the same physical scope; object identity has no meaning. JSONL encodes IDs with encodeURIComponent. The encoded component must be at most 180 ASCII characters; reject longer IDs and lone-surrogate input. The physical filename always has a fixed scope- prefix, so encoded ./.. values cannot become path segments. Do not directly interpolate unencoded caller input into a path.
No create/open-scope transaction exists. The first scoped value/list write creates physical state. A later retireScope(scope) ends everything in that scope through its assigned global sequence. A write under the same ID after retirement begins a new logical lifetime.
Harness operation scopes use ephemeralScope(operationId). Generic application scopes may use stable names. Storage never parses a scope ID as an operation ID and never reads Harness operation state to decide lifetime.
Global sequence order defines reusable generations:
seq 10 scoped write S
seq 20 scoped write S
seq 30 retireScope(S)
seq 40 scoped write S # new lifetime
For a given scope ID, only scoped records with seq greater than its latest committed retirement sequence are live. Memory and SQLite realize this by deleting current rows/maps at retirement and allowing later writes to recreate them. JSONL uses the retirement sequence as its replay boundary (§6).
Retirement is idempotent in the storage sense: retiring a scope with no current state is legal and advances the boundary. A later write still begins a new lifetime. Storage does not maintain an all-time used-ID registry or reject reuse.
A scoped write reaching the commit line after retirement is therefore a new-lifetime write. Harness invocation fencing and settlement drains must continue to ensure a late assistant/tool progress job cannot recreate operation output after terminal retirement. Add explicit tests for this boundary.
export function retireScope(scope: EphemeralScope): Write<SessionScope>;
Storage never infers orphanage from operation absence, namespace, current state, or a missing owner. The only valid lifetime boundary is a committed retireScope write. If an owner never retires its scope, that is an owner defect and the scope remains live.
Crash behavior:
Repository deletion separately removes every physical sidecar belonging to the deleted Session. It does not need operation semantics.
Addresses carry a covariant scope tag; writes carry an invariant scope tag. Preserve value-type invariance independently:
interface Value<T, Sc extends Scope = SessionScope> { /* existing fields + scope */ }
interface ValueList<T, Sc extends Scope = SessionScope> { /* existing fields + scope */ }
declare function value<T>(namespace: string, key?: string): Value<T, SessionScope>;
declare function value<T>(namespace: string, key: string, scope: EphemeralScope): Value<T, EphemeralScope>;
declare function list<T>(namespace: string, key?: string): ValueList<T, SessionScope>;
declare function list<T>(namespace: string, key: string, scope: EphemeralScope): ValueList<T, EphemeralScope>;
A session-scoped address has no runtime scope ID. An ephemeral address carries its scope ID. Scope is part of physical identity: equal namespace/key addresses in session scope and ephemeral scope, or in two different ephemeral IDs, are distinct.
setValue, deleteValue, appendList, and deleteList preserve the address scope in Write<Sc>. Entry and usage writes are session-scoped. retireScope is session-scoped even though applying it deletes ephemeral state.
All writes in an ordinary transaction have one static scope. A session transaction may include session values, entries, usage, and one or more retireScope writes. It may not include a direct ephemeral set/delete/append. An ephemeral transaction may include value/list writes only. At runtime, every ephemeral write in one transaction must carry the same scope ID; two different IDs have the same TypeScript tag and require this assertion.
Make the minimum generic changes necessary to preserve this through Write, CommittedWrite, Storage.commit, Session mutation capabilities, CommitDecision, and lane commands. Follow the variance proof in scopes.variance.ts; do not make readers invariant or propagate unnecessary scope type parameters through read-only APIs.
Memory, JSONL main records, JSONL sidecar records, and SQLite rows share one Session-global sequence space. Every admitted commit remains serialized and receives increasing sequences in admission order. Gaps remain legal.
nextSeq.sessions.next_seq inside the write transaction.nextSeq.JSONL open computes the high-water mark from the main header and every complete record in every matching sidecar before deleting or ignoring retired physical files. A retired sidecar's sequences may then become gaps but can never be reused. Torn uncommitted final transactions do not advance the durable high-water mark.
No reservation record, per-scope sequence counter, or range allocator is permitted.
Step 1 adds only the mechanism required by later tracked-output recovery:
export interface ListElement<T> {
seq: number;
value: T;
tag?: string;
}
export interface ListReadOptions {
cursor?: ListCursor;
order?: "asc" | "desc";
limit?: number;
stopAtTag?: string;
}
export function appendList<T, Sc extends Scope>(
address: ValueList<T, Sc>,
element: NoInfer<T>,
tag?: string,
): ListAppendWrite<Sc>;
Tags are non-empty strings when present. Storage stores and returns the tag without interpreting the element.
Read semantics:
limit elements.stopAtTag.stopAtTag never searches beyond the page limit. If absent from the page, the caller pages again from the last returned sequence. Do not add the draft tag filter; no current consumer needs it and combining filtering with the existing cursor shape is ambiguous.
SQLite may fetch the indexed limit rows and truncate at the first returned tag in TypeScript; storage still never parses payloads. Preserve the existing primary-key query plan with no temporary sort.
This package changes lifetime and routing, not output representation:
pendingToolOutput remains Value<AgentToolResult<unknown>, EphemeralScope>;pendingAssistantFrames remains ValueList<AssistantMessageFrame, EphemeralScope>;operationToolMemo remains a JsonValue scalar;message_update and tool_update event shapes do not change;Do not rename pendingAssistantFrames, introduce pendingAssistantOutput, store WireOp[], add Chord trackers/codecs, redesign AgentHarnessTool, or change output cadence here.
Construct these addresses with ephemeralScope(operationId):
operationToolMemo(operationId, invocationId, name);pendingToolOutput(operationId, invocationId);pendingAssistantFrames(operationId, responseEntryId).All other current built-ins remain session-scoped. In particular, pi.op.state, pi.op.meta, pi.op.tool_args, pi.op.preparation, and pi.pending.entry remain in the main scope because their writes coordinate atomically with lane/operation state.
Remove direct ephemeral deletes from transactions that also write session state:
drive/tools.ts;Current drive/tool-placement.ts has no ephemeral delete; do not invent a change there.
operationCleanupWrites stops scanning tool memos and tool outputs and stops constructing a state-directed assistant-frame delete. It keeps session-scoped operation/meta/args/preparation/pending-entry cleanup and adds exactly one retireScope(ephemeralScope(operationId)) in the universal terminal suffix.
After removing the current staging/terminal scans, remove operationToolMemoPrefix and pendingToolOutputPrefix; current source has no other production consumers. Update the normative exported prefix-constructor count from five to three and update exact constructor tests rather than retaining dead inventory APIs.
An ephemeral-only delete remains legal. For example, the current safe-replay checkpoint delete remains because its transaction contains no session write; fixing its behavior belongs to the later tool-output package, not scoped storage or JSONL optimization.
After a response settles, a deferred response is superseded, or a tool reaches outcome_ready, its former frames/checkpoints/memos may remain physically and logically addressable inside the still-active operation scope. Current scalar operation state no longer references them; no recovery or snapshot path scans the scope to infer authority. The terminal retirement deletes the whole scope.
Call this unreachable scoped residue, not orphanage. Update harness.md invariants that currently require an outcome_ready call to have no physically present memo/checkpoint. The replacement invariant is that settled child state never consumes or exposes residue and terminal retirement removes all scoped state atomically with operation completion.
Close and fault are controlled crashes and never retire the scope. Reopen restores scoped progress only through addresses derived from current authoritative operation state.
The main session file remains unchanged except for committed scope-retirement records. Every main file owns one known sidecar directory at ${mainPath}.scopes; this requires no dirname API or FileSystem extension. Create it lazily with createDir(..., { recursive: true }). Discover scopes with listDir on that exact path. A sidecar filename is exactly scope-${encodeURIComponent(scope.id)}.scope, subject to §2.1's 180-character encoded-component limit. The suffix does not end in .jsonl, and the sidecar directory itself is a directory, so repository listing never mistakes either for a Session.
A sidecar begins with this exact header:
interface JsonlScopeHeader {
v: 4;
kind: "scope_header";
sessionId: string;
scopeId: string;
storageVersion: 1;
}
Validate the exact session/scope identity before replay. Create the first header through a temp file inside ${mainPath}.scopes plus atomic rename. Temporary filenames end in .tmp and are never discovered as sidecars. Use the existing FileSystem capability; do not add a dirname method, extend JsonlStorageOptions, or add Node-only filesystem access to agent core.
Step 1 sidecar writes retain the current object record spelling and carry/validate their scope ID in committed shapes. Each physical transaction remains one complete line or one array line. Sidecar transactions may contain only value/list writes for that exact scope.
The main log records an explicit committed write:
interface CommittedScopeRetireWrite {
kind: "scope";
op: "retire";
seq: number;
scopeId: string;
}
Its Step 1 JSON representation is the corresponding readable object. Do not use the draft ['!', addrId, seq]: a scope is not an interned value/list address. Compact tuple spelling belongs to Step 2 and must be designed there.
Applying a committed retirement removes resident values/lists in that scope. After the complete main transaction is durable, JSONL synchronously serializes lifecycle cleanup on the same commit queue: close any owned sidecar resource, attempt unlink, then release the queue. Failure to unlink is non-fatal to the already-committed business transaction; the sidecar remains logically bounded by the retirement sequence and deletion is retried on reopen.
Open performs these logical stages before admitting writes:
Main transactions have sequence gaps where sidecar transactions occurred. Sidecars have gaps where other scopes/main commits occurred. Both remain valid.
A sidecar containing records after a retirement is an active reused lifetime and must not be deleted. Serialize retirement cleanup against later writes so an old unlink cannot race and remove a newly reused sidecar.
Malformed interior records or identity mismatch in an active sidecar are storage corruption and fail open. A torn final transaction is discarded wholly. Temporary files left by interrupted atomic creation are cleanup artifacts and never become sidecars.
Legacy v3 has no scopes. An ordinary Harness operation commits session-scoped acceptance before any progress write, so it upgrades first in the normal path. Still define and test the generic storage behavior when the first caller write is ephemeral: complete the existing v3-to-v4 main rewrite/usage adjustment under the global commit queue, then commit the scoped transaction without placing it in the main file. A crash before the sidecar append leaves no committed caller transaction and may reuse its uncommitted sequence.
list() ignores ${mainPath}.scopes directories and their files through its existing non-file/.jsonl filtering.delete() removes the main file through its current path, then removes ${mainPath}.scopes recursively with force: true; a scope-directory cleanup failure rejects deletion after the main-file result is known, without inferring Harness state. Missing scope directories are legal.JsonlStorage.close() drains its admitted main/sidecar commits as today; JsonlSessionRepo.close() retains its current lifecycle behavior because repository ownership is a separate package. No progress scope is retired merely because a handle closes.J1 compaction is excluded. When J1 eventually lands, it may omit a historical retirement record only after no physical sidecar can be made live by losing that boundary.
Extend InMemoryStorageState physical value/list identity with scope. Session scope and each ephemeral ID are distinct. Preserve one global nextSeq and current stats behavior.
Applying CommittedScopeRetireWrite deletes every scalar/list in that scope in the same synchronous application of the surrounding transaction. It does not delete entries or usage, which cannot be ephemeral. A later scoped write recreates state under the same ID.
Snapshots expose scope so fork code can reject ephemeral state, but createForkSnapshot keeps its current built-in allowlist and must not start copying generic session-scoped application values or lists. Instrumentation must still report exact committed write order, including retirement.
Change WIP schema in place with no migration:
scalar_values(
session_id TEXT NOT NULL,
scope_id TEXT NOT NULL,
namespace TEXT NOT NULL,
key TEXT NOT NULL,
seq INTEGER NOT NULL,
value TEXT NOT NULL,
PRIMARY KEY(session_id, scope_id, namespace, key)
) WITHOUT ROWID;
list_values(
session_id TEXT NOT NULL,
scope_id TEXT NOT NULL,
namespace TEXT NOT NULL,
key TEXT NOT NULL,
seq INTEGER NOT NULL,
value TEXT NOT NULL,
tag TEXT,
PRIMARY KEY(session_id, scope_id, namespace, key, seq)
) WITHOUT ROWID;
Use scope_id = '' for session scope and the exact scope ID otherwise. Every point read, prefix scan, set/delete, append/delete, snapshot, and query-plan assertion includes scope identity.
Inside the same BEGIN IMMEDIATE transaction, a retirement write executes:
DELETE FROM scalar_values WHERE session_id = ? AND scope_id = ?;
DELETE FROM list_values WHERE session_id = ? AND scope_id = ?;
It still consumes its assigned global sequence through sessions.next_seq, even though no retired-scope tombstone row remains. A later write under that ID recreates rows naturally. Retirement rollback must restore both scoped rows and all sibling main writes.
Fork paths copy only scope_id = ''. Shared-container deletion remains Session-scoped and removes all values/lists regardless of scope along with the other Session rows.
Port tests before or with each implementation slice. Use the repository Vitest binary from the owning package. Do not run the full Vitest suite directly.
SessionScope.EphemeralScope while preserving invariant T.retireScope typecheck.@ts-expect-error.Run identically on Memory, JSONL, and SQLite:
stopAtTag includes the marker;nextSeq and all complete sidecar records contribute to reopen high-water;scope_id; list rows preserve nullable tags;sessions.next_seq advances for retirement;Update existing exact-write tests rather than weakening them:
operationCleanupWrites callers in drive/{response,reconcile,structural,boundary}.ts is covered, including each operation family and cancellation.Likely focused files include:
packages/agent/test/harness/values.test.ts
packages/agent/test/harness/{memory,jsonl}-storage*.test.ts
packages/agent/test/harness/{memory,jsonl}-session-repo*.test.ts
packages/agent/test/harness/runtime/drive-{terminal,retry-deferred,tools,reconcile,generation}.test.ts
packages/session-backends/sqlite-node/test/{storage,storage-conformance,repo,repo-conformance}.test.ts
Use compiler and grep guards to find additional affected tests rather than assuming this list is exhaustive.
Primary files:
CREATE packages/agent/src/harness/session/scope.ts for `SessionScope`, `EphemeralScope`, `Scope`, `ephemeralScope`, scope phantoms/helpers, and runtime scope-ID validation
MODIFY packages/agent/src/harness/session/{types,values,commit,in-memory-storage-state,memory,session,index}.ts
MODIFY packages/agent/src/harness/session/testing/{conformance/storage,instrumented-storage,storage-decorator,gating-storage}.ts
MODIFY packages/agent/test/harness/{values,memory-conformance,memory-storage,storage-backed-session}.test.ts
MODIFY packages/agent/docs/mobile-handoff/01-harness/02-scopes/scopes.variance.ts
stopAtTag behavior.Primary files:
MODIFY packages/session-backends/sqlite-node/src/sqlite/migrations/001_initial.sql
MODIFY packages/session-backends/sqlite-node/src/sqlite/{storage,repo}.ts
MODIFY packages/session-backends/sqlite-node/src/sqlite/session/{values,session-sequences}.ts
MODIFY packages/session-backends/sqlite-node/test/{storage,storage-conformance,repo,repo-conformance}.test.ts
DELETE ... WHERE scope_id retirement.Primary files:
CREATE packages/agent/src/harness/session/jsonl/scope-files.ts for fixed directory/file naming, exact header parsing/serialization, discovery, atomic first creation, torn-tail parsing/repair, and recursive cleanup helpers
MODIFY packages/agent/src/harness/session/jsonl/{types,codec,storage,repo,legacy-v3,index}.ts
MODIFY packages/agent/test/harness/jsonl-{storage,storage-conformance,session-repo,session-repo-conformance,v3-migration}.test.ts
Primary files:
MODIFY packages/agent/src/harness/session/values.ts
MODIFY packages/agent/src/harness/runtime/{progress,lane}.ts to propagate scoped write generics through progress channels and lane commands
MODIFY packages/agent/src/harness/runtime/drive/{terminal,response,deferred,tools,reconcile,structural,boundary}.ts
INSPECT packages/agent/src/harness/runtime/drive/tool-placement.ts; current source needs no scoped cleanup change
MODIFY all focused runtime tests found by the §12 greps
Update current normative/reference docs:
packages/agent/docs/harness.md
packages/agent/docs/values.md
packages/agent/docs/post-wp05-roadmap.md
packages/agent/docs/mobile-handoff/README.md
packages/agent/docs/mobile-handoff/01-harness/02-scopes/scopes.md
packages/session-backends/sqlite-node/README.md
packages/agent/src/harness/telemetry.ts and generated `packages/agent/docs/telemetry-schema.md`: add the `scope` session-write item kind without implementing spans
Do not rewrite historical WP00–WP07 handoffs or released changelog sections. On dev, do not add changelog entries under the repository's main-only rule.
Run:
npm run check
./test.sh
Also run every modified focused test after each slice. Record full command results for the approval checkpoint.
Do not include:
Op/WireOp storage integration;ToolOutput, tool API changes, replay seeding, memo/checkpoint atomicity, progress cadence, rate limiting, exec-env changes;message_update, assistant incremental reducer changes, or protocol replication changes;If implementation requires an excluded item, stop and revise this handoff before expanding scope.
Before final review, inspect every remaining match:
operationToolMemoPrefix
pendingToolOutputPrefix
deleteList(pendingAssistantFrames
deleteValue(pendingToolOutput
scopeId
retireScope
stopAtTag
Expected outcomes:
Step 1 is implementation-complete when:
stopAtTag work identically across backends;npm run check, and ./test.sh pass;Only after explicit approval may Step 1 be committed and pushed. Step 2 starts only after that push is confirmed.