packages/agent/docs/work-packages/08-named-branch-streaming-forks.md
Status: in progress — implementing Slice A.
This package replaces the fork contract: ForkOptions gains a mandatory scope and a mandatory named source branch, branch forks validate a complete configured source AgentLane and ancestry membership, tree forks copy the complete immutable tree plus current application values/lists, and all three backends replace materialized source snapshot arrays with bounded-memory streaming copies. A JSONL fork never repairs or mutates its source. One closed core classifier owns every namespace's fork disposition.
WP07 is a hard dependency and is preserved unchanged: no-create database modes, canonical (containerPath, sessionId) identity, independent read-only WAL reader for external/live-worker sources, repository-local deletion reservation, and all-settled close. This package supersedes only the roadmap's "SQLite fork cost" performance item.
Read completely before editing:
packages/agent/docs/harness.md §§0.6, 1.3–1.7, 2.3, 2.7–2.9, Part 9 (invariants 3–5, 13, 16; ledger completeness).packages/agent/docs/values.md, especially "Forks and rewrites" and the backend sections.packages/agent/docs/post-wp05-roadmap.md (SQLite fork cost, repository lifecycle context).packages/agent/src/harness/session/fork.ts, types.ts (ForkOptions, SessionRepo), values.ts, in-memory-storage-state.ts, memory.ts, session/index.ts.packages/agent/src/harness/session/jsonl/repo.ts, jsonl/storage.ts, jsonl/codec.ts, jsonl/legacy-v3.ts, jsonl/types.ts.packages/session-backends/sqlite-node/src/sqlite/repo.ts, storage.ts, session/values.ts, session/entries.ts, session/branch-entries.ts, types.ts.packages/agent/src/harness/session/testing/conformance/session-repo.ts and every test named in §5.packages/agent/src/harness/session/testing/benchmark/session-repo.ts and both session-repo.bench.ts files.Do not use dist/ output as implementation input. WP00–WP07 documents and released changelog sections are immutable in this package.
export type ForkOptions =
| { scope: "branch"; branch: string; entryId?: string; position?: "before" | "at"; id?: string }
| { scope: "tree"; id?: string };
scope is required. branch is required for branch scope. There is no default scope, no implicit main, and no compatibility alias.
Branch scope requires a complete configured source AgentLane: pi.branch.tip/{branch}, pi.lane.config/{branch}, and pi.lane.state/{branch} must all exist. A missing tip rejects (unknown branch); a data-only Branch (tip without config/state) rejects; a partial config/state pair, or laneConfig/laneState present without a branchTip, is corruption and rejects. entryId, when supplied, must be an entry on that Branch's current tip ancestry (inclusive); an entry elsewhere in the tree rejects. Omitted entryId means the current tip. position defaults to "at"; "before" selects the target's parent and may yield a null destination tip (before the root entry, or a null source tip with no entryId) — legal. The destination contains exactly one Branch with the same name, the selected tip, the copied LaneConfiguration, and fresh idle lane state { currentOperationId: null, lastOperationId: null, inbox: [] }. No other Branch or lane exists in the destination.
This deliberately makes branch forks of data-only Branches — including legacy v3 imports whose main lacked reconstructable model/thinking history — unavailable. Tree scope remains available for those sources.
Tree scope copies: every immutable entry, including entries unreachable from every current Branch tip; every Branch tip verbatim; every configured lane's config plus fresh idle lane state under the same name; data-only Branches (tip present, config/state absent) as data-only Branches. A partial config/state pair, or laneConfig/laneState without a branchTip, is corruption in both scopes: tree scope rejects it rather than dropping it.
Both scopes: copy pi.session.name; copy pi.entry.label values only for copied entries; exclude the usage ledger, pi.result, all pi.op.*, all pi.pending.* (entries, tool checkpoints, assistant frames), and every trace of open-operation state. Destination usage totals start at zero; destination messageCount equals the number of copied message entries, exactly as current conformance already proves. Destination metadata records parentSessionId = source.id. Copied entries, values, and list elements preserve their original seq; the destination nextSeq high-water mark is the source's, so no sequence is reused. Transformed built-ins keep coherent seqs: the rewritten branchTip and each fresh idle laneState reuse the seq of the source's current corresponding value row, and copied config, name, labels, application values, and list elements retain theirs — destination writes stay in original seq order under an unchanged high-water mark.
Application values/lists (outside the reserved pi/pi.* namespaces): tree scope copies every current scalar value and every surviving list element with original seqs; branch scope copies none. A seq <= tipSeq cutoff is forbidden as "historical reconstruction" — it is provably not one:
Scalar: TX[seq 10: set my-app.state = v1] · TX[seq 12: insert e1] ·
TX[seq 50: set my-app.state = v2] (v2 describes work after e1)
Branch fork at e1, cutoff seq <= 12: only v2 exists (v1 was replaced, no
history is retained); 50 > 12 excludes it → state absent, though the app
demonstrably had state v1 at the fork point. The cutoff cannot recover v1.
List: append seq 5 · append seq 20 · deleteList seq 40 · append seq 60
Cutoff seq <= 30: elements 5 and 20 no longer exist (whole-list delete
destroyed them); only 60 survives and is excluded → empty list, though the
list held {5, 20} at seq-30 time.
Any cutoff filters survivors, silently mixing rewound intent with post-deletion reality. Branch scope therefore copies nothing application-owned, matching its fresh idle lane state and zero ledger; applications own their own re-derivation.
All namespace fork knowledge lives in one core module beside session/values.ts and session/fork.ts (e.g. session/fork-policy.ts). It is a closed switch, not a registry, plugin policy, or DSL:
pi.op.*, pi.pending.*, pi.result → exclude, both scopes.pi.session.name → copy.pi.entry.label → copy iff the keyed entry is copied.pi.branch.tip, pi.lane.config, pi.lane.state → structured lane actions; the scope-specific rules (branch keeps only the named lane and rewrites its tip; lane state is always replaced with fresh idle state; tree keeps all) live in one shared driver consumed by all backends.pi and any other pi.* namespace → the fork fails, but only when current surviving state exists at fork time (a current scalar row or a surviving list element). Historical writes later replaced or deleted are absent from current state on every backend and must not alone fail a JSONL fork — behavior is backend-equivalent. Introducing a new built-in namespace without declaring its fork semantics must break fork tests, not silently copy or drop state.pi nor pi.* → application: copy on tree, exclude on branch. The only built-in list namespace (pi.pending.assistant_frame) excludes; application lists follow the application rule.The driver exposes a streaming shape — accept one committed value/list write (or current row), emit zero or more destination writes, finish() emits fresh idle lane states and the rewritten branch tip — holding only bounded state (lane names, destination tip). Entry-copy membership is a backend-supplied predicate so each backend uses its own index. createForkSnapshot, forkSnapshotWrites, ForkSourceSnapshot, ForkDestinationSnapshot, and the entriesComplete escape hatch are deleted.
Auxiliary memory during a fork must stay bounded regardless of source size. The unavoidable exceptions are Memory's destination state and the returned open destination Session (JSONL destination state after open). Source copy paths must not call array-returning full reads: no snapshotEntriesAndValues(), captureForkSource(), readAllScalarValueRows(), readAllEntryRows(), whole-file readTextFile, or SQLite .all() over unbounded row sets.
Memory. At a source commitQueue boundary, build the destination InMemoryStorageState directly by iterating source maps once through the classifier/driver. Branch scope walks parentId from the branch tip toward the root to compute the ancestry id set and verify entryId membership; copy exactly that set. No intermediate snapshot arrays; MemoryStorage.fromSnapshot and captureForkSource are removed.
JSONL. The source is read through a read-only path that captures a fixed file prefix — open a read handle, record the current length at capture time, and read only that prefix — and never writes the source: no torn-tail truncation/rewrite, no v3 normalization persistence, no .tmp beside the source. A torn or incomplete final line inside the captured prefix is discarded in memory. The source nextSeq high-water mark is derived as max(header.nextSeq when present, highest complete captured write seq + 1) — ordinary headers are not rewritten on every append, and a torn incomplete final write never advances it.
finally on success and failure). Pass 1 streams every line and folds value/list writes into the index: per scalar address, the seq of the current surviving set (absent after a trailing delete); per list address, the set of surviving element seqs (whole-list delete clears it); plus entry id → parentId (parents always precede children) where branch scope needs ancestry. The fold applies the same current-state semantics replay does, without whole-state RAM.set whose seq equals the index's current-row seq for its address; a list append whose element seq is in the survivor set. Deletes, superseded sets, and dead application value/list history are never emitted as destination state — the destination file contains only current selected rows, already in original seq order because source lines are seq-ordered.branchTip row for a kept branch it emits the (possibly rewritten) tip under that same seq, and when it reaches the seq of the current laneState row for a kept configured lane it emits fresh idle lane state under that same seq. No tail appends exist, so replay stays seq-monotonic and the high-water mark is untouched.entryId membership before pass 2; labels emit only for member entries.publishFileAtomically); failure removes the temp file and the index. Legacy v3 splits by source state. Forking an open legacy-v3 JsonlStorage rejects with a clear error until a normal non-empty commit has upgraded and persisted its normalized format-4 ids: v3 normalization mints nondeterministic UUIDv7 tails, so an independent disk reparse cannot reproduce the ids the open Session exposes or validate a caller entryId; the fork must not mutate/upgrade the source itself. A closed legacy-v3 source uses the bounded fixed-prefix disk-backed parser/normalizer and never touches the file: tree forks succeed; branch forks may use an omitted entryId (the default normalized tip) when the import reconstructs a complete configured lane, while any caller-supplied process-local id from an earlier open is not stable across reparses and rejects normally; a data-only reconstructed lane rejects as everywhere. Ordinary writable JsonlStorage.open may retain in-memory normalization, but resident state is never a fork source.commitQueue whose only job is to capture the fixed read-only file-handle prefix (open handle, record length), then releases the queue; the fork runs the same two-pass disk-backed procedure over that prefix while later source appends proceed. Committed writes are durable in the file before they are applied to resident state, so the prefix is authoritative at the boundary, and file order preserves seq order without resident-Map ordering ambiguity.SQLite. Iterator-based row transfer through a bounded-memory temporary on-disk SQLite staging database, used uniformly for per-file and shared-container layouts. The source reader never streams directly into the destination transaction — a WAL reader and one writer can coexist, but in shared-container mode the destination writer would hold the container's sole write lock and block the post-boundary source writer while capture is still streaming; staging into a separate file removes that coupling. Use prepared-statement iteration (iterate/stepwise), never source-sized .all() snapshot arrays:
openReadOnly on the exact canonical path, one deferred read transaction, session row and storage version validated inside it.commitQueue whose only job is to BEGIN and establish the independent reader's snapshot (issue a trivial read) before releasing the queue. Do not begin a read transaction on the source writer connection and then release its queue, and do not hold the queue for the copy duration.ORDER BY seq and classifier-selected current scalar/list rows (SQL-level namespace prefilters matching the classifier — enumerated built-in namespaces plus the application predicate excluding exact pi and pi.% — with each row still passing the classifier) into the temporary staging database in bounded batches. Branch scope enumerates the ancestry through the branch_entries segment chain up to the selected entry and answers entryId and label membership through that index, not an in-RAM id set. Later source commits use the original writer connection and may complete while staging streams, in both layouts, because stage writes target another file.BEGIN IMMEDIATE transaction — entries in seq order maintaining the destination branch index and message_count incrementally, then values/list elements — and delete the staging database in finally on success and failure. Destination next_seq is the source's. Shared-container destinations write only the new session's rows.Destination id reservation across create/open/fork/delete, no-create opens, foreign-metadata rejection, per-file and shared-container layouts, WAL commit-boundary wholeness (a source commit is wholly inside or wholly outside one fork), and all-settled repository close are unchanged.
/fork, /clone, and --fork run entirely on the legacy SessionManager (createBranchedSession, forkFrom) and are not migrated here. Future mapping when coding-agent adopts SessionRepo: /fork → { scope: "branch", branch: "main", entryId, position: "before" }; /clone → { scope: "branch", branch: "main" }; --fork → { scope: "tree" }.
ForkOptions defaults scope to "branch" and hard-codes source/destination main (fork.ts, both backend branch readers).selectForkContents walks parents from any supplied entryId anywhere in the tree and labels the result main.snapshotEntriesAndValues() (Memory/JSONL), readAllScalarValueRows + readAllEntryRows/scanBranchEntries arrays (SQLite), all funneled through in-memory createForkSnapshot.createForkSnapshot renumbers copied scalar values with fresh seqs and computes nextSeq as max entry seq + 1, losing original value seqs.JsonlStorage.open(), which rewrites the source file on a torn tail — a fork can mutate its source today.values.md explicitly defers it).fork.ts, values.md prose, and conformance assertions; nothing forces a new pi.* namespace to declare fork semantics.entriesComplete?: false exists only to let SQLite branch snapshots skip tip validation for other branches.ForkOptions and validation exactly as §1.1; all rejection paths create no destination file, database rows, or reserved-but-leaked ids.pi or undeclared pi.*) fail the fork only when current surviving state exists, identically on every backend.createForkSnapshot/captureForkSource/snapshot() fork plumbing and their exports removed from session/index.ts and the sqlite-node import surface.nextSeq high-water on every backend (JSONL derives it per §1.3); identical logical destination state across backends for identical sources (conformance).harness.md §2.7 (and the §1.7 fork-related sentences), values.md "Forks and rewrites" and backend snapshot mentions, post-wp05-roadmap.md (retire the SQLite fork-cost item, add/point to this package), packages/session-backends/sqlite-node/README.md fork paragraphs, packages/agent/benchmark/session/README.md if dataset wording changes. Historical WP docs and released changelogs untouched. Format/storage versions unchanged; no migration.Files: session/types.ts, new session/fork-policy.ts, session/fork.ts (rewritten or deleted), session/values.ts (only if new helpers are needed), session/index.ts, session/memory.ts, session/in-memory-storage-state.ts, conformance testing/conformance/session-repo.ts, test/harness/memory-conformance.test.ts, test/harness/memory-session-repo.test.ts.
ForkOptions; implement validation and the classifier/driver.Files: session/jsonl/storage.ts, session/jsonl/repo.ts, session/jsonl/types.ts, streaming reader support in the FileSystem capability (harness/types.ts, harness/env/*) if required, test/harness/jsonl-session-repo.test.ts, jsonl-session-repo-conformance.test.ts, jsonl-storage.test.ts, jsonl-v3-migration.test.ts.
commitQueue boundary for open sources; no source writes on any fork path.Files: sqlite/repo.ts, sqlite/storage.ts, sqlite/session/values.ts, sqlite/session/entries.ts, sqlite/session/branch-entries.ts, sqlite/types.ts, test/repo.test.ts, test/repo-conformance.test.ts.
SqliteStorage.snapshot()/array snapshot helpers with the independent-reader-plus-boundary-callback design and iterator transfer into the temporary staging database, then stage-to-destination publication with finally cleanup.Files: testing/benchmark/session-repo.ts, both session-repo.bench.ts files, benchmark/session/README.md, harness.md, values.md, post-wp05-roadmap.md, sqlite-node README.md, changelogs only under normal branch rules.
Update fork option literals, add large-source fork benchmarks (tree and branch), and land the §3.6 documentation set.
entryId not on the named Branch's tip ancestry (present elsewhere in the tree); entryId unknown; entryId with a null tip. Each rejection leaves no destination artifact and releases its reserved id.entryId (tip), explicit tip, mid-ancestry entry, position: "before" at a mid entry and at the root entry (null destination tip), null source tip with no entryId.parentSessionId set.pi.result, pi.op.*, pi.pending.* (pending entries, tool checkpoints, assistant frame lists), and usage rows absent; exact stats expectation — a fork copying N message entries reports getStats() = { messageCount: N, usage: all-zero }, matching current conformance; the ledger-completeness invariant ("a fork's ledger starts at zero") retained.pi.* scalar and every surviving list element with original seqs; list cursors from the source page identically in the destination; a list deleted-then-reappended in the source reproduces only survivors.pi or an undeclared pi.* scalar or surviving list element) fails the fork on every backend; the same namespace set then deleted before the fork fails no backend — including JSONL, where the dead history remains as physical lines and the disk fold must classify it as absent. Construct both cases via raw committed writes.nextSeq equals the source high-water mark and the first post-fork commit allocates above it on every backend (JSONL header nextSeq, SQLite next_seq).max(header.nextSeq when present, highest complete captured write seq + 1); a torn incomplete final write does not advance it.snapshotEntriesAndValues, captureForkSource, readAllScalarValueRows, readAllEntryRows, whole-file readTextFile, or source-sized .all() arrays on entries/values/lists — bounded iterator steps and staging-database operations are permitted; deterministic large fixtures (extend the existing benchmark dataset generators) exercise tree and branch forks over multi-thousand-entry sources..tmp appears beside the source; temporary ancestry index files are removed on success and failure.entryId succeeds when a complete configured lane is reconstructed, a caller-supplied id from an earlier open rejects, and a data-only reconstructed main rejects.commitQueue boundary: a commit admitted before the boundary appears wholly; a source append completed after boundary capture proceeds while the fork is still streaming and is wholly absent from that fork.commitQueue establishes its snapshot and releases the queue, and a subsequent source commit on the writer connection completes while the reader is still streaming into the stage and is wholly absent from that fork; a later fork includes it wholly. Assert this in both per-file and shared-container layouts (satisfiable in shared containers because stage writes target another file), plus the WP07 live external-worker case unchanged.After each slice: npm run check, then the modified focused tests via the repository Vitest binary from the owning package; ./test.sh before final review. Grep guards: no remaining createForkSnapshot|captureForkSource|ForkSourceSnapshot|entriesComplete outside historical docs and this handoff; no production fork source path referencing "main" as a literal (this document and the recorded coding-agent mapping in §1.5 intentionally mention main).
Review checkpoints (delegated reviews use provider anthropic, model claude-fable-5):
Do not include:
main;seq <= tip cutoff or other historical-state reconstruction;pi.result, or any open-operation state under any option;/fork, /clone, --fork, or RPC migration;SessionRepo interface changes beyond ForkOptions (repository close() belongs to the lifecycle package);If implementation requires an excluded item, stop and revise the handoff.
WP08 is complete when:
ForkOptions is exactly the §1.1 union with the stated validation, on all three backends;pi or undeclared pi.*) fail forks exactly when current surviving state exists, equivalently on all backends;BEGIN IMMEDIATE after reader close, stage deleted in finally), with the independent-reader boundary design and concurrent later writer commits proven in both layouts, preserving all WP07 behavior;nextSeq high-water marks are preserved and backends produce identical logical destinations;npm run check, and ./test.sh pass;