plans/2026-07-16-endless-mode-message-in-a-bottle.md
Date: 2026-07-16 (rev 2 — same day: all four §13 open questions resolved by code investigation; §4/§5/§6/§8/§9 corrected against the actual codebase)
Status: Draft spec, code-verified
Feature name: Endless Mode
One-liner: Sessions survive compaction (and /clear, crashes, restarts) by rendering a parallel transcript — verbatim conversation + observations in place of tool traffic — and pointing the post-compact model at it with a single instruction: read the bottle and continue.
Claude Code sessions don't die at the context limit — they get amnesia. Auto-compact replaces the conversation with a lossy, batch-generated summary written by a summarizer that doesn't know which details matter. The model loses its plan, its promises, its voice, and its knowledge of what was already tried. claude-mem's timeline injection at SessionStart partially recovers long-term memory, but nothing today recovers the conversational position — where the model actually was, mid-work, in its own words.
We do not rely on Claude Code's generated summary. We cannot suppress it (PreCompact hooks are observe-only), so we route around it and mark our injection as authoritative.
A session transcript is 80–95% tool traffic by tokens. The conversation itself — user messages and assistant messages — is the thin, precious part. claude-mem already compresses the tool traffic in real time: that's what observations are. So a near-lossless, 10–20x smaller session record already exists in pieces, at generation time:
bottle = merge_by_position( transcript.jsonl , observations.db )
get_observations.The bottle is a derived view, not a maintained file. Nothing new is captured; the real-time pipeline stays untouched. The file on disk is a render cache — regenerated whole, written atomically, never appended, never a source of truth.
Renders happen worker-side (the worker owns the DB, the SDK agent, and per-session state; hooks stay thin per the existing graceful-degradation contract). Hooks only trigger and inject.
PreCompact hook ──POST /api/sessions/seal──▶ Worker
├── bounded drain-wait on SessionMessageBuffer
├── render bottle (transcript + observations + buffer stubs)
├── write atomically to bottles dir
└── arm injection marker (consumed_at=NULL)
SessionStart(compact|resume) hook ── claim marker ──▶ inject one-line pointer
SessionStart(clear) ── archive bottle, no injection
SessionStart(startup) ── existing behavior, unchanged
Stop hook ── await render-bottle queue-and-return (fire-and-forget effect)
New surface area (all locations verified against the codebase):
| Component | Change |
|---|---|
plugin hooks | New PreCompact handler; SessionStart handler branches on source; one awaited call added to the summarize (Stop) handler |
| Plugin normalization | Thread tool_use_id through: src/cli/types.ts (NormalizedHookInput), src/cli/adapters/claude-code.ts (normalizeInput currently drops it), src/cli/handlers/observation.ts (POST body — the worker route already accepts tool_use_id, SessionRoutes.ts:288-299) |
| Worker HTTP | New BottleRoutes (POST /api/sessions/seal, POST /api/sessions/render-bottle, GET /api/sessions/drain-status) — BaseRouteHandler subclass, constructor-injected, zod-validated, registered in worker-service.ts registerRoutes() |
SessionMessageBuffer | Emit 'mutate' on confirm()/clear() (alongside existing 'message' on enqueue); add waitForDrain() primitive (mirrors existing waitForMessage() SessionMessageBuffer.ts:196-226); add peekStubs() (richer peekTypes) |
| Worker services | New BottleRenderer (transcript parse + merge + markdown emit + stripMemoryTags), with a per-session render mutex (copy ChromaSync.backfillInProgress guard, ChromaSync.ts:913-931) |
| SQLite | New session_bottles table (injection markers + transcript byte cursors). No changes to the observation path required for v1 (§5, §6) |
| Settings | Optional master gate CLAUDE_MEM_ENDLESS_MODE_ENABLED default 'true' in SettingsDefaultsManager (interface + DEFAULTS), checked worker-side (§8) |
Everything else — SDK agent, ResponseProcessor, ChromaSync, timeline, search — untouched.
Stale-doc note:
docs/architecture-overview.mdlists aSessionEnd | session-completehook; it does not exist (no entry inplugin/hooks/hooks.json, no handler insrc/cli/handlers/index.ts). The §8 session-end archive step is new wiring, not an extension — see §13 R5 for the decision.
The bottle is keyed by contentSessionId (Claude Code's ID — invariant for the life of the session, delivered in every hook's stdin, and survives compaction).
Corrected from rev 1: there is no "gather across all memorySessionIds" problem. sdk_sessions holds one row per (platform_source, content_session_id); on worker restart the same row's memory_session_id is UPDATEd in place (SessionStore.updateMemorySessionId :1475-1482), and both observations and session_summaries FKs are ON UPDATE CASCADE (migration v21, SessionStore.ts:1040-1180; PRAGMA foreign_keys = ON at connection.ts:45). All prior legs' observations cascade onto the current memory_session_id automatically. The render needs exactly one join:
contentSessionId → sdk_sessions row → memory_session_id → observations + session_summaries
Community-edge caveat: live DBs may carry observations.content_session_id from community migrations v36–38 that are not in this repo (gap acknowledged at SessionStore.ts:447-451), and it is only partially populated. Main-line code must always use the sdk_sessions join, never observations.content_session_id.
At the moment PreCompact fires — usually mid-turn, since auto-compact triggers during long working turns — the newest tool events may not yet have observations. Observation count per response is variable by design (the agent decides what's worth observing — that variability is the product's magic and must not be constrained). So completeness can never be established by counting observations against tool calls.
We never need to know "how many observations should exist." We need to know what input has been fully processed. That is a drain question, and the machinery already exists — with two corrections from rev 1, both verified in code:
Corrections: (1)
clearPendingForSessionis dead code — zero call sites (SessionManager.ts:207-209exists but is never invoked). The actual completion signal isconfirmClaimedMessages(SessionManager.ts:219-231), called byResponseProcessor.ts:210afterstoreObservations()commits (:126), and at :87 for the prose/no-op path. (2) The pending queue is in-RAM —SessionMessageBufferexplicitly replaced the durablepending_messagesSQLite table (doc commentSessionMessageBuffer.ts:21-41: "If the worker dies, the buffer is gone and recovery is a transcript replay"). The table survives only as migration/cleanup residue. Rev 1's claim that "pending rows persist across worker restarts" is false; the honest-tail machinery below is designed around that.
The drained predicate is one number. claimNext() only marks messages claimed: true (SessionMessageBuffer.ts:186-194); only confirm() removes them (:74-84). A batch stays in the buffer for the entire time the SDK is generating a response for it. Therefore:
buffer.getPendingCount(sessionDbId) === 0 ⟺ queue empty AND no batch in flight
No separate "generating" boolean is needed for the drain test. (For drain-status reporting, generating = session.claimedMessageIds.length > 0 or session.generatorPromise !== null, worker-types.ts:17,23.)
POST /api/sessions/seal { contentSessionId, transcriptPath, timeoutMs? } // default 20_000, max 60_000
Worker behavior (see §13 R2 for the full investigation):
sessionDbId via createSDKSession upsert (house pattern, SessionRoutes.ts:358).buffer.waitForDrain(sessionDbId, deadline, generatorDead) — hybrid wait: wake on the buffer's per-session EventEmitter ('mutate' emitted on confirm/clear), recheck the predicate on every wake, with a 250ms safety tick and hard deadline. Pure event-wait is unsafe because generator death mutates no buffer state; pure polling wastes the instant-wake the existing emitter infrastructure gives for free.ensureGeneratorRunning from seal during a quota pause — the provider is rate-limited and a forced start burns budget on a doomed generation.session && session.generatorPromise === null && pendingCount > 0 — the durable form of this state is the quota pause (GeneratorExitHandler.ts:28-65: quota exit preserves session + buffer with generatorPromise = null; every other exit finalizes and disposes the buffer entirely). Because ingestObservation enqueues before ensureGeneratorRunning (http/shared.ts:121,138), the condition is transiently true on every fresh enqueue — require it to hold across two consecutive wakes (~500ms grace) before returning early.{ sealed: true, gap: 0 }. On timeout or generator-dead → render with honest tail, return { sealed: true, gap: pendingCount }.Within a worker lifetime the buffer itself provides the stubs: every queued and in-flight message retains tool_name, full JSON-parseable tool_input (stripMemoryTags(JSON.stringify(toolInput)), http/shared.ts:114-116), enqueuedAt, and toolUseId (worker-types.ts:63-74, SessionMessageBuffer.ts:7-12). The only missing piece is a read surface: add peekStubs(sessionDbId) returning {message_type, tool_name, tool_input, enqueuedAt, toolUseId, claimed} (~10 lines; current peekTypes :138-143 projects only {message_type, tool_name}). No schema change, no migration. Brief-slice derivation happens at render time from the parsed input: Bash→command, Edit/Write/Read→file_path, Grep→pattern, else first key; skip summarize-type messages (no tool fields).
Across a worker restart the buffer is gone and the queue-derived gap is unknowable. The renderer therefore also derives the tail from the transcript: tool_use entries newer than the newest observation's anchor position render as stubs. Queue-based gap is authoritative within one worker lifetime; transcript-based comparison is the floor that always works.
Rendered form:
*Recent activity (observations pending — will appear in the timeline shortly):*
- ⏳ Bash — `npm test` (14:32:11)
- ⏳ Edit — src/services/worker/SessionManager.ts (14:32:40)
Three reasons this gap is acceptable by construction:
The seal never blocks compaction and never blocks the user: on any worker failure the hook follows the existing contract (transport errors → exit 0) and the injection falls back to the current timeline-only context block. Endless Mode degrades to today's behavior, never below it.
For chronological merge, observations need a source position, not just a generation timestamp.
Status (verified): the anchor is already half-shipped. Drain yields _originalTimestamp = enqueuedAt (SessionMessageBuffer.ts:160-164); SessionManager tracks the batch minimum as earliestPendingTimestamp (:362-367); ClaudeProvider.ts:355 passes it as originalTimestamp; ResponseProcessor.ts:126-135 passes it to storeObservations as overrideTimestampEpoch → stored as observations.created_at_epoch (SessionStore.ts:2088-2100). Stored observations are already stamped with batch-min tool time, not generation time.
Remaining work, split in two:
tool_use_id through the Claude Code hook path — it arrives in hook stdin (PostToolUseHookInput.tool_use_id, SDK types) but claude-code.ts normalizeInput (:15-25) drops it. Add toolUseId? to NormalizedHookInput (src/cli/types.ts), map it in the adapter, include it in the observation POST body (observation.ts:18-31) — the worker route already accepts and forwards it (SessionRoutes.ts:288-299,308-332; the transcript-replay path already sends it, transcripts/processor.ts:205,233,254). Optionally also send occurred_at_epoch: Date.now() from the hook so the anchor is tool-completion time rather than worker-receipt time. Backward-compat is trivial in both skew directions: the route schema is .passthrough() with optional fields.anchor_ts_min/max, tool_use_id) on observations — requires a nullable-column migration plus ResponseProcessor/storeObservations signature changes. Not needed for v1 rendering.Positioning fallback that costs nothing: both observations and user_prompts carry prompt_number. Grouping observations under their prompt number (ordered by created_at_epoch within the group) gives exact conversational positioning with no transcript and no new columns — this is also what makes degraded bottles (§10) properly ordered. Batch-range precision within a turn is inherently approximate (the SDK agent writes free-form against batches); the bottle needs chronologically plausible placement, not exact causality.
~/.claude-mem/bottles/<contentSessionId>.md # active
~/.claude-mem/bottles/archive/<ts>-<id>.md # archived on /clear or session end
Per-session filename — concurrent sessions in one project must not share a file.
Rendered whole, written to a temp sibling, rename()d over the target (atomic on POSIX; precedent: src/shared/atomic-json.ts:50,100). Never appended. Per-session render mutex (copy the ChromaSync.backfillInProgress static-guard pattern, ChromaSync.ts:913-931): a Stop-render and a seal-render for the same session must not interleave — without the mutex, two concurrent renders are last-writer-wins with different cursors.
The renderer keeps a per-session byte-offset cursor into the transcript (session_bottles.transcript_cursor) so routine re-renders parse only new bytes — precedent is working code: JsonlTailer (src/services/transcripts/watcher.ts:10-72, offset high-water mark with truncation reset). Two honest caveats from investigation:
readFileSyncs the entire transcript on every Stop, transcript-parser.ts:15 — full-file I/O per Stop is established practice).Every rendered user and assistant message MUST pass through stripMemoryTags (src/utils/tag-stripping.ts:4-17 — strips <private>, <claude-mem-context>, ``, …). Everything claude-mem persists today honors this contract (prompt storage SessionRoutes.ts:444-461 including the entirely-private skip; summarize path summarize.ts:85,94); a verbatim transcript render is the one new path that would bypass it. The filter lives in BottleRenderer so it applies identically to seal-time, Stop-time, and on-demand renders. Beyond tag-stripping, the bottle is a filtered subset (text blocks only) of a transcript that already exists in plaintext on the same disk — no new data class (docs/ip-boundary.md is licensing-scope only; no data-at-rest encryption commitment exists to violate).
tool_use, tool_result, and thinking blocks.<task-notification>, <context_guidance>, <tool-result>, command wrappers), mirroring the existing prompt-capture filter.# Session bottle — <contentSessionId>
project: <cwd> · started: <ts> · legs: <compact_count + 1> · rendered: <ts>
mode: full (transcript + observations)
## Original request
<first genuine user prompt, verbatim>
---
**User** (10:32)
<verbatim>
*What happened:*
- [#102401] Explored hooks/ — every platform hook wraps 5 shared cores
- [#102402] Read start.mjs — 6 self-heal layers before MCP boot
- ⏳ Bash — `npm test` (observation pending)
**Assistant** (10:41)
<verbatim>
---
**User** (10:44)
...
Rules:
[#id] title — first clause of narrative. The ID makes every line losslessly expandable via search tools. Never inline full narratives.mode: header line is load-bearing — degraded bottles (§10) declare themselves.session_summaries content may appear only as a visibly-generated block (> Session summary (generated by claude-mem — not verbatim)), never in an **Assistant** speaker slot (§10 provenance rules).Input: stdin { session_id, transcript_path, cwd, trigger, custom_instructions }.
POST /api/sessions/seal { contentSessionId, transcriptPath, timeoutMs: 20_000 }.session_bottles: content_session_id, bottle_path, armed=1, consumed_at=NULL, gap).source)compact / resume: atomically claim the marker —
UPDATE session_bottles SET consumed_at = ?
WHERE content_session_id = ? AND armed = 1 AND consumed_at IS NULL
RETURNING bottle_path, gap;
SessionStart can double-fire around a compact; the atomic claim guarantees exactly-once injection. On a claimed row, inject the pointer (§9). On no row (compact happened with worker down, or resume of an old session): request a synchronous render (POST /render-bottle { wait: true } — the fire-and-forget default won't do; the file must exist before the pointer is injected), which internally walks the degradation ladder (§10); if even that fails, inject the existing timeline context (today's behavior).
clear: archive the bottle (timestamped move), inject nothing. /clear means forget.
startup: existing behavior, unchanged.
Hook-in point: src/cli/handlers/summarize.ts, immediately after the isWorkerFallback(queueResult) check (:147-149), before the return:
try {
await executeWithWorkerFallback('/api/sessions/render-bottle', 'POST',
{ contentSessionId: sessionId, transcript_path: transcriptPath, platformSource },
{ timeoutMs: 5000 });
} catch { /* render failure must never escalate to BLOCKING_ERROR */ }
process.exit and kill un-awaited POSTs mid-flight (hook-command.ts:144-156, worker-utils.ts:648) — house style for fire-and-forget from a hook is await a queue-and-return route. The worker responds {status:'queued'} immediately (copy handleSummarizeByClaudeId's shape, SessionRoutes.ts:347-383) and renders async with .catch(logger.error(..., '(non-blocking)')) (precedent worker-service.ts:594-606), under the per-session mutex.transcriptPath is undefined (the input.lastAssistantMessage branch, summarize.ts:84-85).async: true, timeout: 120 (plugin/hooks/hooks.json) — invisible to the user — and already reads the entire transcript synchronously every Stop. One extra localhost queue-and-return call is negligible next to the LLM summarize generation the same Stop already triggers.'true' (CLAUDE_MEM_TRANSCRIPTS_ENABLED, CLAUDE_MEM_CHROMA_ENABLED, …; 'false' is reserved for experimental injection or user-repo writes, SettingsDefaultsManager.ts:99-186); a render-on-Stop toggle wouldn't protect anything (seal-time renders write the same file); and render-on-Stop is the stub-tail healing pass — gating it off silently degrades sealed-bottle quality. If a kill-switch is wanted, make it the Endless Mode master gate: CLAUDE_MEM_ENDLESS_MODE_ENABLED: 'true', checked worker-side (hooks resolve settings from env+defaults only; the worker is where loadFromFile honors ~/.claude-mem/settings.json).No SessionEnd hook currently exists (see §3 note). v1: archive bottles lazily — a startup-branch sweep moves bottles whose sessions are no longer live. Adding a real SessionEnd hook is §13 R5's open decision.
Exactly one block, small, and framed as authoritative:
# [claude-mem] Endless Mode — session continuation
Before doing anything else, Read this file and continue the session from
where it ends:
<absolute bottle path>
(Page with offset/limit if it is long — read the tail first if you must
prioritize.)
- It is the authoritative session record: verbatim conversation, with
observations in place of tool activity. Where it conflicts with the
system-generated summary above, the bottle wins.
- The final assistant message in it was already delivered to the user.
Do not repeat it or redo work it describes as done — continue from its
end state.
<if gap > 0>
- The last <N> tool events have observations still being written; they
will appear in the timeline shortly.
</if>
Current task: <one-line, from latest session summary or last user prompt>
Degraded-mode variant (§10): the "authoritative verbatim conversation" and "do not repeat your final message" lines are false for a reconstructed bottle and must be swapped:
- It is a partial reconstruction: your messages to the user were not
preserved — only the user's messages are verbatim. Session summary
blocks are generated, not your words. Do not assume prior phrasings.
- The last session summary describes where you left off; continue from
there.
Design notes:
The premise is empirically real (measured on a live install): 97.6% of stored sessions (5,278/5,407) are older than the oldest surviving transcript (~30 days — consistent with Claude Code's documented cleanupPeriodDays default of 30; assumption, not found in-repo). Of those old sessions, 4,361 have ≥1 stored user prompt, 2,358 have ≥1 observation, 1,492 have ≥1 session summary. Timeline injection cannot reconstruct any specific old session; a degraded bottle can.
| Source | Recoverable | Caveats |
|---|---|---|
user_prompts (SessionStore.ts:330-339) | User messages, verbatim, keyed by content_session_id, numbered (prompt_number), timestamped | 4,000-char cap (prompt-storage.ts:4-21, ~3.5% of prompts truncated with trailing …); memory tags stripped; fully-private prompts skipped; dedupe window |
observations | Full richness: type, title, subtitle, narrative, facts, concepts, files_read/modified, prompt_number | Timestamps are batch-anchored or generation-time; position by prompt_number (exact), show coarse times only |
session_summaries (SessionStore.ts:614-629) | request / investigated / learned / completed / next_steps / notes — the only assistant-perspective narrative that survives | LLM-generated, not verbatim — provenance rules below |
sdk_sessions | project, first prompt, custom_title, started_at | — |
| Assistant messages | Not recoverable, anywhere. Nothing persists assistant output raw; pending_messages.last_assistant_message rows are deleted after processing | Permanently lost with the transcript |
prompt_number: **User** (verbatim) → *What happened* (observations) → > Session summary (generated). Same file format, same injection pointer, mode: reconstructed header.Detection: hooks pass transcript_path when they have it → existsSync. transcript_path is not persisted in any table (arrives in hook stdin, dies with the hook), so for the CLI path (npx claude-mem bottle <contentSessionId>) glob CLAUDE_CONFIG_DIR/projects/*/<contentSessionId>.jsonl — Claude Code names transcripts by contentSessionId; the convention is already codified at ObservationCompiler.ts:199. Miss → check sdk_sessions + prompt/observation counts → degraded vs timeline.
request/learned/completed prose reads exactly like assistant voice; rendered in an **Assistant** slot, the post-resume model will "remember saying" things it never said. Always the quoted generated-block form.mode: reconstructed (transcript unavailable) — user messages verbatim; assistant messages not preserved; summaries are generated.[truncated at 4000 chars] marker so the stored … isn't read as the user trailing off.Inherits the existing contract wholesale:
Server.close() calls closeAllConnections() first (Server.ts:159-178) — the in-flight seal request is severed, the hook sees a transport error, exits 0. Buffered work dies with the process (in-RAM by design); recovery is transcript replay; the transcript-side stub derivation (§5) reports the gap honestly on the next render.confirm() and renders gap=0 within budget. Repeat with a quota-paused session (generatorPromise === null, count > 0); assert early return with gap>0 after the two-wake grace, not after the full timeout — and assert the seal did not call ensureGeneratorRunning.ensureGeneratorRunning runs); assert the two-wake grace prevents a spurious generator-dead verdict.<private> blocks in user and assistant messages; assert the bottle contains none of the private content on all three render paths (seal, Stop, on-demand)./clear: assert archive + no injection.mode: reconstructed header, prompt_number interleaving, summaries in generated-block form (never **Assistant**), truncation markers, and the degraded injection variant.Four parallel investigations against the live codebase resolved rev 1's open questions:
R1 — Honest-tail stubs: no schema change needed. The in-RAM buffer already retains tool_name, fully parseable JSON tool_input, enqueuedAt, and toolUseId for queued and in-flight messages; only a read surface (peekStubs(), ~10 lines in SessionMessageBuffer) is missing. Companion plugin-side fix: the Claude Code adapter drops tool_use_id on the floor (claude-code.ts:15-25) while the worker route already accepts it — thread it through 3 files (types.ts, adapter, observation.ts). Per-observation anchor columns are a separate v2 migration; the batch-min anchor already ships (§6).
R2 — Seal wait: hybrid event + poll, on corrected foundations. clearPendingForSession is dead code; the real signal is confirmClaimedMessages post-storeObservations. Drain predicate = getPendingCount() === 0 (claimed batches stay counted until confirmed). No internal event bus exists (SSE is outbound-only; the one onPendingMutate slot is taken by worker-service) — but the buffer's per-session EventEmitters and the waitForMessage pattern are the right layer: emit 'mutate' on confirm/clear, add waitForDrain with a 250ms tick (which also catches generator death, which mutates no buffer state) and a two-wake grace for the enqueue-before-ensureGenerator race. Quota pause is the durable generator-dead-with-work state; never force-start it from seal.
R3 — Render-on-Stop: default-on, no dedicated setting. The Stop hook is async: true/120s and already reads the whole transcript every Stop; the render adds one awaited localhost queue-and-return call (un-awaited POSTs die at process.exit — house rule). House precedent defaults cheap local features to 'true'; a toggle wouldn't protect anything (seal renders the same file); and render-on-Stop is the healing pass for sealed-bottle stubs. Optional master gate: CLAUDE_MEM_ENDLESS_MODE_ENABLED in SettingsDefaultsManager, enforced worker-side. New hard requirement discovered: BottleRenderer must apply stripMemoryTags to every rendered message — the verbatim render is the one path that would otherwise bypass the <private> contract every other persistence path honors.
R4 — Old sessions: degraded bottle, decisively better than timeline fallback. Verbatim user prompts survive in user_prompts (keyed by contentSessionId, prompt_number-ordered, 4,000-char cap); observations survive with prompt_number giving exact interleaving; session_summaries provides the only (generated) assistant-perspective narrative; verbatim assistant messages are unrecoverable. Ladder: full → reconstructed → timeline, with mandatory provenance marking (mode header, generated-block summaries, truncation markers, degraded injection variant). Also corrected §4: the memorySessionId cascade means one join, not a multi-ID gather.
R5 — Remaining open decisions (small):
observations anchor-column migration (anchor_ts_min/max, tool_use_id) — bundle with the next schema bump rather than standalone.occurred_at_epoch from the hook (tool-completion time vs worker-receipt time) is worth the extra field in v1 — recommend yes, it's one line at each layer.