plans/inbox/2026-05-21-viewer-screenshot-references.md
Status: inbox
Created: 2026-05-21
Goal: Surface screenshot images (PNG/JPG/GIF/WebP) referenced by tool calls as inline thumbnails in the observation feed at the worker viewer (http://127.0.0.1:<worker-port>), with a click-to-zoom lightbox. Worker serves the image bytes from disk through a path-safe endpoint.
When a session uses screenshot tools (gstack/browse screenshot, MCP screenshot tools) or reads/writes image files, those paths flow through pending_messages.tool_input but are dropped before reaching the observations table. The viewer feed only sees text summaries — never the actual images that were captured. We want screenshots inline in the feed.
127.0.0.1, so an image endpoint that reads arbitrary local paths is reachable only by the user — but path-traversal must still be guarded (defense in depth, and to avoid leaking unrelated files via a malicious request from a browser tab).observations must be nullable, no migration of historical rows required.src/services/sqlite/schema.sql:57-92 — observations table. Columns include files_read, files_modified (both TEXT, JSON-stringified arrays), metadata (TEXT, JSON), facts, narrative, concepts. UNIQUE(memory_session_id, content_hash) with ON CONFLICT DO NOTHING.src/services/sqlite/schema.sql:126-152 — pending_messages table holds raw tool_input / tool_response JSON during processing. Discarded after the AI summarization step.src/services/sqlite/observations/store.ts:19-80 — storeObservation() writes 17 columns from ObservationInput. Raw tool I/O is not persisted here today.ObservationRow in src/services/sqlite/types.ts — what every observation API returns.src/services/server/Server.ts:98-105 — setupCors(), body-parser, route registration. CORS already on.GET /api/observations registered in DataRoutes.ts via handleGetObservations → paginationHelper.getObservations(...).GET /api/search/observations in src/services/worker/http/routes/SearchRoutes.ts:109.src/services/worker/http/routes/ViewerRoutes.ts:49 — app.use(express.static(path.join(packageRoot, 'ui'))). Serves built viewer assets only; no tool-artifact endpoint.src/shared/worker-utils.ts:64-73 — getWorkerPort() reads CLAUDE_MEM_WORKER_PORT from settings.json; default 37700 + (uid % 100). User's local port happens to be 37777.src/ui/viewer/index.tsx → App.tsx.src/ui/viewer/components/Feed.tsx — merges observations, summaries, prompts, sorts by created_at_epoch, renders <ObservationCard> / <SummaryCard> / <PromptCard>.src/ui/viewer/constants/api.ts — central list of endpoint paths./stream (plus paginated GET via usePagination).scripts/build-viewer.js (esbuild) bundles to plugin/ui/viewer-bundle.js; viewer-template.html → plugin/ui/viewer.html. Invoked from scripts/build-hooks.js.ContextSettingsModal.tsx (backdrop + centered panel + close). No existing lightbox lib — build a small overlay component.express.static(dir) — already imported.res.sendFile(absPath, { headers, dotfiles: 'deny' }) — Express built-in, streams the file.fs.promises.stat, fs.promises.open (for magic-byte sniff), path.resolve, path.extname.useState / useEffect / portals (already used elsewhere).tool_input JSON from the live pending_messages queue inside an HTTP handler. That table is a transient processing buffer.* for the image endpoint — keep it scoped like the rest of the worker (same-origin for the viewer).tool_use table read path — observations are the surface area for the feed.Goal: When an observation is created, record any image file paths that were touched by the underlying tool calls so the viewer can render them.
What to implement:
New nullable column on observations:
image_refs TEXT (JSON-stringified array of absolute paths). Mirrors the pattern of files_read / files_modified already present at src/services/sqlite/schema.sql:57-92.src/services/sqlite/schema.sql; copy the additive pattern used for any prior column add — ALTER TABLE observations ADD COLUMN image_refs TEXT).Extend ObservationInput in src/services/sqlite/observations/store.ts with image_refs?: string[] and include it in the INSERT statement (lines 35-41). Stringify on the way in.
Add image_refs: string[] | null to ObservationRow in src/services/sqlite/types.ts so it flows out of every read API.
Populate image_refs at observation generation time. Two sources, applied in order:
pending_messages.tool_input / tool_response, scan for absolute file paths whose extension is in IMAGE_EXTENSIONS = ['.png','.jpg','.jpeg','.gif','.webp']. Find the file that builds ObservationInput from pending messages (search for the call site of storeObservation — likely src/services/observations/ or src/services/queue/) and inject the extraction there.files_read ∪ files_modified for image extensions. This catches images that survived only in the summary.Add a tiny pure helper extractImagePaths(toolInput: unknown, toolResponse: unknown): string[] in src/utils/image-refs.ts. Unit-testable, no I/O, no DB access. Handles:
Read tool: tool_input.file_path.Write / Edit: tool_input.file_path.image_path, screenshot_path, output_path keys.file://...png or absolute paths.path.isAbsolute) with image extensions.Documentation references:
ALTER TABLE observations ADD COLUMN ... in src/services/sqlite/schema.sql.files_read / files_modified lifecycle: written by the same function that builds ObservationInput, parsed by the viewer as JSON.parse(row.files_read ?? '[]').Verification checklist:
sqlite3 ~/.claude-mem/claude-mem.db ".schema observations" shows image_refs TEXT.extractImagePaths covering each tool shape (Read, Write, screenshot, nested arrays, non-image extensions excluded, relative paths excluded).image_refs is populated on the resulting row via sqlite3 query.image_refs returns as null and is tolerated by readers.Goal: Add GET /api/images?path=<absolute-encoded-path> that streams an image file from disk if-and-only-if it passes path-safety checks.
What to implement:
New route module: src/services/worker/http/routes/ImageRoutes.ts. Copy the handler-wrap pattern from DataRoutes.ts (this.wrapHandler((req, res) => { ... })).
Register the route in the same place existing DataRoutes / SearchRoutes are registered (find the setupCoreRoutes/setupRoutes call chain off Server.ts:98-105). Register before express.static in ViewerRoutes.ts:49 so it takes precedence on /api/*.
Handler contract:
req.query.path (URL-encoded absolute path).400) if missing, not a string, or path.isAbsolute(decoded) === false.path.resolve(decoded) — if the resolved string differs from the decoded input, reject (catches .. traversal).415) if path.extname(resolved).toLowerCase() not in ['.png','.jpg','.jpeg','.gif','.webp'].process.env.HOME, the CLAUDE_MEM_DATA_DIR, the OS temp dir, or any absolute path currently present in any observation's image_refs column. The DB-membership check is the strongest guard — only paths the system has already chosen to surface can be fetched.
SELECT 1 FROM observations WHERE image_refs LIKE '%' || ? || '%' LIMIT 1 against the resolved path (parameterized; the LIKE is safe because the path is already absolute and we further JSON.parse and .includes() to confirm exact match).415) on mismatch.Content-Type from extension. Set Cache-Control: private, max-age=60. Stream with res.sendFile(resolved, { dotfiles: 'deny' }).404 with no body (don't leak existence).Add IMAGES: '/api/images' to src/ui/viewer/constants/api.ts.
Documentation references:
DataRoutes.ts for the wrapHandler + req.query parsing convention.ViewerRoutes.ts:49 for static-mount ordering reference.Verification checklist:
curl 'http://127.0.0.1:<port>/api/images?path=<encoded>' returns 200 + correct Content-Type for a known image referenced in an observation.curl '.../api/images?path=../../etc/passwd' → 400.curl '.../api/images?path=/etc/passwd' → 404 (absolute, but not in DB and wrong magic bytes)..png → .txt after the path lands in DB still rejects on magic bytes.Anti-pattern guards:
path as a request body — query string only, keeps it GET-cacheable.fs.readFile the whole image into memory — res.sendFile streams.Goal: Get image_refs from the API into the React Observation shape and parsed into string[].
What to implement:
Observation TypeScript type (likely src/ui/viewer/types.ts — confirm via grep interface Observation). Add image_refs: string[] (parsed) plus update the raw ObservationRowFromApi (or equivalent) with image_refs: string | null for the JSON-string form.JSON.parse(row.files_read or similar). Add a sibling image_refs: row.image_refs ? JSON.parse(row.image_refs) : [] line./stream payload reflects the new column (same row shape — no separate change needed if it reuses the same serializer).Verification checklist:
console.log shows image_refs: [...] on observation objects that have images.tsc --noEmit passes (no type drift).ObservationCardGoal: Show a horizontal strip of small thumbnails inside each observation card; clicking opens a lightbox overlay.
What to implement:
New component src/ui/viewer/components/ImageStrip.tsx:
paths: string[], onOpen: (path: string) => void.paths.length === 0.cursor: pointer.title="missing image") — handles deleted files gracefully.New component src/ui/viewer/components/Lightbox.tsx:
ContextSettingsModal.tsx.path: string | null, onClose: () => void.path === null.max-width: 90vw; max-height: 90vh.Escape key, and on close button.Wire into ObservationCard.tsx:
const [lightboxPath, setLightboxPath] = useState<string | null>(null);.<ImageStrip paths={observation.image_refs} onOpen={setLightboxPath} /> after the existing card body content.<Lightbox path={lightboxPath} onClose={() => setLightboxPath(null)} />.Add minimal CSS in the viewer's existing stylesheet (find it via grep .observation-card):
.image-strip { display: flex; gap: 8px; margin-top: 8px; flex-wrap: wrap; }
.image-strip img { height: 80px; width: auto; border-radius: 6px; object-fit: cover; cursor: zoom-in; }
.lightbox-backdrop { position: fixed; inset: 0; background: rgba(0,0,0,.85); display: flex; align-items: center; justify-content: center; z-index: 1000; }
.lightbox-backdrop img { max-width: 90vw; max-height: 90vh; }
Rebuild: npm run build-and-sync regenerates plugin/ui/viewer-bundle.js and plugin/ui/viewer.html.
Documentation references:
src/ui/viewer/components/ContextSettingsModal.tsx for backdrop + close pattern.src/ui/viewer/components/Feed.tsx for how ObservationCard is invoked.Verification checklist:
http://127.0.0.1:<port>/. Observations with image_refs show a thumbnail strip.Anti-pattern guards:
loading="lazy".gstack screenshot).GET /api/observations?limit=1 — confirm image_refs is a JSON array containing the screenshot's absolute path.curl cases listed in Phase 2.viewer in test directories — Vitest or Playwright).image_refs) still render and pass type checks.npm run build-and-sync succeeds without warnings./api/images)./api/images endpoint, no extra core work.image_refs on historical observations.