Back to Nanoclaw

NanoClaw Database Architecture — Overview

docs/db.md

2.0.638.4 KB
Original Source

NanoClaw Database Architecture — Overview

Orientation for the data model: the three databases, how they fit together, and the invariants that hold across them. For table-level schemas, follow the links below.

  • db-central.md — every table in data/v2.db (identity, wiring, approvals, Chat SDK state) plus the migration system.
  • db-session.md — the per-session inbound.db + outbound.db pair, seq parity, and session folder layout.

Related: architecture.md for the high-level design; api-details.md for inbound/outbound message content shapes; isolation-model.md for channel-to-agent wiring modes.


1. The three databases

NanoClaw uses three kinds of SQLite database, all on the host filesystem:

DBLocationWriterReadersPurpose
Centraldata/v2.dbhosthostIdentity, permissions, routing, wiring — the admin plane
Session inbounddata/v2-sessions/<agent_group_id>/<session_id>/inbound.dbhosthost (sync), container (read-only)Host → container messages + routing projections
Session outbounddata/v2-sessions/<agent_group_id>/<session_id>/outbound.dbcontainerhost (poll), containerContainer → host messages + processing status

Single-writer rule. Every SQLite file has exactly one writer. Host writes the central DB and every inbound.db; container writes only its own outbound.db. This eliminates write contention across the Docker/Apple Container mount boundary — SQLite locking across that boundary is unreliable.

Everything is a message. There is no IPC, stdin piping, or file watcher between host and container. The two session DBs are the sole IO surface. Heartbeat is a file touch(2) on .heartbeat, not a DB write.

Journal mode. Session DBs use journal_mode = DELETE (not WAL). Cross-mount WAL visibility is a bug farm; DELETE mode + open-write-close forces the page cache to flush so the other side sees changes.


2. Database map

data/
  v2.db                                   ← CENTRAL (host ↔ host)
  v2-sessions/
    <agent_group_id>/
      .claude-shared/                     ← shared Claude state for the agent group
      agent-runner-src/                   ← per-group agent-runner overlay
      <session_id>/
        inbound.db                        ← host writes, container reads
        outbound.db                       ← container writes, host reads
        .heartbeat                        ← mtime touched by container
        inbox/<message_id>/               ← decoded user attachments
        outbox/<message_id>/              ← attachments the agent produced

Path helpers: sessionDir(), inboundDbPath(), outboundDbPath(), heartbeatPath() — all in src/session-manager.ts.


3. Central vs. session: what goes where

Kind of dataWhereWhy
Identities, roles, membershipscentralStable, cross-session, rarely written
Channel wiring, routing rulescentralAdmin plane
Destination ACLcentral (+ projection per session)Source of truth centrally; fast local lookup per session
Session registry (ids, status)centralHost orchestrates lifecycle
Approvals & pending questionscentralSurvive container restarts, admin-visible
Dropped-message auditcentralGlobal ops view
Inbound messages, retry statesession inbound.dbPer-session workload; host is sole writer
Outbound messages, agent statesession outbound.dbContainer is sole writer; host polls
Delivery outcomesession inbound.db (delivered)Host writes on success; container reads for edit targeting
Processing statussession outbound.db (processing_ack)Container can't write to inbound.db

Heuristic: if the value is a message, routing projection, or runtime ack, it goes per-session. Everything else is central.


4. Cross-mount visibility

Session DBs are bind-mounted into the container. A few rules you need to know before touching the DB code:

  • journal_mode = DELETE, not WAL. WAL files don't reliably cross the mount and the container can read stale pages. DELETE mode forces each writer to flush the main file.
  • Open-write-close on the host. Host-side writes to inbound.db open a connection, write, and close it. Keeping a handle open makes cached pages invisible to the container.
  • Container reads read-only. The container opens inbound.db with readonly: true and never writes — all container→host state goes through outbound.db (see processing_ack in db-session.md).
  • Heartbeat is a file touch. .heartbeat mtime is the liveness signal, not a DB column. A DB write per heartbeat would serialize behind other writers.

These rules are enforced by convention in src/session-manager.ts and container/agent-runner/src/db/. If you change how the DBs are opened, re-read that code first.


5. Design patterns at a glance

  1. Two-DB session split. inbound.db and outbound.db each have one writer, one direction of flow — no cross-mount lock contention.
  2. Seq parity. Even = host, odd = container. Disjoint namespace across both tables lets the agent reference any message by seq alone. Details in db-session.md §3.
  3. Projection pattern. agent_destinations and session_routing are projected from the central DB into each session's inbound.db on container wake — the container gets a fast, local read path without querying across the mount.
  4. Ack via reverse channel. Container never writes to inbound.db. Status sync happens through processing_ack in outbound.db, which the host polls and reconciles.
  5. Heartbeat out of band. File touch on .heartbeat, not a DB write, so liveness doesn't serialize behind other writers.
  6. Lazy session-DB migrations. Central DB uses numbered migrations; per-session DBs use IF NOT EXISTS + ad-hoc ALTER TABLE helpers for older session folders.
  7. ACL = row existence. agent_destinations membership is itself the permission — no separate permissions table.

6. Readers & writers — at a glance

TableDBWriter(s)Reader(s)
agent_groupscentralsrc/db/agent-groups.tssession resolver, delivery, router
messaging_groupscentralsrc/db/messaging-groups.ts, channel setuprouter, delivery, session resolver
messaging_group_agentscentralsrc/db/messaging-groups.tsrouter
userscentralsrc/db/users.ts, auth flowspermission checks
user_rolescentralsrc/db/user-roles.tssrc/access.ts, all permission gates
agent_group_memberscentralsrc/db/agent-group-members.tsmembership checks
user_dmscentralsrc/user-dm.ts (ensureUserDm)approval + pairing delivery
sessionscentralsrc/db/sessions.ts, src/session-manager.tsdelivery, sweep, container runner
pending_questionscentralsrc/db/sessions.ts (via ask_user_question)container response matcher
agent_destinationscentralsrc/db/agent-destinations.ts, migration 004 backfillwriteDestinations(), delivery ACL
pending_approvalscentralsrc/db/sessions.ts, src/onecli-approvals.tsadmin-card delivery, sweep
unregistered_senderscentralsrc/db/dropped-messages.tsops tooling
chat_sdk_*centralsrc/state-sqlite.tsChat SDK bridge
schema_versioncentralsrc/db/migrations/index.tsmigration runner
messages_ininboundsrc/db/session-db.tscontainer/agent-runner/src/db/messages-in.ts
deliveredinboundsrc/db/session-db.ts (markDelivered)container edit/reaction targeting
destinationsinboundwriteDestinations() in src/session-manager.tscontainer routing / ACL
session_routinginboundwriteSessionRouting() in src/session-manager.tscontainer send_message defaults
messages_outoutboundcontainer/agent-runner/src/db/messages-out.tssrc/delivery.ts poll loop
processing_ackoutboundcontainer/agent-runner/src/db/messages-in.tssrc/host-sweep.ts (syncProcessingAcks)
session_stateoutboundcontainer/agent-runner/src/db/session-state.tscontainer on startup