PROPOSAL-pluggable-storage-backends.md
Status: Historical — partially implemented, then superseded on 2026-07-16
Date: 2026-07-02
Informed by: the bts-rs Rust spike (a from-scratch Rust reimplementation
of the gc-contract surface that validated a backend-agnostic storage seam at 299/299 byte-parity
across two radically different backends), a five-dimension recon of this repo's storage
architecture, and a two-reviewer adversarial verification pass. All file:line citations verified
against worktree-beads-new.
Superseded scope: This proposal is retained as the design record for the storage interface, shared issue core, SQLite adapter, and differential conformance work that grew from it. Those pieces remain. The direct PostgreSQL and MySQL adapters were rolled back before entering a tagged release because supporting additional general-purpose server databases adds substantial dialect, credential, schema-lifecycle, migration, CI, and operational complexity. Our goal is to keep Beads as simple as possible and consume as few resources as possible. The supported paths are now embedded Dolt, Dolt server, and SQLite; Dolt server's use of the MySQL wire protocol is unchanged. See Storage Backends for the current design.
Everything below this notice is retained as historical design material, not as current backend support or migration guidance.
bd init --backend=dolt|sqlite|postgres — the user picks the engine at init. The workspace
locator records the choice; opening consults the locator, never the environment.bd history, bd dolt push/pull,
bd vc, bd branch, bd diff, bd federation, bd backup, bd sql are live only when the
opened store advertises the matching capability; otherwise they are deterministic stubs that
explain what backend provides them (Decision D2, resolved: stubs, not hidden — see §6).
Pick Dolt → you get history and remotes. Pick SQLite → you get a fast, zero-dep, pure-Go
single-writer store and those commands stub out.bd export → re-init on the new backend → bd import,
with documented fidelity limits (§5, Phase 4 gate). In-place cross-backend data migration is
a separate initiative.Evidence: bts-rs/docs/ (SEAM_VALIDATION_FINDINGS.md, CONFORMANCE_GAPS.md,
RED_TEAM_FINDINGS.md, HISTORY_ON_YUGA_DESIGN.md, PERF_PG_VS_YB.md, REMOTE_DEPLOYMENT.md).
Six small core traits (33 data methods) + optional capability traits reached via accessors defaulting to "absent". Two backends (Postgres, file-backed memory) hit byte-parity with real bd on 299/299 scenarios with zero command-handler changes — handlers never name a backend.
Go gets the spike's hardest problem for free. Rust's non-object-safe async traits forced a
399-line delegation enum; in Go, storage.Storage is already an interface and Open() returns
the interface value. Budget the port on semantics and conformance, not plumbing.
Seam-size caveat (from review): the spike's 33-method core is NOT evidence that beads' much larger core (§4.1: ~107+24 methods after the fold) is cheap to implement per-backend. The spike's "~1–10k LOC per adapter" anchor was measured against 33 methods. Phase 0 therefore includes a core-shrink review: which core methods are truly primitive vs derivable-above-the-seam vs capability material. Sizing in §8 assumes the core does NOT shrink (worst case).
Open(locator, identity) / Init(backend, ...) are seam
methods; otherwise every entry point names concrete backends.{backend, workspace_id}); backend
is read FROM the locator at open; --backend/env applies only at init.The spike's red team hit this bug class three times; in beads it lives on the core surface:
issueops/create.go:601, labels.go:151, helpers.go:147; read via core
GetAllEventsSince, storage.go:84, and bd audit). Any new backend must do the same —
events emitted outside the mutation tx strand or duplicate audit rows on crash.is_blocked is the most conformance-dangerous core semantic. It is
maintained across 10+ issueops write paths (dependencies/close/reopen/delete/promote/…) and
read by ready queries (sqlbuild/ready.go:96). Delete/purge must recompute surviving
neighbours. A fresh backend reimplements this propagation on every mutation; the harness
must carry required transition fixtures (close-blocker→ready, purge-orphan re-seed).<TS>/<UUID>/actor); per-step exit + stderr + JSON-aware stdout diff. A
static canonicalized corpus "pins a formatter, not a system".ordered flag valid only where
ordering is implementation-independent.--graph).ready --claim drain applies only
to backends advertising multi-writer.ready's 48h recency rule), and anything outside the in_scope
predicate. Each gate below pins its in_scope coverage explicitly so "proven identical" is
never claimed beyond what ran.cp over an exec-mapped binary fails
silently and scores a stale binary).bd history's minimal substrate is a per-issue, ordered, never-pruned field-delta changelog
written in the same tx as each mutation + HistoryViewer{History, AsOf, Diff} as a backward
fold. Dolt provides all three natively (dolt_history_*, AS OF, dolt_diff). So "pick Dolt →
get history" can later relax to "any backend with the Changelog capability" without
re-architecting.
Postgres bounded ops flat to 100k (list ~2ms). Remote cost = round-trips × RTT plus ~6 RTT of
connection setup per command in the shell-out model (~300ms pure overhead at 50ms RTT) —
mitigations: co-location, local PgBouncer ≥1.21 transaction mode, daemon endgame. Collapse write
paths to 1–2 statements; FOR UPDATE SKIP LOCKED IS the claim guarantee. YugabyteDB ≈ 2–3×
latency / 10× write cost — multi-region only. An embedded pure-Go store is the cheapest second
backend and the one that validates the seam.
storage.DoltStorage (internal/storage/storage.go:200-213) composes core Storage (62
methods) + 11 sub-interfaces (144 total; + Transaction 24). The Dolt-shaped surface is
already isolated in 5 named sub-interfaces — VersionControl (16), HistoryViewer (3),
RemoteStore (12), SyncStore (2), FederationStore (4) = 37 methods, ~87 genuine call sites in
non-test cmd/bd. The other 6 sub-interfaces are backend-neutral (BulkIssueStore,
DependencyQueryStore, AnnotationStore, ConfigMetadataStore, CompactionStore,
AdvancedQueryStore — ~45 methods, ~155 cmd/bd call sites serving CORE commands like
ready --claim, dep, comment) and fold INTO the core seam (§4.1).storage.go:217-287), UnwrapStore + type-assert at 22 cmd/bd sites, graceful errors
(sql.go:54-57).beads.go:20 aliases
Storage = beads.Storage (= storage.Storage); RemoteStore/SyncStore are exported as
separately assertable interfaces. The retype does not break the public type surface — but see
§4.4 for the public Open* constructors and the extension-compat note.tests/regression (built for the SQLite→Dolt migration;
found 70+ bugs) contributes workspace isolation, binary management, container TestMain, and a
normalizer that already encodes cross-backend representational divergences
(regression_test.go:587 local-vs-UTC midnight; :647 metadata {} vs absent). Its
execution core does NOT transfer (§5 Phase 1).storage.Storage); the
tracker engine already demonstrates capability-with-fallback (tracker/engine.go:695-717).issueops/sqlbuild/domain/db.
Dolt-specific SQL inside issueops sits in 7 files: as_of.go, commit_pending.go,
diff.go, federation.go, blocked_merge.go, history.go (dolt_history_issues), and
blocked_consistency.go (dolt_status guard — backend-internal, moves with the backend).
The first six map 1:1 onto gateable capabilities.DoltStorage path covers
100% of commands. The uow/domain layer covers 13 commands via *_proxied_server.go
duals and is an ACTIVE workstream (one command per PR since 2026-05-10, latest merges
2026-06-24; the init gate at init.go:135-140 is deliberate staging per PR #4488, and the
CI lane is dark only until the gate lifts). It is Dolt-flavored at its Tx layer
(uow/doltserver_tx.go:28 = CALL DOLT_COMMIT), duplicates issueops (kept honest by
Seam-A parity tests), and is already feature-diverging from the embedded path (--offset
is proxied-only). Proxied-server mode exists ONLY on the uow path — newDoltStore
refuses it (store_factory.go:48-53). D1 must resolve the seam relationship AND proxied's
fate, with the uow workstream owner participating (§6).transactHonoringAutoCommit (dolt_autocommit.go:28-47)
blanks the commit message to suppress mid-tx commits depending on mode; a non-blank
message + success sets commandDidExplicitDoltCommit which suppresses PostRun auto-commit
(main.go:1223-1227); the tips system defers metadata writes and PostRun issues a SECOND
commit for them (tips.go:154-183, main.go:1233-1257); auto-export/backup freshness reads
GetCurrentCommit AFTER the commit; auto-push gates and error semantics differ from
auto-commit; the autocommit-mode default itself is chosen by the CLI from usesSQLServer()
(main.go:1099-1106). commitPendingIfEmbedded/transactHonoringAutoCommit have 21 call
sites across 18 files. This is a protocol, not a hook — it gets a specified contract (§4.2)
and its own phase (2b).store storage.DoltStorage
(main.go:45; 159 non-test refs / 61 files, 15 more test files). Paths: PreRun choke
point; lazy direct_mode.go; read-only; ~13 files opening routed/extra stores; and the
public API — root beads.go:63-73 Open/OpenFromConfig hard-code dolt.New*
(+ beads_nocgo.go OpenBestAvailable), used by external Go extensions. The registry must
own all five. The HookFiringStore decorator EMBEDS DoltStorage
(hook_decorator.go:28-29) and is applied only on the main path today — the registry must
own decorator application or lazy/read-only/routed opens keep silently skipping hooks.*.up.sql, MySQL
dialect; migrations 0019/0028/0040/0041 execute dolt_ignore/CALL DOLT_COMMIT inline. New
backends need per-backend migration sources (the schema runner's two-stream design is the
extension point).RunInTransaction(ctx, commitMsg, fn) — where the message
is load-bearing suppression semantics per H2, NOT advisory; GetCurrentCommit as freshness
token (export_auto.go:85, backup_auto.go:127); ApplyCompaction(commitHash); bd edit's
raw *sql.DB keepalive (edit.go:150); "dolt-ignored" LocalMetadata contract.noDbCommands (27), readOnlyCommands
(14), dolt-subcommand patch lists (main.go:770-808,122-137).dolt_remotes/dolt_status.GetCurrentCommit never advances, so
auto-export/backup freshness checks silently stop firing in server mode.internal/storage/dolt.DoltStore) commits to HEAD per write method,
unconditionally — independent of --dolt-auto-commit and of maybeAutoCommit
(dolt/issues.go:49 CreateIssue → CALL DOLT_COMMIT; UpdateIssue/CloseIssue/DeleteIssue
likewise; RunInTransaction → versioncontrolops.StageAndCommit). So in non-proxied server
mode HEAD advances on every persistent write, DOLT_HASHOF('HEAD') changes, and the
auto-backup freshness check DOES fire. Separately, auto-export never even reaches the
freshness check in server mode — it early-returns (export_auto.go:38-41, if serverMode { return nil }). The maybeAutoCommit machinery governs the EMBEDDED working-set-persistence
flow, not the server store's per-method commits; the H8 draft was written against an
inaccurate model of the server store. The rest of H8 stands: auto-commit ON embedded / OFF
server; single-writer flock embedded-only (store_factory.go:66-85); server DoltStore has
real streaming iterators (dolt/iter_issues.go:48) while embedded ships NewSliceIter
stubs; bd sql/bd admin work in server mode but error in embedded (sql.go:45,
admin.go:17) — and that embedded error string is gc contract class 3.maybeAutoImportJSONL runs in PreRun for every command
(main.go:1163-1164), takes storage.DoltStorage, auto-imports JSONL into empty databases,
and has a documented data-clobber history (auto_import_upgrade_unit_test.go:74-87). Its
"auto-importing … into empty database" strings are gc error class 4 (the write-loss guard).
A second backend that is empty-on-fresh-open is exactly the trigger condition; if its strings
differ, gc's guard goes blind.BEADS_DOLT_SERVER_MODE=1 is
precedence #1 over metadata.json (configfile.go:232-243); shared-server env/config
overrides pinned dolt_mode with a Notice (main.go:247); --global hard-requires
shared-server (main.go:1071-1076). Parsing is scattered (configfile, doltserver, init,
bootstrap). The spike's "locator wins at open" rule must extend to topology, with an explicit
ownership and inertness rule (§4.3).~271 cobra commands (~111 top-level), all registered unconditionally in package init() —
before any workspace resolution (this kills naive hide-at-registration; see D2). Classified:
CORE ≈ 67 (incl. the gc-contract 16), STORAGE-SPECIFIC = 11 (+3 mixed: bd gc, bd migrate,
bd restore's history fallback), INTEGRATION = 6 families, INFRA ≈ 27. 12 of the gc-16 have
dual (store+uow) implementations; purge, count, version, stats do not — and purge
is precisely where the spike's red team found re-seeding bugs, so it is a required Phase 1
scenario, not a "covered elsewhere" case.
Two things are described here and it matters to keep them distinct: §4.0 the destination (the clean-room object model we would build greenfield, which the migration accretes toward) and §4.5 the bridge (the flat-interface waypoint the mechanical phases pass through). The destination is what new code is written AGAINST from Phase 4 onward; the bridge is scaffolding that de-risks the retype and is thinned afterward (Phase 7).
┌────────────────────────────────────────────────────────────────────────┐
│ cmd/bd — commands, formatting, streams, contract strings │
│ command registry: declared metadata {needsStore, readOnly, │
│ requires: Capability} — replaces the name-keyed string lists (H6) │
└──────────────────────────────┬─────────────────────────────────────────┘
│ consumes ONLY:
┌──────────────────────────────▼─────────────────────────────────────────┐
│ package store — THE SEAM (small, frozen, documented) │
│ Driver / Registry / Locator / Identity (construction, rules 1–3) │
│ Store / Tx + ~7 domain interfaces (core; Tx-first) │
│ Capability interfaces + CapabilitySet (optional surface, rule 4)│
│ Error taxonomy + retry classes (typed, rendering-free) │
│ storetest.RunContract(t, driver) (the compliance suite) │
└──────┬───────────────┬───────────────┬───────────────┬─────────────────┘
│ │ │ │
backend/dolt backend/postgres backend/sqlite backend/fake
(topologies: (multi-writer; (pure-Go; (in-memory/file;
embedded | no history v1) single-writer; unit-test keystone)
server | fills nocgo)
proxied
INTERNAL)
The core, sketched (signatures indicative, not final):
// construction — the spike's leak rules 1–3, verbatim
type Driver interface {
Open(ctx context.Context, loc Locator, id Identity) (Store, error)
Init(ctx context.Context, opts InitOptions) (Locator, error)
}
type Locator struct{ Backend string; Raw json.RawMessage } // self-describing; env only at Init
type Identity struct{ Actor, Email string } // on the seam; injectable for tests
type Store interface {
// Run is the PRIMARY seam op: one retryable unit of work. The seam owns
// phase-aware retry (Serialization -> bounded auto-replay of fn;
// CommitIndeterminate -> surfaced, never auto-retried; fn must be replayable,
// no external I/O while the Tx is open). Semantics lifted from uow/run_in_tx.go.
Run(ctx context.Context, op OpInfo, fn func(Tx) error) error
Begin(ctx context.Context, op OpInfo) (Tx, error) // low-level escape hatch
Capabilities() CapabilitySet // per-INSTANCE (topology-aware)
Limits() Limits // {MaxAtomicTxBytes, MaxValueBytes, MaxTxDuration,
// ConsistentScanBudget} — CLI pre-flights + chunks instead
// of discovering backend ceilings as commit failures
ChangeToken(ctx context.Context) (string, error) // equality-only freshness cursor
Close() error
}
// OpInfo + CommandScope are where the H2 commit protocol collapses to. NOTE
// (red-team blocker): commit messages are OUTCOME-derived in bd (composed after
// close/claim results — close_proxied_server.go:242-255, uow.RunInTxMsg), so the
// description travels on Commit, NOT on Begin. A CommandScope above Tx owns the
// one-version-commit-per-command rule (N txs -> ONE dolt commit), the tips
// deferred-metadata second commit, and PostWritePush ordering.
type OpInfo struct{ Command, Actor string; ReadOnly bool }
// the transaction is the first-class object — one per command/logical op
type Tx interface {
Issues() Issues // CRUD, search, ready, claims; wisps as a TIER param, not a
Deps() Dependencies // parallel ×2 method family
Labels() Labels
Comments() Comments
Events() EventReader // reads; event WRITES happen implicitly inside mutations, same tx
Config() Config
Commit(ctx context.Context, desc string) error // desc outcome-derived;
// Dolt: DOLT_COMMIT(desc); PG: COMMIT; reads: Rollback, never Commit
Rollback(ctx context.Context) error
}
// capabilities: extend core behavior, never gate it (rule 4)
type CapabilitySet struct {
History, VersionControl, RemoteSync, Backup, RawSQL bool
// MultiWriter bool was too coarse (red-team, 2 lenses independently):
Concurrency struct {
ConcurrentReaders, ConcurrentWritersSafe bool
WriterExclusion string // enforced-lock | lease | none
ContentionModel string // row | workspace-CAS
}
}
func History(s Store) (HistoryViewer, bool)
type HistoryViewer interface{ /* History, AsOf, Diff */ }
// + VersionControl, RemoteSync, Backup, RawSQL, Maintenance(GC/Flatten/Compact)
// errors: typed below the seam, rendered above it
var ErrNotFound, ErrClaimConflict error
type ErrUnsupported struct{ Op, Backend string }
type RetryClass int // Transient | Serialization | CommitIndeterminate | Permanent
Every commitment traces to a scar in the current system or the spike:
| Commitment | Scar it heals |
|---|---|
Tx-first seam (Begin → domain ifaces → Commit) | DoltStore's per-method commits = the stale-snapshot data-loss class (#3822) that forced the May pivot; command-scoped short tx is the proven concurrency unit. This is uow.UnitOfWork promoted to the seam. |
| ~7 small domain interfaces, no god interface | the 805-line panic wall (PR #3792); "can't easily mock DoltStorage"; 168 methods before one command works |
OpInfo on Begin | H2: commit protocol currently lives in CLI globals (message-blanking, commandDidExplicitDoltCommit, tips double-commit) |
ChangeToken() on core | backend-neutral freshness: GetCurrentCommit/DOLT_HASHOF is Dolt-specific, so export/backup freshness must key on a neutral opaque token any backend can supply (SQLite/Postgres have no Dolt commit hash). (NB: the earlier "H8 live bug" rationale is withdrawn — see H8; ChangeToken stands on backend-neutrality alone.) |
per-instance CapabilitySet, extend-never-gate | bd sql works on Dolt-server, stubs on Dolt-embedded — capability varies by topology; the spike's only runtime divergence was a capability gating core behavior |
topologies INSIDE backend/dolt | embedded/server/proxied are three connections to ONE engine; CLI knowledge of them produced 38 scattered mode checks |
| events written inside mutations, same tx | the spike's red team hit emit-outside-tx three times; the use-cases already do it right — make it structural + contract-tested |
backend/fake + storetest.RunContract | nothing tests against the interface today; every suite needs a Dolt container |
| wisps as a tier param | today every use-case method is duplicated ×2 (41 where ~25 would do); the --offset union-paging divergence is a symptom |
| slices, not streaming iterators, in v1 | all 10 Iter* methods are slice stubs in the embedded backend anyway; streaming returns later as a measured need |
One-sentence summary: the clean-room seam is Dustin's UnitOfWork widened to the full
domain surface, wearing bts-rs's construction/capability/error discipline.
Core = today's Storage (62) + the six backend-neutral sub-interfaces (~45) = ~107 methods,
Transaction (24), all mandatory for every backend. The ~155 cmd/bd call sites of the six
folded interfaces need zero conversion. The ~87 call sites of the 5 Dolt-shaped interfaces
convert to capability asserts. DoltStorage is deleted; the Dolt backends keep a var _
compile assertion for core + all capabilities. Phase 0's core-shrink review may pull some of
the ~107 above the seam (derivable) — treat any shrink as upside, not plan.A specified seam protocol, designed in Phase 2b BEFORE code moves:
WriteSummary{didWrite, explicitCommitOccurred, readOnlyCommand, messageParts, deferredMeta}
carried per command (absorbs today's commandDidWrite/commandDidExplicitDoltCommit
globals and the tips deferred-write protocol).PostWriteCommit(summary)
(Dolt: auto-commit incl. the second tips commit; others: no-op) then — after the CLI's
auto-export/auto-backup, whose freshness check consumes a neutral ChangeToken() (core,
with the capability-shortcut-plus-fallback shape; motivated by backend-neutrality, since
DOLT_HASHOF is Dolt-specific — NOT by the withdrawn H8 "live bug", see H8) —
PostWritePush(summary) (Dolt: auto-push with its swallowed-error semantics; others: no-op).transactHonoringAutoCommit's blank-message trick) becomes an
explicit seam behavior: the backend decides mid-tx checkpointing from the summary + its own
mode; the message parameter on RunInTransaction stays but is re-specified as "operation
description; versioned backends MAY use it as a commit message" (D3).usesSQLServer()) moves into the Dolt
backend's topology config — the CLI no longer knows modes exist..beads/metadata.json: {backend: "dolt"|"sqlite"|"postgres", ...}. dolt_mode
becomes Dolt-internal topology config.backend field is UNTRUSTED legacy —
workspaces in the wild may carry backend:"sqlite" from the pre-removal era
(removed in 87493ce91), and GetBackend()'s hard-return of "dolt" (configfile.go:207-210)
is a deliberate shim over exactly that. Rule: introduce a NEW locator key (e.g.
storage_backend, plus a metadata schema version); absent/legacy/unknown values + presence
of a Dolt data dir resolve to dolt, never to a new backend. A Phase 1 harness fixture
with a legacy backend:"sqlite" metadata.json pins this. This closes the trap where a legacy
value resolves to the new SQLite backend, opens an empty store, and H9 auto-import silently
re-seeds/forks the workspace.BEADS_DOLT_* (19 vars) are owned by the Dolt backend and are inert
for non-Dolt locators; env parsing consolidates behind the registry; the Phase 1 harness
pins the precedence table (H10). --global/shared-server: the CLI keeps a seam-level
"workspace redirection" concept whose only current provider is Dolt shared-server topology —
on SQLite/Postgres, --global errors with a capability stub (D2 pattern).bd sql embedded string), class 4 (H9 auto-import strings — now backend-neutral pinned
contract), and the claim-conflict text.beads.Open* route through the registry (H3 fifth path). Compat note for
extension authors: on non-Dolt workspaces, type-asserting RemoteStore etc. on the
returned Storage starts returning false — document in release notes; the assertion pattern
itself (beads.go:27-31) is unchanged.beads_nocgo.go:27).The mechanical phases (2a/2b/3) retype the CLI onto the FLAT core interface (§4.1) because
that keeps a 159-reference migration mechanical. That flat interface is scaffolding, not
the destination: from Phase 4 onward, the uowStore adapter's inner surface is built as
the PUBLIC clean-room seam (§4.0 Tx + domain interfaces) rather than as a private adapter
detail — so the destination accretes underneath the migration instead of requiring a second
one. Phase 7 then thins the flat layer opportunistically (or keeps it as sugar). Existing
components map as: uow.UnitOfWork/use-cases/domain/db → the seam's embryo;
EmbeddedDoltStore/DoltStore/issueops → backend/dolt interior; the 13 duals,
DoltStorage, the string lists, and the Seam-A parity tests → deleted.
The seam must not assume SQL — two of the three named future backends are key-value and one is object storage. Design constraints this imposes, to be validated by adversarial review:
bd import-scale work cannot be one Tx. The seam therefore must NOT promise unbounded
transaction size — bulk operations need an explicit chunked/bulk path on the seam, and the
contract suite must pin what atomicity bulk ops actually guarantee. Ordered scans and
secondary indexes are backend-built (the bts-rs readiness-projection pattern applies).Commit returns a Serialization retry class on CAS
loss) — which the Tx+RetryClass model can express, but claim latency and contention
behavior must be honestly capability-flagged (likely MultiWriter: false or
claims-unsupported). S3 may be BETTER fitted as the target of RemoteSync/Backup/cold
history (the bts-rs BlockStore/RefStore substrate is S3-native with CAS on refs) than as
a primary Store; the seam should permit both without contortion. ChangeToken maps
naturally (manifest ETag).These contract additions override/extend §4.0 where they conflict; fold into the sketch on the next full edit. Convergent = demanded independently by 2+ lenses.
Convergent (adopted):
Store.Run(closure) is the primary op; commands migrate ONTO it in Phase 7 (not off
it). Retry ownership lives on the seam; typed errors gain backend-supplied RetryAdvice
(backoff hint + budget); storetest includes an error-injecting fake and requires that a
conforming driver's advice lets 24 contending claimers converge (no livelock).create --graph atomicity resolved against Limits().PartialCommit error class or
AtomicTiers flag); a tier×capability locality matrix (History/RemoteSync/Backup/export
MUST exclude local-tier data) with fixtures: purge-spares-no_history,
backup-excludes-wisps. All of this feeds D5.{"k": null} counts as present;
clock ownership moves client-side (a Clock provider joins Identity on the construction
seam) so relative-time predicates and the 48h-hybrid ordering are deterministic +
contract-testable.Single-lens (adopted):
7. Mutation-observation contract (repo lens, blocker): Tx.Commit yields a
MutationSummary (feedable from the same-tx event rows) consumed by a registry-owned
hook dispatcher; the decorator wraps Tx, not just Store; hook-artifact fixtures join the
Phase 1 harness; Phase 7 conversions gate on hook parity.
8. Interleaving rule: capability methods may not run between Begin and Commit/Rollback
on the same Store (typed error, storetest-pinned); per-topology connection budgets are
documented backend-author contract (embedded's eager second dolt_ignore connection binds
lazily or every command pays issue-#4303's dominant cost).
9. Sync/archive TARGETS are a first-class seam concept distinct from Store (S3 lens):
BlockStore{Put,Get}/RefStore{Get,CAS} target interfaces + per-scheme drivers
(s3/gs/file); CapabilitySet.RemoteSync rescoped to backend-NATIVE remotes (Dolt);
Phase 6 cold history and portable backup-to-URL route through targets — S3's best fit,
without pretending it is (or isn't) a primary Store. Backends whose commit protocol can
strand garbage (CAS-race orphans) MUST implement + declare Maintenance.GC; doctor
surfaces it.
10. Error taxonomy additions: ErrBusy (Transient; the embedded flock string is its
pinned rendering); claim errors carry structured holder/status (the gc class-2 string
is rendered ABOVE the seam from data, incl. idempotent self-reclaim success);
Open/Begin blocking-vs-fail-fast behavior specified per concurrency class; backends MAY
resolve CommitIndeterminate internally (verify-after-timeout) and return a definite
class.
11. ChangeToken contract: opaque, equality-only, data-changed ⇒ token-changed
(false positives permitted); Dolt = hash over non-ignored core tables (NOT HEAD);
per-tier advance semantics pinned; S3 = seq embedded in the manifest, not raw ETag;
FDB = versionstamped last-mutation key.
12. Testing-story honesty (repo lens): backend/fake + storetest get their own phase,
gate, and sizing row; evaluate a go-mysql-server-backed fake running the REAL SQL
(reuse, not a fifth semantics copy); Seam-A parity-suite deletion re-gated on
single-SQL-stack convergence inside backend/dolt (embedded TxProvider over the embedded
engine → delete issueops), not on dual-file collapse; §3.1's "integrations are free" is
WRONG — the tracker keys its Dolt fast path on DB() presence and must key on the
History capability (same audit for every RawDBAccessor consumer: SQL-speaking ≠
has-Dolt-system-tables).
13. Streaming: unbounded consumers (audit, sync, export) keep a cursor/since+limit
path (or a StreamingReads capability) before Phase 7 converts them — "slices only" is
bounded-by-workspace materialization, stated as such.
§4.6 corrections from the future-backend lenses: FDB limits apply to reads too (~5s
read-version expiry; 100kB value cap vs bd's unbounded LONGTEXT — chunked content or
declared Limits); the readiness-projection pattern transplants only with rework (chunked
catch-up outside the op Tx, lease-based projector); redb: what the spike validated is "an
embedded single-file backend behind the seam for the gc-16 surface" — the 9-sort-order
index design, ~107-method surface, and cross-process behavior remain unvalidated, and redb
has no C API (a Go binding does not exist; that backend arrives via FFI work or the bts-rs
lineage); S3 claims stay CORE (never "claims-unsupported") — the drain gate keys on
Concurrency.ConcurrentWritersSafe with a disclosed contention model; 412/409/503/timeout
→ RetryClass mapping table to be added.
Each phase lands green with the default backend byte-identical; each has a verification gate that names its in_scope coverage.
bd gc's portable decay phase; tracker
fast path). Classification defines the core command set.commitPendingIfEmbedded sites; the
seven issueops Dolt files; the WriteLifecycle protocol inventory (H2).BackendCapabilities (configfile.go:183-204).A new differential runner reusing tests/regression scaffolding (workspace isolation,
binary management, container TestMain, the battle-tested normalizer) — NOT an extension of its
execution core: today's runner is 177 Go closures compared by end-state snapshot only, t.Fatals
on any non-zero exit, and merges stdout+stderr (regression_test.go:363-374,811-831). New:
per-step recorder (exit code, split streams, JSON-aware diff), replayable scenario format,
multiset-default + ordered flag, in_scope predicate, backend/mode parameter on workspace
setup, property tests for minted IDs, the claim drain (multi-writer only), unique
candidate-binary paths.
store + decorator +
molecules.NewLoader + remotecache.StoreOpener + helpers from DoltStorage to the core
interface (159 refs / 61 files + 15 test files); convert the ~87 Dolt-shaped call sites
to UnwrapStore asserts; delete DoltStorage.beads.Open*) into the registry;
registry owns decorator application (closing the existing lazy/read-only/routed hook gap);
new locator key + legacy rule (§4.3).commitPendingIfEmbedded/transact* call
sites, PersistentPostRunE, the tips deferred-write protocol, ChangeToken for export/backup
freshness (on backend-neutrality grounds — the "fixes the silent server-mode
export-freshness bug" rationale is withdrawn; that bug was refuted on verification, so
ChangeToken is a portability requirement, not a bug fix, and needs no behavior-change release
note), retype H9 auto-import and pin its strings.Capabilities() is instance-level (computed after open, backend × topology — review
fix: bd sql works on Dolt-server but not Dolt-embedded, same backend). Gating happens at
RunE-time via stubs (D2 resolved): registration stays in init() (no bootstrap
restructuring; bd --help and the CI docs-regen stay deterministic); absent capability →
deterministic stub error naming the backend that provides it (federation_nocgo precedent;
bd sql's embedded string preserved verbatim as contract).bd dolt (16 subcmds), bd vc, bd branch, bd history, bd diff,
bd show --as-of, bd federation, bd backup, bd sql, bd flatten, Dolt legs of
bd compact/bd gc, db-proxy-child, --global (§4.3).bd doctor: per-backend check registry (D6); Dolt checks (incl. blocked_consistency's
dolt_status guard) move into the Dolt backend's set.--backend=sqlite deprecation contract (init.go:215-225 hard-errors
today; docs/scripts written against that string need a coordinated, release-noted flip).--help byte-identical; a test-only capability-less backend
exercises every stub; docs regen green.Owner-stated success criterion: beads runs on a Postgres backend and the bd CLI simply doesn't have the history functionality. Postgres — not SQLite — is therefore the seam validator and the critical-path deliverable. (The first draft's SQLite-first ordering followed the spike's cheapest-second-backend guidance; the owner's proof criterion overrides it.)
Two candidate build routes — choosing is part of D1, because they hinge on the uow relationship:
TxProvider (plain BEGIN/COMMIT; the commit-message param is ignored) + a dialect port of
the domain/db repositories (~75 methods of EXISTING, Seam-A-parity-tested SQL; the
MySQL-isms are enumerable — ON DUPLICATE KEY→ON CONFLICT, backticks, no CALL DOLT_*)
uowStore adapter + the five gap units +
per-backend DDL. This REUSES the already-implemented dangerous semantics (same-tx events,
is_blocked propagation) instead of re-deriving them, and leans on the workstream's own
stated invariant (issue #4489): "the local-vs-proxied difference is confined to the uow
transaction layer (the mode-polymorphic db.Runner)."FOR UPDATE SKIP LOCKED, typed retry classes, optional readiness projection). Independent of the uow
roadmap; more code; a THIRD implementation of the business semantics to keep honest —
the hard part is semantics, not SQL (§2.3: same-tx events, is_blocked propagation,
delete/purge neighbour recompute, wisp/tier semantics, D5 local-tables).Proof-wedge milestone (demoable before Phases 2–3 fully land): minimal locator backend
field + one factory arm + the PG stack + a TEMPORARY full-DoltStorage compat shell whose 37
Dolt-shaped methods return typed Unsupported errors (graceful errors, NOT panics — the D1
design-history lesson) + PersistentPostRun neutralized for non-Dolt. This demonstrates "bd on
Postgres, bd history absent" without waiting for the 159-ref retype; the shell is deleted
when Phase 2a lands. Accepted sharp edges for the demo: auto-export/backup freshness and
parts of bd doctor degrade on the wedge; the gc-16 harness subset is the demo's acceptance
test.
Pure-Go (modernc.org/sqlite), documented single-writer (claim-drain gate N/A); fills the
nocgo hole (beads_nocgo.go:27 — non-CGO builds currently have NO local backend). Under
Route A it rides the same production line (a SQLite TxProvider + dialect port); under
Route B, a direct implementation with the WAL lessons (synchronous=NORMAL,
wal_autocheckpoint=0 + explicit checkpoint cadence, pragmas on the read pool, no unclosed
rows). Slice-iterator stubs are legal (embedded Dolt ships them; server streams — both
satisfy the interface).
Changelog capability (same-tx field-delta log) + HistoryViewer-as-fold (§2.5). Dolt keeps its
native implementation. Mirror the spike-documented wire quirks (bare JSON array, PascalCase
keys, full snapshots, client-side --limit, volatile CommitHash normalized).
Migrate commands off the flat conveniences onto Begin/Tx/Commit directly — file by file,
no flag day, each conversion deleting a RunInTransaction wart — until the flat layer is
thin enough to delete or small enough to keep as sugar. No deadline: the system is already
multi-backend and capability-gated before this phase starts. Exit criterion: the store
package's godoc IS the backend-author contract, and storetest.RunContract is the only
thing a new backend must pass before the differential harness.
bd migrate --to-backend=X) — the interim adoption
path is export→re-init→import with Phase-4-documented fidelity limits.storage.DoltStorage implementation over the proxy WAS built first — DoltServerStore,
PR #3792, May 6–7 2026 — and died in four days as an 805-line wall of
panic("unimplemented") (128/132 methods), pivoted to the UOW design with only a two-line
TODO as rationale. It died because the god interface cannot be implemented incrementally
from scratch — a cause the uow workstream's own six weeks of use-case building has since
removed (the adapter delegates to them), and which Phase 2's interface shrink + capability
split removes for every future backend. "We tried a store over the proxy already" is
therefore true but not dispositive.
Recommended reconciliation — compose, don't pick: storage.Storage remains the backend
seam (what the CLI consumes; what SQLite/Postgres implement; where the capability/addon
model lives — use-cases have no answer for command gating). Build the missing uowStore
adapter (core Storage implemented over UnitOfWork use-cases; today no such adapter
exists and store_factory.go:51/96/169 TODOs point the other way — this is the crux to
align with the uow owner). Consequences: proxied-server becomes a topology of the Dolt
backend behind the registry; the 13 *_proxied_server.go duals collapse; the domain
repositories over db.Runner (3 methods) become an ACCELERATOR for future SQL backends
(supply Runner+TxProvider+dialect, reuse the repos) rather than a rival seam; the
Tx.Commit = CALL DOLT_COMMIT (uow/doltserver_tx.go:28) adopts the §4.2 WriteLifecycle
contract one layer down. Full sketch with the method-by-method mapping and gap analysis
(headline: ~60% of core maps 1:1 to existing use-case
methods; the true remaining cost of full proxied coverage is five small gap units, not 54
more command conversions).
The alternative (full use-case trajectory): commands migrate to use-cases everywhere and
backends supply repositories. Coherent, but: ~54 more command conversions, dual
implementations + parity tests for the duration, non-SQL backends become impossible
(Runner assumes SQL), and the capability model is still needed regardless.
Divergence governance either way: the two paths already diverge in features (--offset
is proxied-only because the embedded issues+wisps union cannot page — PR #4488) — assign an
owner for keeping the paths semantically converged until they reunify, and add the
divergences to the Phase 1 allowlist as tagged, expiring entries.
This decision requires the uow workstream owner in the room.init() before workspace resolution; hiding requires pre-cobra locator
resolution and breaks help/docs determinism. Stubs preserve the gc error contract and the
federation_nocgo precedent. (If hiding is ever wanted, it is its own sized work item.)RunInTransaction message param: keep, re-specified per §4.2 (the blank-message
suppression semantics become explicit seam behavior; "advisory" was wrong).| Risk | Mitigation |
|---|---|
Two-seam divergence widens while D1 is undecided (uow conversions merge weekly; --offset-class proxied-only features accumulate) | D1 decided WITH the uow owner in Phase 0; until then, new proxied features carry a convergence note + Phase 1 allowlist entry; the uowStore adapter collapses the duals once agreed |
Legacy backend:"sqlite" metadata resolves to the new backend and auto-import forks data | §4.3 legacy rule (new locator key, dolt-default fallback) + Phase 1 fixture — was a review blocker, now designed-out |
| Silent drift during the Phase 2a retype | Harness EXISTS FIRST; zero-new-divergence gate; commit per file-cluster |
| WriteLifecycle migration changes semantics invisibly | Own phase (2b) + dedicated ordering scenarios; only enumerated bug-fix diffs accepted |
| Allowlist waivers rot and mask real regressions | Pair-scoped, field-scoped, tagged, expiring entries; no carryover to new-backend runs |
| Green harness over-trusted | §2.4 epistemics: hooks/timing/coverage limits stated per gate; coverage artifact required; CORE-67 widening enforced before Phase-4 announce |
| A from-scratch backend (Route B) re-opens the spike's transactional bug class | §2.3 rules are Phase 4/5 implementation requirements with mandatory transition fixtures; Route A sidesteps by reusing the parity-tested use-case semantics |
bd doctor/backup partial coverage confuses users | Capability-aware stubs + doctor prints per-backend coverage |
| Public-API extensions break on non-Dolt workspaces | §4.4 compat note; assertions return false (defined behavior), release-noted |
| Schema drift between per-backend migration streams | Shared logical-schema contract suite each backend's fresh DB must pass |
| Phase | Size | Notes |
|---|---|---|
| 0 | ~1 week | decisions + leak-4 audit + core-shrink review + WriteLifecycle spec |
| 1 | 2–3 weeks | new runner on old scaffolding; gc-16 scope + fixtures (was 1–2; review: per-step core is new code) |
| 2a | 2–3 weeks | fold + retype (159 refs/61 files) + registry (5 paths) + metadata properties |
| 2b | 1–2 weeks | WriteLifecycle contract migration (semantic; own gate) |
| 3 | 1–2 weeks | instance-level capabilities + stubs + doctor registry + deprecation flip |
| 4 | Route A: 4–6 weeks / Route B: 5–8 weeks | Postgres proof: A = PG TxProvider + repo dialect port (~75 methods of existing SQL) + adapter + 5 gap units + DDL; B = ~107-method direct core + is_blocked/events semantics from scratch + schema; both + conformance to CORE-67 |
| 5 | 2–4 weeks (Route A) / 4–7 (Route B) | SQLite riding the same production line (A) or direct (B); nocgo hole |
| 7 | opportunistic | thin the flat bridge onto the §4.0 seam; no deadline |
| 6 | 2–4 weeks | optional; independent — and under Route A, the Changelog capability could later give PG history too (§2.5), making "no history on Postgres" a v1 statement, not a permanent one |
Phases 4/5 are independent of each other; 6 independent of both. Critical path to the owner's PROOF ("bd on Postgres, no history"): 0 (D1 + route choice) → 1 (gc-16 harness subset) → Phase-4 proof wedge (~6–9 weeks). Critical path to the full end state ("users choose a backend", shipped properly): 0 → 1 → 2a → 2b → 3 → 4 (~12–19 weeks sequential); the wedge work is on that path, not thrown away — only the compat shell is deleted when 2a lands.
Recon: 5 parallel read-only explorers (storage interface census; Dolt leak census; command
surface census; existing-abstraction assessment; bts-rs learnings extraction) — ~578k tokens,
311 tool calls. Adversarial pass: 2 reviewers (feasibility-vs-repo, completeness-vs-spike),
verdicts sound-with-fixes; 2 blockers, 8 majors, 5 minors, 13 missing items — all folded into
this revision. Notable review discoveries retained above: the six-neutral-interfaces blocker
(§4.1), the legacy-locator trap (§4.3), the WriteLifecycle protocol structure (§4.2/H2), the
instance-level capability requirement (Phase 3), the H8 live bug (server-mode export
freshness) (corrected after verification: refuted — the server store commits per write and
auto-export is server-mode-disabled; ChangeToken is retained on backend-neutrality grounds),
and the H9 auto-import hazard. Clean-room shred: 5-lens Fable panel
(object-model, repo-reality, FDB, embedded-KV, S3) — all verdicts holds-with-fixes; 2
blockers (outcome-derived commit messages vs OpInfo-at-Begin; hook-observation contract), 17
majors; all folded as §4.7.