docs/sync-and-op-log/operation-log-architecture.md
Maintainer routing: use the Sync Architecture Field Guide for the current whole-system mental model. This long-form document preserves deep rationale, migration policy, and implementation history; volatile mechanics in its historical inventories can lag the focused contracts and executable owners linked from the field guide.
Status: Deep rationale and migration/implementation history; receiver-side cross-version migration is active. Routing reviewed: July 20, 2026
Historical overview warning: the introductory inventory below predates later file-v3, conflict, recovery, and server-retention work. All providers (SuperSync, WebDAV/Nextcloud, Dropbox, OneDrive, and LocalFile) now enter the unified client operation-log pipeline; use the field guide and its focused source map for current behavior.
The operation log records replayable state transitions instead of persisting each NgRx model independently. It is event-sourcing-inspired, but it is a bounded operation log rather than an immutable history from the beginning of time:
DELETE is recorded as an operation,
but neither it nor a rejected loser is permanent audit history.When a user performs an action (like ticking a checkbox):
The reducer deliberately runs before the asynchronous append. If persistence fails, live state can therefore be ahead of the durable log; the client surfaces a reload action and prevents compaction from baking that phantom change into a snapshot. Open tabs do not exchange operation payloads: the web startup guard blocks a second active instance instead.
Replaying every operation since the beginning would be too slow. We use Snapshots to speed this up:
The Operation Log enables two types of synchronization:
A. SuperSync operation transport
Operations, not full files. This saves massive amounts of bandwidth.B. File-provider operation transport
sync-data.json; each op-bearing upload rewrites that monolith.sync-ops.json the hot commit point and rewrites
the snapshot/archive file only for bootstrap, compaction, migration, force-upload,
or gap recovery.Validation occurs at operation ingress and at the hydration/sync checkpoints
described in Part D. Repairable state may produce a full-state REPAIR
operation; unrepaired failures prevent the session from claiming success. A
repair row is retained like other operations, not as permanent audit history.
After 500 durable local appends, the client attempts compaction. The attempt can skip safely when remote reducer/archive work, pending local writes, a persistence divergence, hydration fallback, or an empty/degraded store makes snapshotting unsafe. A successful pass writes a new state-cache boundary and deletes only old, terminal rows covered by that boundary; unsynced and incomplete rows remain.
The Operation Log serves four distinct purposes:
| Purpose | Description | Status |
|---|---|---|
| A. Local Persistence | Fast writes, crash recovery, event sourcing | Complete ✅ |
| B. File-Based Sync | Default v2 monolith or opt-in v3 split files | Complete ✅ |
| C. Server Sync | Upload/download individual operations (SuperSync) | Complete ✅¹ |
| D. Validation & Repair | Prevent corruption, auto-repair invalid state | Complete ✅ |
¹ Cross-version sync: receiver-side op migration (A.7.11) runs before conflict detection. The remaining caveat is the released fleet: v17.0.0–v18.14.0 clients apply newer-schema ops (up to schema 5) unmigrated — see the A.7.11 Bump Policy.
✅ Migrations Active: Migration safety (A.7.12), tail ops consistency (A.7.13), and unified migration interface (A.7.15) are implemented, and real migrations exist —
CURRENT_SCHEMA_VERSION = 4(v1→v2 misc-to-tasks-settings split; v2→v3 and v3→v4 are semantic compatibility barriers). See A.7 and the A.7.11 Bump Policy.
This document is structured around these four purposes. Most complexity lives in Part A (local persistence). Part B handles file-based sync via the FileBasedSyncAdapter. Part C handles operation-based sync with SuperSync server. Part D integrates validation and automatic repair.
Local intent ──► NgRx reducer ──► live state
│
└──► capture + ordered append ──► SUP_OPS
│
├──► SuperSync operation transport
└──► file-provider envelopes
Remote input ──► migrate/filter/resolve ──► reducers + archive side effects
│
└──► durable checkpoint/cursor
The operation log is not incidental complexity. It is the minimum design that satisfies one hard, non-negotiable constraint:
Design goal: no silent data loss on concurrent multi-device edits, offline-first, with a "dumb" server that cannot merge (file providers have no server logic; SuperSync payloads can be end-to-end encrypted and opaque to the server).
This is the constraint the architecture is intended to satisfy, not a claim that all races are closed. The #9073 no-pending mitigation reconstructs retained concurrent local operations and routes supported overlapping crossings through deterministic LWW, but it cannot do so when the local side is no longer retained or cannot be decomposed safely. The focused conflict contract documents that residual and its possible class-level fixes.
Independent prior analyses (three separate model reviews) evaluated every simpler approach against that constraint and rejected each:
| Alternative | What it is | Why rejected |
|---|---|---|
| Last-Write-Wins (global timestamp) | Drop logical clocks; newest wall-clock write wins | User devices have unreliable clocks; concurrent independent field edits silently overwrite each other. Unacceptable for a personal productivity app. (Survives only as a field-level tie-break inside conflict resolution.) |
| Delta / state-diff sync | Keep a shadow copy, upload changed fields, server shallow-merges | Shadow state has no atomic coupling with the watermark → a crash mid-sync corrupts permanently; LWW shallow-merge loses concurrent independent edits; O(N) JSON.stringify diffing freezes the UI at 10k+ tasks; requires server-side merge, incompatible with opaque E2EE payloads. |
| Full-state / snapshot sync | Sync whole model files (the old PFAPI model) | Re-transfers everything on every change; no per-entity conflict granularity; cannot reconstruct intent after an offline edit. Retained only as a bootstrapping mechanism (snapshot + tail replay), not the sync mechanism. |
| CRDTs (Yjs/Automerge/etc.) | Math-guaranteed convergence | High conceptual complexity; most implementations assume a trusted server or relay, clashing with the dumb-file + E2EE constraint. The op-log deliberately borrows op-based-CRDT properties (UUID idempotency, causal ordering) without the full machinery. |
| Server-assigned sequence numbers | Let the server impose a total order | Requires server connectivity for ordering — incompatible with offline-first and file-based providers that have no server. Used only as a complement (SuperSync seq for global order; vector clocks still required for the file-based/offline case). |
Consequences any future redesign must preserve: classify concurrent independent edits before overwriting them and keep any remaining residuals explicit; work without a trusted/merging server and with opaque E2EE payloads; rebase offline edits cleanly on reconnect; retain tombstones long enough; bound growth via snapshot + compaction; prefer false-concurrency over false-ordering in conflict metadata (compare clocks before pruning); scale to 10k+ active / 20k+ archived tasks without main-thread O(N) work.
The only self-identified over-engineering historically was the vector-clock
pruning defense layers, which were since removed (see
vector-clocks.md).
The operation log is the durable transition log for local persistence. It is WAL-like, but the reducer runs before asynchronous capture/append, so the divergence guard is part of its safety contract. It provides:
SUP_OPS contains more than a single append-only table. Its current stores and
indexes are defined by
OperationLogStoreService,
which is the authority for upgrades and transaction boundaries:
| Durable concern | Role |
|---|---|
| Operation rows | Retained local and remote operations plus mutable delivery/application metadata |
| State cache | A boot-time baseline with its covered local sequence, vector clock, schema version, and entity frontier |
| Clock/client/meta rows | Working causal state, device identity, full-state metadata, and replacement/rebuild recovery markers |
| Young/old archive stores | Archived tasks and time-tracking data that live outside NgRx |
The exact operation envelope is owned by
@sp/sync-core and narrowed for
the application in
operation.types.ts. Do not copy
those interfaces into design docs: both the envelope and row metadata evolve.
Synced application-model recovery data lives in SUP_OPS. Provider credentials,
conflict-journal records, plugin caches, and local UI/browser settings have
separate owners; see the user-data reference.
Downloaded operations use a durable status transition so reducer state, archive IndexedDB side effects, vector clocks, and the server cursor cannot disagree after a crash:
pending — the remote op is stored, but no reducer-commit checkpoint exists yet.archive_pending — reducers committed and the op's vector clock was merged atomically; archive side effects have not yet completed.failed — an archive side-effect attempt failed. retryCount is charged only to the attempted row, so later rows in the same batch do not consume retry budget.applied — reducer and archive work both completed.Bulk replay isolates conversion/reducer exceptions per operation. The reducer-successful
subsequence is checkpointed and receives archive side effects; ordinary reducer-failed remote
rows are marked with terminal rejectedAt plus reducerRejectedAt metadata in the same transaction.
reducerRejectedAt is distinct from an ordinary sync rejection: hydration excludes that row
because its migrated reducer effect never entered state. Pending rows deliberately removed by a
schema migration receive the same terminal marker. This prevents one malformed operation from
terminating NgRx's state pipeline or receiving archive work, without creating a crash window
where startup could mistake it for incomplete reducer work.
Full-state and local operations are exceptions: a full-state failure discards the entire speculative bulk batch, while a local replay failure aborts hydration without rejecting the user intent. Neither case is terminally acknowledged when its state never entered NgRx.
Startup recovery leaves surviving pending rows pending, then hydration replays them through the
same per-operation reducer-failure collector. Successful rows and reducer failures are durably
partitioned before archive retry or snapshot creation. A pending full-state operation never uses
the direct-load shortcut. During live sync, a full-state reducer failure aborts before the reducer
checkpoint and server cursor advance, so the pending row remains recoverable. Hydration retries
archive_pending/failed rows with reducer dispatch disabled. Ordinary sync refuses to download,
upload, or advance its cursor while any incomplete rows remain. Version 8 introduced a downgrade
barrier for the reducer/archive checkpoint; version 9 similarly prevents older readers from replaying
operations quarantined with reducerRejectedAt.
Local actions buffered during a remote-apply window stay ordered until each operation is durable. Transient persistence failures keep the failed suffix queued and block the current sync so a later sync can retry. A deterministically invalid buffered action also remains queued, but requires reload: its reducer already changed live state, so discarding it would let live state diverge from the durable operation log.
User Action
│
▼
NgRx reducer commits live state
│
└──► capture meta-reducer marks the local persistent action pending
│
└──► non-dispatching effect, ordered with concatMap
│
├──► validate identifiers/payload
├──► create operation + incremented clock
└──► lock + atomic operation/clock append
│
├──► success: upload/compaction bookkeeping
└──► failure: surface reload + fence compaction
Only actions with explicit meta.isPersistent: true enter the capture path.
Remote/replayed actions set meta.isRemote and are never captured again. During
remote application, new local persistent actions are buffered and appended later
with clocks based on the applied remote frontier.
The contract is executable in
persistent-action.interface.ts,
operation-capture.meta-reducer.ts,
and
operation-log.effects.ts.
UI-only state and hydration/replay plumbing must not masquerade as new user
intent.
App Startup
│
▼
OperationLogHydratorService
│
├──► Load snapshot from SUP_OPS.state_cache
│ │
│ └──► If no snapshot: Genesis migration from 'pf'
│
├──► Run schema migration if needed
│
├──► Dispatch loadAllData(snapshot, { isHydration: true })
│
└──► Load replay range (seq > snapshot.lastAppliedOpSeq)
│
├──► If the final op carries full state and no reducer work is pending:
│ validate and load that state directly
│
├──► Otherwise: migrate operations, then replay the result
│ (migration may transform, split, or drop obsolete rows)
│
└──► If replayed >10 ops and state is valid: save a new snapshot
Two optimizations speed up hydration:
Direct-load a safe terminal full state: When the last replayable operation is a SYNC_IMPORT, BACKUP_IMPORT, or REPAIR, and no row in that replay range still has pending reducer work, the hydrator validates and loads its full state directly. Pending work disables the shortcut so those rows can be replayed and checkpointed.
Save snapshot after replay: After replaying more than 10 tail operations, a new state cache snapshot is saved. This avoids replaying the same operations on subsequent startups.
With no state cache, hydration first runs the local legacy migration check and
then re-reads the cache. If both the cache and operation log are empty, the app
keeps its normal initial NgRx state; it does not manufacture the pseudo-snapshot
shown in older versions of this document. Legacy pf recovery is allowed only
after proving that SUP_OPS contains neither a snapshot nor operation rows, and
the recovery operation plus snapshot are committed atomically. See
operation-log-hydrator.service.ts
and
operation-log-recovery.service.ts.
Without compaction, the op log grows unbounded. Compaction:
Normal compaction drains local capture before taking the operation-log lock. It then refuses to snapshot while remote work is incomplete, a local operation is pending or undrained, a persistence failure left live state ahead, hydration is running in fallback mode, or the live store has no meaningful data. Skipping is safe because the retained log remains the recovery source.
On success it snapshots current state with the latest local sequence, working
vector clock, schema version, and entity frontier, resets the compaction counter,
then prunes only rows that are terminal, covered by the snapshot, and older than
the retention cutoff. Active unsynced rows and incomplete remote rows survive.
The exact guard ordering is load-bearing; follow
operation-log-compaction.service.ts
rather than reimplementing it from prose.
| Setting | Current value | Meaning |
|---|---|---|
| Automatic attempt | 500 appends | In-memory/persisted counter threshold |
| Normal terminal-row retention | 7 days | Recent synced/rejected evidence remains available |
| Emergency retention | 1 day | More aggressive eligible-row pruning after quota failure |
| Phase timeout | 25 seconds | Abort before an overlong compaction outruns lock safety |
| Failure notification | 3 consecutive failures | Surface persistent maintenance failure |
These values are centralized in
operation-log.const.ts.
Browser builds use the Web Locks API to serialize named critical sections over shared IndexedDB. Electron and Android WebView are single-instance and use an in-process promise mutex. Browsers without Web Locks also fall back to that single-tab mutex, which cannot protect two tabs.
The app does not broadcast operation payloads between live tabs. Startup uses
a BroadcastChannel handshake to block a second same-origin instance. That
single-instance policy and the Web Locks layer are complementary safeguards; see
StartupService and
LockService.
Remote/replayed actions carry meta.isRemote: true. Re-running ordinary effects
for them can duplicate notifications, external calls, and—most dangerously—new
persistent actions. Therefore effects inject
LOCAL_ACTIONS, which excludes
remote actions. The sole broad-stream exception is the operation-log capture
effect, whose own filters enforce the persistence boundary.
This is one half of the atomic-intent rule. A state transition that must replay
atomically across synced slices or entities belongs in a meta-reducer so the
reducer pass and captured operation remain one unit; an effect fan-out creates
multiple independently syncable operations. Broader workflows may deliberately
remain independent persistent actions when their normal side effects and
entity-specific conflict boundaries matter, as documented for
project completion in ADR #5.
Selector-driven effects also require the hydration/sync guard. The normative
contributor rules and examples live in
contributor-sync-model.md.
1. Detect: Hydration fails or returns empty/invalid state
2. Verify SUP_OPS has neither a snapshot nor any operation rows
3. Only when SUP_OPS is provably empty, check legacy 'pf' database for data
4. If found: Run recovery migration with that data
5. Otherwise: restore through sync or a user-selected backup
Automatic legacy recovery runs the emptiness check and legacy write under the operation-log
lock, and fails closed. A present snapshot, a non-empty operation log, or an inspection error
prevents the legacy write and propagates the hydration failure. The generic hydration catch
must never place an older pf copy at the current SUP_OPS sequence frontier. When recovery is
allowed, the recovery operation, state-cache snapshot, and vector clock commit in one IndexedDB
transaction; an interrupted write cannot leave a snapshot claiming an operation that rolled back.
The exact branches matter: a corrupt but present snapshot first falls back to
retained-op replay, while legacy recovery is a last resort for a provably empty
SUP_OPS database. Follow
operation-log-hydrator.service.ts
and
operation-log-recovery.service.ts
rather than translating this summary into recovery code.
When Super Productivity's data model changes (new fields, renamed properties, restructured entities), schema migrations ensure existing data remains usable after app updates.
Current Status (2026-07):
CURRENT_SCHEMA_VERSION = 4. Three migrations exist: v1→v2 (misc-to-tasks-settings split, a real payload transformation) and two no-op semantic barriers — v2→v3 (replacement-mode LWW envelopes) and v3→v4 (marked project delete-wins). The barriers change no stored shapes; they gate conflict semantics for receivers that understand them. Read the A.7.11 Bump Policy before adding version 5.
CURRENT_SCHEMA_VERSION and MIN_SUPPORTED_SCHEMA_VERSION are defined in
packages/shared-schema/src/schema-version.ts
and re-exported by the client migration service. Current receivers have no
forward-compatibility skip band; the released-fleet exception is documented in
the bump policy below.
| Concept | Description |
|---|---|
| Schema Version | Integer tracking current data model version (stored in ops + snapshots) |
| Migration | Function transforming state from version N to N+1 |
| Snapshot Boundary | Migrations run when loading snapshots, creating clean versioned checkpoints |
| Forward Compatibility | Newer apps can read older data (via migrations) |
| Backward Compatibility | Older apps receiving newer ops (via graceful degradation) |
┌─────────────────────────────────────────────────────────────────────┐
│ App Update Detected │
│ (schemaVersion mismatch) │
└─────────────────────────────────────────────────────────────────────┘
│
┌───────────────────┼───────────────────┐
▼ ▼ ▼
Load Snapshot Replay Ops Receive Remote Ops
(older version) (mixed versions) (ordered remote batch)
│ │ │
▼ ▼ ▼
migrateState migrateOperation Screen, then migrate
shared chain shared chain compatible prefix
When app starts and finds a snapshot with older schema version:
App Startup (schema v1 → v2)
│
▼
Load state_cache (v1 snapshot)
│
▼
Detect version mismatch: snapshot.schemaVersion < CURRENT_SCHEMA_VERSION
│
▼
Run migration chain: migrateV1ToV2(snapshot.state)
│
▼
Dispatch loadAllData(migratedState)
│
▼
Force new snapshot with schemaVersion = 2
│
▼
Continue with tail ops (ops after snapshot)
Operations in the log may have different schema versions. Before replay, the hydrator runs the shared operation-migration chain.
One source operation can remain unchanged, be transformed, expand into several
operations, or be dropped as obsolete. The hydrator replays only the migrated
result and keeps source-operation IDs for durable reducer checkpointing. See the
shared migrate.ts chain and the
client
operation-log-hydrator.service.ts
for the executable contract.
RemoteOpsProcessingService
screens a downloaded batch in transport order. For each operation it first
validates the schema version. An invalid version, a version below the supported
minimum, a version newer than this client, or a migration failure stops the
batch at that operation. The compatible prefix may finish processing; the
blocked operation and suffix are neither stored nor applied. Callers leave the
transport cursor unchanged so that suffix is downloaded again after an update
or migration fix.
Compatible older operations run through the shared migration chain before
full-state filtering, conflict detection, or reducer application. A migration
may transform or split an operation; null is an intentional terminal drop and
does not block the cursor.
Full-state operations do not have a separate forward-compatibility path. They
pass through the same ordered compatibility screen as every other remote
operation: a too-new full-state operation blocks itself and the suffix, and the
caller freezes the cursor. Current code has no MAX_VERSION_SKIP branch and
does not attempt to load newer state by stripping unknown fields.
For an older full-state operation, the shared operation chain is the receiver
boundary. A migration that changes persisted state shape must cover both
snapshots through migrateState and relevant full-state or incremental payloads
through migrateOperation; replacement semantics run only after that operation
is compatible.
Migrations are defined in
packages/shared-schema/src/migrations/
and executed by the shared migrate.ts
chain. The stable type contract lives in
migration.types.ts:
every SchemaMigration supplies migrateState, while optional
migrateOperation accepts an OperationLike and returns OperationLike,
OperationLike[], or null.
How to create a new migration:
schema-version.ts, add
the next contiguous registry entry, and declare whether operation migration
is required.Transforming-migration residual: the receiver pipeline is implemented, but any future field rename or removal still needs a concrete payload transformation (or intentional drop) and cross-version tests. The existence of the shared chain alone does not make that change safe.
Note: The legacy PFAPI system has been removed (January 2026). This section documents historical migration paths.
For users upgrading from older versions (pre-operation-log), the ServerMigrationService handles migration:
SYNC_IMPORT operation with the imported stateKey file: src/app/op-log/sync/server-migration.service.ts
All future schema changes should use the Schema Migration system (A.7) described above.
Migration Safety (A.7.12) ✅ - Backup created before migration; rollback on failure.
Tail Ops Consistency (A.7.13) ✅ - Tail ops are migrated during hydration to match current schema.
Unified Migrations (A.7.15) ✅ - State and operation migrations linked in single SchemaMigration definition.
| Change Type | State Migration | Op Migration | Example |
|---|---|---|---|
| Add optional field | ✅ (set default) | ❌ (old ops just don't set it) | priority?: string |
| Rename field | ✅ (copy old→new) | ✅ (transform payload) | estimate → timeEstimate |
| Remove field/feature | ✅ (delete it) | ✅ (drop ops or strip field) | Remove pomodoro |
| Change field type | ✅ (convert) | ✅ (convert in payload) | "1h" → 3600 |
| Add entity type | ✅ (initialize) | ❌ (no old ops exist) | New Board entity |
Rule of thumb: Additive changes (new optional fields, new entities) don't need operation migration. Field renames/removals require it.
Status: Implemented receiver-side: compatible remote ops pass
SchemaMigrationService.migrateOperation() before conflict detection; a
too-new op is blocked before migration. Senders upload ops as-is.
Guardrails for newer-schema ops:
schemaVersion > CURRENT_SCHEMA_VERSION outright, freeze the download cursor, and prompt for an app update.CURRENT + 3 (their MAX_VERSION_SKIP) and apply those ops UNMIGRATED after a once-per-session warning — and they advance the cursor even when blocking, permanently skipping blocked ops. This fleet reality drives the A.7.11 Bump Policy.Required before: Any schema migration that renames/removes fields.
Status: Receiver-side state and operation migration is implemented. The Bump Policy below is normative.
The current receiver contract is deliberately one-way:
A version bump only fences receivers that ship AFTER the bump. As of 2026-07:
MAX_VERSION_SKIP = 3): it APPLIES ops up to schema 5 unmigrated after a once-per-session warning snack, and blocks schema ≥ 6 — but these clients advance the server cursor even while blocking, permanently skipping the blocked ops (loss that survives the later app update).Therefore:
CURRENT_SCHEMA_VERSION alone. Only bump when a change genuinely requires it: a transforming migration (renamed/removed field, dropped op) or a semantic you must hard-fence off older clients. Cautionary example — v4 (#9009, project delete-wins) was bumped for a marker-only change old clients degrade on fine: the feature is driven entirely by the payload marker (plus the entityId === projectId auth check); the schemaVersion >= 4 gate adds only narrow malformed-op hardening, not feature correctness. It needed no bump, yet it now fences every lagging post-v18.14.0 client and can't be undone. Don't repeat it.LwwUpdatePayload envelope pattern in packages/sync-core ('patch' ops apply correctly on pre-v3 clients via updateOne; the v4 delete-wins marker is inert for them). If they degrade, bumping is safe at any fleet share (the stamp is a fence for future receivers, not a protection for current ones) — but safe ≠ necessary: if it degrades, prefer a marker/envelope with no bump (see 0).Follow the executable contracts instead of copying their shapes into this guide:
remote-ops-processing.service.tsmigration.types.tsmigrate.tsschema-version.tsBefore release, tests must cover the concrete state transformation, every relevant operation result (including split or drop), retained-tail replay, and a remote batch whose incompatible operation leaves its suffix and cursor untouched.
WebDAV, Nextcloud, Dropbox, OneDrive, and LocalFile have no operation API. The client therefore adapts their file primitives to the same operation-sync interface used by the rest of the pipeline. The full visual tour lives in the field guide's transport section; this part records only the durable format boundary and its owners.
| Format | Remote files | Normal op-bearing sync |
|---|---|---|
| v2 monolith (default) | sync-data.json plus recovery backup | Downloads the changed monolith, merges its retained ops, rebuilds current state plus both archive partitions, and conditionally rewrites the complete monolith. |
| v3 split files (opt-in “Surgical sync”) | sync-ops.json, referenced snapshot generation, compatibility state/backup files, and a v2 tombstone | Conditionally rewrites the bounded ops commit point. A full state/archive snapshot is written only for initial bootstrap, compaction, migration, force-upload, or recovery. |
Both formats carry a vector clock, schema version, synthetic syncVersion, and
a bounded recentOps buffer. That common adapter and envelope do not themselves
provide physical compare-and-swap. The provider interface calls its
conditional-write token rev: a read returns it, and the adapter passes it back
as the expected token for upload. It is provider-native where the backend
supplies revision/ETag CAS and a synthetic content hash on the best-effort
fallbacks:
If-Match. Without a strong ETag, rev falls
back to a content hash; the pre-upload GET detects an already-stale writer but
cannot close the GET→PUT race, so concurrency protection is best effort.Where the provider enforces CAS, a revision mismatch aborts the write and a later cycle downloads before retrying. The best-effort backends cannot broadly guarantee that every simultaneous write race will abort.
The v3 migration is one-way for a sync folder. It leaves a v3 tombstone in the
legacy sync-data.json location so clients that do not understand the split
format stop instead of recreating an independent v2 history.
File providers do not expose a server-assigned operation cursor. The adapter
treats file syncVersion as a synthetic transport watermark and exposes it as
latestSeq to the common sync orchestration. A normal op-bearing commit advances
it once; snapshot replacement can reset it, which the gap path detects. It is
not the provider rev/ETag and does not prove per-operation ordering. One upload
can carry multiple operations under the same new watermark; stable operation
IDs provide durable deduplication, while vector clocks carry causality.
rev, vector clock, and expected synthetic
watermark remain staged until the caller confirms that baseline and ops were
durably applied. Cancelling a data-conflict decision does not advance the
baseline.A bootstrap or gap baseline is transport state, not an automatic new
SYNC_IMPORT for every download. The sync service hydrates the baseline under
the op-log/archive locks and records which retained operations the baseline
already contains; only the suffix beyond that boundary is replayed.
archiveYoung and archiveOld are local IndexedDB partitions, not independent
remote histories. A full file baseline includes both partitions; archive intent
also travels in operations so another client can execute the same idempotent
move/restore side effect.
| Contract | Owner |
|---|---|
| v2/v3 envelopes, filenames, caps, and snapshot reference | packages/sync-providers/src/file-based-sync-data.ts |
| Format selection, conditional IO, migration, gap detection, baseline staging, and archive inclusion | file-based-sync-adapter.service.ts |
| Baseline installation and common download/conflict orchestration | operation-log-sync.service.ts |
| Local archive side effects | archive-operation-handler.service.ts |
For server-based sync, the operation log IS the sync mechanism. Individual operations are uploaded/downloaded rather than full state snapshots.
| Aspect | File-Based Sync (Part B) | SuperSync (Part C) |
|---|---|---|
| Incremental unit | Bounded compact ops inside v2 or v3 file format | Individual retained operations |
| Baseline | Client-written full state/archive snapshot | Causal full-state operation in log |
| Transport watermark | Synthetic syncVersion exposed as latestSeq, plus ID dedup | Server-assigned sequence |
| Write-race guard | Provider rev; atomic only when the backend supports physical CAS | Database transaction |
| Server visibility | No application logic | Payloads opaque when E2EE is enabled |
The shared port is
OperationSyncCapable.
Its API and file-provider modes deliberately share one orchestration contract,
but differ in pagination and baseline behavior. For SuperSync, one normal cycle:
An incompatible operation, failed apply, or cancelled full-state decision leaves
its operation and suffix uncommitted, so a later cycle downloads them again. The
executable orchestration lives in
operation-log-sync.service.ts,
with upload routing in
operation-log-upload.service.ts
and ordered receiver processing in
remote-ops-processing.service.ts.
Operations that contain the full application state (SYNC_IMPORT,
BACKUP_IMPORT, REPAIR) use the dedicated /api/sync/snapshot route rather
than the regular operation-batch route. The route supports compressed large-body
transport, but the accepted request is still validated, quota-checked, and stored
as a full-state operation in the ordered log.
Upload Flow
│
├──► Filter: Is opType in { SYNC_IMPORT, BACKUP_IMPORT, REPAIR }?
│ │
│ ├──► YES: Upload via /api/sync/snapshot
│ │ • Uses uploadSnapshot() method
│ │ • SYNC_IMPORT → initial; BACKUP_IMPORT/REPAIR → recovery
│ │ • Supports E2E encryption
│ │
│ └──► NO: Upload via /api/sync/ops (normal batched upload)
Before upload, the client extracts and validates the wrapped full state, removes
device-local sync settings, optionally encrypts the state, and preserves the
original operation ID, vector clock, schema version, clean-slate/repair scope,
and import reason. Those fields are part of correctness and deduplication; do not
reconstruct the call from this prose. Follow
OperationLogUploadService,
the public provider contract in
provider-types.ts, and
the server's
snapshot handler.
| OpType | Snapshot Reason | Use Case |
|---|---|---|
SYNC_IMPORT | initial | First sync or full state refresh |
BACKUP_IMPORT | recovery | Restoring from backup file |
REPAIR | recovery | Auto-repair with corrected state |
The accepted upload remains a causal full-state operation in the server log, so clients can skip its covered prefix and apply the retained tail. For plaintext payloads only, an optional compressed cache accelerates server-side replay and restore generation; production clients do not download that cache. E2EE payloads remain replayable as operations but cannot populate a plaintext server cache.
Client conflict classification compares each incoming entity operation with a local entity frontier built from snapshot metadata, retained applied operations, and pending operations. A concurrent pending local operation supplies the normal two-sided conflict directly.
No pending row does not automatically mean “safe to apply.” For a concurrent op on a live entity, the #9073 mitigation reconstructs every retained local op still concurrent with the incoming clock. Supported overlapping, single-entity sides become a synthetic conflict and go through the same deterministic LWW path. Pairs that commute (identical content, disjoint real fields, noise-only changes, or positive task-time deltas) intentionally apply without LWW.
Arrival-order behavior remains only where the client cannot construct a safe, deterministic local side: for example, its evidence was compacted into the snapshot frontier, an operation is multi-entity, or the retained side is a local delete/archive that needs compensation machinery. Those fallback cases do not create a conflict object or journal row. See Composition residual (pre-existing class) for the remaining composition and mixed-receiver limitations.
The executable owners are RemoteOpsProcessingService and
ConflictResolutionService; server upload conflict detection is a second gate,
not a replacement for the client arrival-order problem above.
Conflicts first apply explicit semantic precedence and eligible disjoint-field merging, then
fall back to Last-Write-Wins (LWW) via
ConflictResolutionService.autoResolveConflictsLWW(). For the current high-level policy, see
the field guide's causality section; the focused
conflict journal and review contract owns the more volatile
merge and review details.
Winner selection and disjoint-field merging remain active in production. Conflict journaling is
an observe-only capability and is not required for resolution: the production remote-processing
path currently sets disableConflictJournal: true, so it does not emit journal entries. The
journal store and review UI therefore remain dormant/incomplete rather than a complete record of
resolved conflicts. See the focused contract above for current status and lifecycle details.
When local state is newer, we can't just reject the remote ops - that would cause the local state to never sync to the server. Instead:
Date.now() would give unfair advantage in future conflicts)A warning-level log is emitted: OpLog.warn('LWW local wins - creating update op for ${entityType}:${entityId}')
When operations are rejected (either local or remote):
getUnsynced() excludes rejected ops (won't re-upload)When a moveToArchive operation conflicts with a field-level update (e.g., rename, time tracking changes), the archive operation always wins regardless of timestamps. This bypasses the normal LWW timestamp comparison because archiving represents explicit user intent that should not be reversed by a concurrent field update.
Rationale: If Client A archives a task and Client B concurrently renames it, the archive must win — otherwise, the LWW update would "resurrect" the archived task back into the active store by replacing its state.
Implementation: ConflictResolutionService checks whether either the local or remote side contains a TASK_SHARED_MOVE_TO_ARCHIVE action. If so, the archive side wins automatically, and a new archive operation is created with a merged vector clock (via _createArchiveWinOp()).
This is the first level of archive resurrection prevention. The second level is the bulk archive filter, which pre-scans operation batches for archive operations and skips any LWW Update operations targeting entities being archived in the same batch. This two-level defense handles the 3+ client scenario where LWW Updates can arrive before or after archive ops in the same batch.
Key files:
src/app/op-log/sync/conflict-resolution.service.ts — Archive-wins check and _createArchiveWinOp()src/app/op-log/apply/bulk-hydration.meta-reducer.ts — Pre-scan archive filteringThe SupersededOperationResolverService treats moveToArchive as a special case alongside DELETE operations. When a moveToArchive op is rejected by the server due to concurrent conflicts, it is re-created with a merged vector clock instead of being discarded.
This is necessary because moveToArchive removes entities from the NgRx store (via the archive reducer), so getCurrentEntityState() returns undefined for archived entities. Without this special handling, the superseded operation resolver would be unable to re-create the operation, and archived tasks would be lost.
Implementation: Before entity-by-entity processing, SupersededOperationResolverService identifies bulk semantic operations like moveToArchive and re-creates them with the original payload and a merged vector clock, preserving the full task data in MultiEntityPayload format.
Key file: src/app/op-log/sync/superseded-operation-resolver.service.ts
The lwwUpdateMetaReducer handles LWW Update actions (created when the local side wins a conflict) differently depending on the entity's storage pattern:
| Storage Pattern | Entity Types | LWW Update Behavior |
|---|---|---|
| Adapter | TASK, PROJECT, TAG, NOTE, TASK_REPEAT_CFG, etc. | Individual entity replacement via NgRx entity adapter (updateOne or addOne) |
| Singleton | GLOBAL_CONFIG, TIME_TRACKING, MENU_TREE, WORK_CONTEXT | Entire feature state replaced with the winning data |
| Unsupported | Map, array, virtual patterns | Logged as warning; not supported for LWW |
For adapter entities, the meta-reducer also syncs relationships (e.g., project.taskIds when projectId changes, tag.taskIds when tagIds changes, TODAY_TAG.taskIds when dueDay changes, parent.subTaskIds when parentId changes).
Key file: src/app/root-store/meta/task-shared-meta-reducers/lww-update.meta-reducer.ts
A non-blocking snack notification is shown after auto-resolution:
When a SYNC_IMPORT or BACKUP_IMPORT operation is received, it represents an explicit user action to restore all clients to a specific point in time. Operations created without knowledge of the import are filtered out.
The executable owner is
SyncImportFilterService,
with causal classification shared through
classifyOpAgainstSyncImport.
Consider this scenario:
SYNC_IMPORT and BACKUP_IMPORT establish a clean slate. An operation that is
causally greater than or equal to that boundary stays; an operation dominated by
it is already represented and is dropped. A genuinely concurrent operation is
also dropped because it was authored without knowledge of the reset.
Pruned/reset clocks can make a post-import operation compare concurrent. The classifier therefore keeps two provable cases: a higher counter from the import's own client, or an operation carrying at least the import client's boundary counter. These are causal proofs, not timestamp guesses. The latest boundary is chosen by durable batch/store order, never UUID order.
REPAIR is not an explicit clean slate. Causally older work is represented by
its full state, but concurrent work normally replays on top. For a repair in the
same downloaded batch, a concurrent prefix covered by its
repairBaseServerSeq is dropped as already represented; a legacy repair without
that proof moves the prefix immediately after the repair boundary. Concurrent
suffix work remains valid.
All of these decisions use vector clocks because the question is causal knowledge of the full-state boundary, not wall-clock recency.
See the field guide's causality and conflict policy for the visual overview.
Validation is layered, but not every checkpoint automatically mutates user data. In particular, boot hydration prefers showing recoverable data over opening a repair dialog or silently rewriting it.
| Boundary | What runs | Failure behavior |
|---|---|---|
| Local capture | Structural operation-payload validation before append | Do not persist the operation; mark live/durable divergence, surface reload, and fence compaction |
| Snapshot hydration | Structural/cache screening plus schema migration; matching-schema snapshots use the trust fast path | Migration failure falls back to retained-log replay without overwriting the intact cache; validation errors are logged rather than auto-repaired |
| Tail/full replay | State validation after reducer replay | Continue with visible state but do not save an invalid replacement cache |
| Remote apply | Active-state validation, then full state including archives only if repair is needed | Repair and revalidate; persist a REPAIR operation before replacing live state, or mark the sync session failed |
Post-sync repair uses dataRepair() and revalidates the result. A successful
repair is represented by a causal full-state REPAIR operation containing the
repaired state, a repair summary, and—when available—the server sequence on
which the repair was based. The authoritative payload and summary types live in
operation.types.ts.
repairBaseServerSeq still equals the current server sequence. A stale repair is retired locally before the concurrent server suffix is downloaded.Before a local append,
validate-operation-payload.ts
checks the envelope and operation-specific payload structure. This is
intentionally shallower than whole-state Typia and relationship validation.
Internally generated REPAIR operations follow their own construction path.
The hydrator validates a snapshot synchronously when migration ran or its schema stamp does not match. A matching-schema cache is trusted for startup speed; this is why adding a required persisted field without a migration is dangerous. After tail or from-zero replay, validation gates creation of a replacement cache.
Hydration validation does not call dataRepair(): a boot-time native confirm
can steal focus, and silently repairing the only visible copy is worse than
loading it and logging the failure. See
operation-log-hydrator.service.ts.
After remote processing,
RemoteOpsProcessingService
calls ValidateStateService.validateAndRepairCurrentState(). The valid fast path
checks the active snapshot without archive reads. If invalid, it loads the full
state including both archive partitions, repairs and revalidates it, writes the
REPAIR operation/cache under the existing operation-log lock, and only then
dispatches the repaired replacement with remote-effect suppression. Failure
sets the session-validation latch, so the sync wrapper cannot claim IN_SYNC.
| Responsibility | Owner |
|---|---|
| Typia plus cross-model validation and repair orchestration | validate-state.service.ts |
| Pure repair transforms and summary accounting | data-repair.ts |
| Durable repair operation/cache creation and notification | repair-operation.service.ts |
| SuperSync repair base sequence | repair-sync-context.service.ts |
Quota failure is not an optimistic rollback path. The reducer has already run, so an unrecovered append leaves live state ahead of the log; the client marks that divergence, fences compaction, and offers reload.
The specialized quota branch recognizes raw browser variants and contains a
one-retry circuit breaker plus a 24-hour emergency retention policy. Its current
reachability is deliberately narrow: the store wraps the standard Chromium
error into the generic persistence-failure path, while raw legacy variants reach
emergencyCompact(). Because that call is still inside the failing write's stack,
the pending-write guard currently makes the compaction attempt skip. This is why
the retry code must not be described as successful recovery today; a future
delete-only emergency compactor would change that boundary. See the load-bearing
comments in
operation-log.effects.ts
and
operation-log-compaction.service.ts.
The 500-ops compaction trigger uses a persistent counter stored in state_cache.compactionCounter:
The sync clientId —
the device's stable sync identity — lives in the SUP_OPS client_id store
(key current). It used to live in the legacy pf database; storing it in
SUP_OPS lets destructive flows (clean-slate, backup-restore) rotate it
atomically inside runDestructiveStateReplacement's transaction, instead of a
hand-rolled cross-database two-phase commit. pf remains a read-only, one-time
migration source: the first read on a not-yet-migrated device copies the id
forward (ClientIdService). The clientId is non-regenerable (it keys the vector
clock), so a transient IndexedDB read failure propagates rather than minting a
fresh id.
pending remote row existsarchive_pending and failed quarantine rows survive regardless of agefalse when it skips for pending reducer work or an empty/degraded
live state; callers only treat an actually written snapshot/prune pass as successThe application splits active NgRx state from two IndexedDB archive partitions.
archiveYoung receives newly archived tasks and non-today time tracking;
archiveOld holds tasks moved past the 21-day threshold and time tracking moved
during the periodic full young-to-old flush.
Archive partitions can contain tens of thousands of tasks and worklogs. Treating them as an always-rewritten remote file makes a small archive transition pay for the whole historical dataset. The cost depends on the transport: default v2 file sync still rewrites that full baseline, while SuperSync and the opt-in v3 file format can normally transfer the operation without rewriting a remote archive snapshot.
Archive changes are replayable operations with deterministic, idempotent local
side effects. The receiver updates its own IndexedDB archive partitions rather
than installing a separately versioned archive database file. This does not
mean archive data never crosses the network: moveToArchive carries the full
task data needed by a receiver, and full file-sync baselines include both archive
partitions.
| Transport | What crosses the network for archive changes |
|---|---|
| SuperSync | The archive operation payload; no separate archive-file upload. |
| File v2 (default) | The operation buffer plus complete state, archiveYoung, and archiveOld in the rewritten monolith. |
| File v3 (opt-in) | Normally the operation in sync-ops.json; complete archive partitions when a snapshot is created or replaced. |
When a user archives tasks:
moveToArchive operation.archiveYoung and moves non-today local
time-tracking data out of active state.moveToArchive operation.ArchiveYoung.Result: Both clients apply the same archive transition locally. SuperSync does not transfer a separate archive file; file v3 normally avoids a full archive snapshot rewrite between compactions; default file v2 does not.
The originating client moves eligible data, then emits flushYoungToOld with
the captured timestamp. Remote clients run the same threshold calculation under
the archive mutex and atomically commit the new young/old pair. Passing the
timestamp in the operation keeps replay independent of each receiver's wall
clock.
All archive operations MUST be idempotent:
| Operation | Guarantee |
|---|---|
moveToArchive | Skip if task already in archive |
flushYoungToOld | Move only items not already in Old |
restoreFromArchive | Skip if task already in Active |
The exact mutation and retry behavior belongs to
ArchiveOperationHandler
and ArchiveService.
Time tracking is a nested project/tag → context ID → date → compact session
map. Its archive boundary differs from the task age boundary:
Daily (finish work):
all non-today active entries → archiveYoung
Every ~14 days (flush):
all archiveYoung time tracking → archiveOld
Task archive flush in the same operation:
only task families older than 21 days → archiveOld
Full-state assembly merges the three sources at field level with priority
current > archiveYoung > archiveOld. Incremental sync remains operation-based;
notably, concurrent positive task-time deltas commute and apply both instead of
being reduced to whole-entry LWW.
| Responsibility | Owner |
|---|---|
| State shape | time-tracking.model.ts |
| Full-state three-source merge | merge-time-tracking-states.ts |
| Daily and periodic partitioning | sort-data-to-flush.ts |
| Remote archive application | archive-operation-handler.service.ts |
TaskService.moveToArchive()
persists the selected parent-task batch, then dispatches one
moveToArchive({ tasks }) action. Capture records that persistent action as one
operation carrying the selected batch's full task payload. The receiver needs
that data because archive storage is outside NgRx and the tasks may no longer
exist in its active state.
There is no archive-specific chunking. SuperSync's
DEFAULT_SYNC_CONFIG
limits each operation payload to 20 * 1024 * 1024 bytes (20 MiB) by default.
A sufficiently large archive action can therefore exceed the per-operation
limit; even below it, one very large payload is a known scalability and failure
boundary. Current code neither splits nor compresses this action, and no
replacement design is specified here.
This section documents the architectural principles ensuring that related model changes happen atomically, preventing state inconsistency during sync.
When a user deletes a tag, multiple entities must be updated:
tagIds updatedIf these changes happen in separate NgRx effects:
Principle: Related entity changes that must replay as one atomic transition should happen in a single reducer pass.
Meta-reducers wrap the root reducer and can update every affected slice during
that one pass. For example,
tag-shared.reducer.ts
owns tag deletion cleanup across tasks, repeat configurations, and time
tracking. Its tests are the executable contract; do not reproduce the reducer
shape from this guide.
| Meta-Reducer | Purpose |
|---|---|
tagSharedMetaReducer | Tag deletion cleanup (tasks, repeat cfgs, time tracking) |
projectSharedMetaReducer | Project deletion cleanup |
taskSharedCrudMetaReducer | Task CRUD with tag/project updates |
taskSharedLifecycleMetaReducer | Task lifecycle (archive, restore) |
taskSharedSchedulingMetaReducer | Task scheduling with Today tag updates |
plannerSharedMetaReducer | Planner day management |
taskRepeatCfgSharedMetaReducer | Repeat config deletion with task cleanup |
issueProviderSharedMetaReducer | Issue provider updates |
operationCaptureMetaReducer | Marks the action as pending capture (increments counter) |
The OperationCaptureService and operation-capture.meta-reducer work together using a pending counter to track captures (no positional queue — see the note below):
OperationCaptureService.incrementPending() with the actionentityChanges via OperationCaptureService.extractEntityChanges(), writes the operation, then decrements the counter in a finallyentityChanges[] arrayflushPendingWrites() polls getPendingCount() to know when every dispatched action has been written. NgRx reducers process actions sequentially and the effect uses concatMap, so writes stay ordered.
Why a counter, not a positional FIFO queue (#8306 / #8318): the old design queued an EntityChange[] per action and correlated meta-reducer push with effect shift purely by position. If a write threw before its dequeue ran (e.g. a LockAcquisitionTimeoutError), the entry leaked and flushPendingWrites() could never reach 0 — every later sync then failed after its 30s timeout. A counter decremented in a finally cannot leak. entityChanges is now computed in the write path from the action (a pure function), so there is nothing to keep positionally aligned.
Note: Most actions return empty entityChanges[] - the action payload is sufficient for replay. Only TIME_TRACKING and TASK time sync actions have special handling to extract entity changes from the action payload. The field is still emitted (even as []) because the Android background provider reads it and the isMultiEntityPayload guard requires it.
User Action (e.g., Delete Tag)
│
▼
tagSharedMetaReducer (+ other meta-reducers)
├──► Atomically update all related entities
│
▼
Feature Reducers
│
▼
operation-capture.meta-reducer
├──► Call OperationCaptureService.incrementPending(action)
│ └──► Increments the pending counter
│
▼
OperationLogEffects (per-action wrapper: writeOperationFromEffect)
├──► Call OperationCaptureService.extractEntityChanges(action)
├──► Create + persist single Operation with action payload
└──► finally: OperationCaptureService.decrementPending()
| Scenario | Pattern |
|---|---|
| One replay-atomic transition across slices/entities | Meta-reducer |
| Independent persistent workflow step | Ordinary reducer action |
| Entity deletion whose cleanup must replay with deletion | Meta-reducer |
| UI notifications (snackbar, sound) | Effect using LOCAL_ACTIONS |
| External API calls | Effect using LOCAL_ACTIONS |
| Archive operations (async I/O) | Dedicated archive operation handler path |
| Navigation/routing | Effect using LOCAL_ACTIONS |
Rule of thumb: state changes that must replay atomically across slices or
entities use a meta-reducer. Independent workflow steps remain ordinary
persistent actions; effects own I/O and UI side effects and use LOCAL_ACTIONS.
The deliberate workflow exception and its costs are documented in the
Contributor Sync Model
and ADR #5.
For references between entities (e.g., tag.taskIds), we use a "board-style" pattern where:
task.tagIds)tag.taskIds) is for ordering onlySelectors recompute membership from the source of truth, filter stale ordering
IDs, preserve the stored order, and append missing members. The current handling
of nested tagged tasks is subtle; use
computeOrderedTaskIdsForTag
instead of a copied implementation.
When adding new entities or relationships:
entityType, entity IDs, opType)
and cover capture/replay with the real action shapeLOCAL_ACTIONS in effects for side effects onlyUse the field guide's stable
executable source map instead of maintaining
a copied file tree here. The main implementation boundaries are
src/app/op-log/,
packages/shared-schema/src/,
packages/sync-core/src/,
packages/sync-providers/src/, and
packages/super-sync-server/src/.