Back to Cherry Studio

FileManager Architecture

docs/references/file/file-manager-architecture.md

2.0.0112.6 KB
Original Source

FileManager Architecture

SoT scopethis document owns: FileEntry / FileRef data models, physical storage layout, version detection & concurrency control (OCC), atomic writes, recycle bin, reference cleanup, DirectoryWatcher internals, on-demand orphan sweep, DanglingCache state machine, and AI SDK integration design. Module-level concerns (type system, IPC / DataApi contracts, layered architecture, business-service integration, lifecycle assignment) live in architecture.md. In case of conflict, the layer ownership above decides: positioning / contract → the module-level doc, implementation → this document.

When a section describes a behavior (dispatch, OCC, atomic writes, orphan sweep, etc.), read it as the specification the implementation must satisfy. Sections explicitly tagged "(deferred)" describe planned architecture that is not yet implemented.


1. Core Concepts

1.0 Management Principle

FileManager provides file management capabilities for two origins; callers choose based on their own needs:

  • internal: Cherry owns the file content, stored at {userData}/Data/Files/{id}.{ext}. The caller hands the source content to FileManager, which copies it and takes over the lifecycle.
  • external: Cherry only records an absolute path reference on the user's side; does not copy content. File availability and content changes are determined by the user side.

The caller decides the origin; FileManager makes no assumptions about the business layer.

Best-effort semantics for external: an external entry is a persistent record that "the caller expressed intent to reference this path at some point in time"—no guarantee the file remains stable, no guarantee content matches the reference-time content. Cherry does no bidirectional DB-FS sync, doesn't track external rename/move; external changes naturally surface as "reading new content next time" or "dangling".

Data categories that do not enter FileManager (auto-derived data, logs, Agent workspace, OCR intermediates, MCP config, files self-managed by FS-first modules, etc.) are detailed in architecture.md §1.3.

1.1 FileEntry

Each FileEntry corresponds to a file the user uploaded/saved. FileEntry is a flat individual record—no directory tree, no parent-child relationship, no mount concept.

FileEntry
├── id: UUID (v7 for v2-native entries; v4 preserved from v1 Dexie migration)
├── origin: 'internal' | 'external'
├── name: filename (without extension)
├── ext: extension (without leading dot), nullable
├── size: bytes
├── contentHash: '{algo}:{hex}' content-dedup detection hash; internal-only, null while unknown/in-flight/awaiting repair
├── externalPath: absolute path, non-null only when origin='external'
├── deletedAt: ms epoch | null
├── createdAt / updatedAt

1.2 Origin: internal vs external

The origin field of each FileEntry defines content ownership:

originPhysical locationOwnershipMutability
internal{userData}/Data/Files/{id}.{ext}Fully owned by CherryRead-write
externalAbsolute path pointed to by externalPathOwned by user, referenced by CherryChangeable by explicit user action (write / rename / permanentDelete apply, delegated to the FS primitives); Cherry does no automatic/watcher-driven modifications; does not track external rename/move—external changes cause the entry to naturally go dangling

Path uniqueness: at most one entry can exist whose externalPath agrees with another under case folding. Implemented via SQLite functional unique index:

sql
CREATE UNIQUE INDEX fe_external_path_lower_unique_idx
  ON file_entry (lower(external_path));

fe_external_path_idx (plain index on the raw external_path) backs byte-exact lookups (findByExternalPath, rename re-finds, path-resolution call sites). The functional index simultaneously serves the case-insensitive lookup path (WHERE lower(externalPath) = lower(?)) used by findCaseInsensitivePeers and enforces the uniqueness invariant — ensureExternalEntry MUST resolve case-collisions at the application layer before INSERT (see "Duplicate-entry detection on insert" below) because a DB-level rejection would otherwise surface as an opaque SQLITE_CONSTRAINT. Internal rows (externalPath = NULL) are exempt — SQLite treats multiple NULLs as distinct in a UNIQUE index.

Stored form of externalPath — byte-faithful: externalPath is persisted exactly as the OS handed it to us, with only lexical cleanup (null-byte reject, .//..//repeated-separator collapse, trailing-separator trim). It is not Unicode-normalized, so the stored string always reaches the real file on every filesystem — including normalization-sensitive ones (Linux ext4/btrfs) where an NFC-rewritten path would not exist on disk (see Rejected: Unicode (NFC) normalization below). Uniqueness is therefore byte-exact plus case-fold only: the lower() functional index catches identical and case-different paths; it deliberately does not fold Unicode forms.

Compile-time enforcement via the AbsoluteFilePath brand: AbsoluteFilePathSchema.parse() returns a branded AbsoluteFilePath (Zod brand, zero runtime cost; see src/shared/types/file/common.ts), and the external-entry lookup/write surfaces (findByExternalPath, setExternalPathAndName) accept the narrower CanonicalFilePath produced by the canonicalizeFilePath() factory. Every DB read/write surface that filters by externalPath MUST accept the branded type, not a plain string, so a caller cannot pass an unvalidated raw path and silently miss all matches. (AbsoluteFilePathSchema only validates the absolute-path shape — it does not mutate the value; see AbsoluteFilePath vs CanonicalFilePath.) "Narrower" is a proper subset, not a spelling normalization: canonicalizeFilePath() is partial over AbsoluteFilePath and throws on input it cannot reduce to a key (today: UNC — see UNC paths). Holding an AbsoluteFilePath does not entitle a call site to assume it can obtain the CanonicalFilePath these surfaces demand.

SourceNatively canonicalRelies on normalization to disambiguate
Electron showOpenDialog✅ (OS returns the on-disk true case)None
Drag-drop from Finder/Explorer✅ (OS drag source)None
User-typed @/path/... / clipboard pasteRisk of case / NFD/NFC
External URL scheme / shell integrationSame as above
v1 migration (inherits Dexie stored values)❌ (inherits legacy value quality)Canonicalize once during migration

Normalization scope (synchronous, no FS IO) — lexical cleanup only:

  • Null-byte rejection — raw.includes('\0') → throw, so poisoned paths never reach DB persistence (reject at the earliest boundary, not at use-time inside resolvePhysicalPath)

  • path.resolve(raw) → absolutize + eliminate ./ ../

  • Trailing separator trimming

  • Windows: drive-letter case folding (c:\C:\) and separator normalization to \ (so C:/a/b and C:\a\b share one stored form)

  • UNC rejection — \\server\share\… throws. \\server\share is an indivisible root that ../ must not be able to escape, and neither the POSIX nor the drive-letter branch models that; producing a silently wrong key is worse than refusing. Note the asymmetry this creates with the brand, and that it is deliberate: see UNC paths below.

These steps are purely lexical (no FS IO). Unicode (NFC) normalization is deliberately NOT a step here — an NFC-rewritten NFD path would not exist on disk on normalization-sensitive filesystems, so byte-faithful storage is what keeps the stored path reachable (see Rejected below). The .//../ collapse, by contrast, is not guaranteed to preserve the on-disk target across symlinks/junctions (/a/link/../b/a/b if link resolves elsewhere) — this cleanup is a lexical dedup-key normalizer, not a reachability/security primitive; use fs.realpath at the main-process boundary when true target equivalence matters.

Intentionally omitted (deferred until concrete user feedback warrants the cost):

  • fs.realpath as a step inside canonicalizeFilePath itself (would require async FS IO at every call site and a file-existence precondition). fs.realpath IS used on the ensureExternalEntry collision path described below — that is a per-collision probe, not a per-canonicalize step.
  • Symlink target merging at canonicalize time
  • Windows 8.3 short-name resolution

See the JSDoc for canonicalizeFilePath in src/shared/utils/file/canonicalize.ts for the detailed contract.

UNC paths

AbsoluteFilePathSchema accepts UNC (\\server\share\…); canonicalizeFilePath rejects it. The split is the point, not an oversight — the two gates answer different questions:

GateQuestionUNC
AbsoluteFilePathSchemaIs this shape safe to hand to fs?✅ Node reads UNC natively on Windows
canonicalizeFilePathIs this safe to persist as a byte-compare dedup key?❌ no defined \\server\share root handling

Consequence: a UNC path can be read, previewed, and copied into an internal entry (createInternalEntry copies bytes; it never canonicalizes). It cannot become an external entry, because that requires a canonical externalPathensureExternalEntry and findByExternalPath both throw a descriptive UNC error rather than storing a corrupted key.

Why the brand accepts it. UNC is an absolute filesystem path, and the schema was already inconsistent about it: fileUrlToPath decodes a UNC file:// URL to the forward-slash form //server/share/…, which has always passed the POSIX branch. Rejecting only the backslash spelling of the same path meant a network-share file:// snapshot (the shape v1 migration produces) silently became an unreadable, dropped attachment. Accepting both spellings makes the gate describe a property of the path instead of a property of how it happened to be spelled.

Known gap (pre-existing, unchanged): the forward-slash form //server/share/x still passes the POSIX branch of canonicalizeFilePath and is silently reduced to /server/share/x. Only the backslash form is caught. Fixing it means deciding whether // is UNC or a POSIX double-slash path, which is platform-dependent — tracked separately, not addressed here.

URL encoding puts the server in the authority. toFileUrl emits file://server/share/… for a UNC path, not file:////server/share/…. The leading // produced by separator normalization already is the URL authority marker, so appending it to file:// would leave the authority empty and demote the server to path text — a URL Node rejects with ERR_INVALID_FILE_URL_PATH. This also makes toFileUrl the exact inverse of fileUrlToPath, which decodes file://host/… to //host/….

Containment checks degrade, they do not throw. isPathWithinAccessiblePath / getAccessiblePathRelativePath (src/renderer/components/composer/variants/agent/accessiblePath.ts) canonicalize before comparing. A path that cannot be canonicalized is not provably inside anything, so they return false / the input unchanged. Containment is a predicate; it must stay total.

Rejected: Unicode (NFC) normalization of externalPath

An earlier design NFC-normalized externalPath (inside canonicalizeFilePath) before persistence, so a file named with a decomposable character (é = e + combining acute vs the precomposed é) would dedup to one row on macOS. This is deliberately rejected. externalPath is stored byte-faithful (see "Stored form" above) and is never Unicode-normalized.

  1. Reachability (blocking). On a normalization-sensitive filesystem (Linux ext4/btrfs, some SMB/NFS mounts) the directory entry stores exact bytes. NFC-normalizing an NFD-named file's path yields a string that does not exist on disk, so ensureExternalEntry's fs.stat, the copy source, and resolvePhysicalPath all ENOENT — the file becomes unimportable and unreachable. Breaking Linux functionality to dedup on macOS is not an acceptable trade.
  2. Unverified premise. The "most common duplicate trigger for CJK users" justification was AI-authored in a comment; this v2 code has never shipped, and there is no observed evidence that this app's path sources (showOpenDialog, drag-drop, fs.readdir) ever surface the same file in divergent normalization forms.
  3. Rare and cosmetic even on macOS. A duplicate row needs a conjunction: the same file added more than once, through two ingestion routes that disagree on normalization, with a decomposable non-ASCII name, stored on disk in a divergent form. Even then APFS is normalization-insensitive, so both rows resolve to the same physical file — the harm is one extra, user-deletable list entry, not data loss or breakage. And the fs.realpath collision probe below does not catch it (NFD vs NFC are not lower() case-peers), so NFC-in-storage was the sole mechanism folding it — bought at the cost of item 1.
  4. YAGNI. Storage-time normalization plus its rule-evolution migration discipline is real, permanent complexity for an unverified, rare, cosmetic condition.

If real NFD/NFC duplicates are ever observed, the correct fix is an FS-aware fs.realpath fold at the collision probe (run only when the file exists), never re-introducing Unicode normalization into canonicalizeFilePath or the stored form.

Rule-evolution discipline

Removing the NFC step removed a reachability hazard (an NFC-rewritten path does not exist on disk). It did not remove the desync hazard, which is a separate problem with a separate cause: the canonical form is application-layer logic whose output is used as a byte-compare key. Any change to canonicalizeFilePath that changes its output desynchronizes historical rows (written under the old rule) from new queries (running under the new rule) — regardless of whether the step consults the filesystem. The failure is silent: byte-compare misses, the user sees "my file is in the library but the app says it isn't", and ensureExternalEntry inserts a duplicate.

Every remaining step is output-sensitive. The .//../ collapse is the likeliest one to be revisited, precisely because "Normalization scope" above already records that it is not target-preserving across symlinks; changing it would invalidate every stored row containing a collapsed segment.

Rule: modifying canonicalizeFilePath ≡ ship a paired Drizzle migration that re-canonicalizes every existing file_entry row with origin='external' in the same PR. No exceptions — even if the new rule is claimed "strictly more permissive", the byte-compare will still miss.

Current exception (expires at the first shipped release). v2 has never shipped, so there are no historical rows to desync — src/main/data/db/schemas/ and migrations/sqlite-drizzle/ are pre-release throwaway per CLAUDE.md. That is why removing the NFC step needed no paired migration. This exemption is temporal, not a property of the rule: it stops applying the moment a release writes real user rows, and it never covered the rule itself.

When a rule change additionally collapses previously-distinct strings to the same canonical form (e.g. folding / and \ on Windows merges C:/a and C:\a), the migration MUST also merge the colliding rows. The rules below are prescriptive; follow them exactly rather than improvising per-migration.

Winner selection when merging rows:

  1. Oldest createdAt wins (preserves user-visible history — a 3-year-old entry's creation timestamp is more valuable than a 3-day-old one's).
  2. Tiebreaker: highest ref count (keeps the entry that more of the user's data already points at).
  3. Final tiebreaker: smallest id by lexicographic order (deterministic, no FS-state dependency).

Losers' dependents (executed in the same Drizzle transaction as the merge):

  • Association rows with fileEntryId = loser.iddeduplicate, then update to winner.id. chat_message_file_ref and painting_file_ref are unique on (fileEntryId, sourceId, role), not on (sourceId, role) — one message legitimately references many files under the same role. So a source that referenced both a loser and the winner (or two losers) yields duplicate rows the moment they are rewritten to winner.id. Delete every loser row whose (sourceId, role) the winner already covers, then update the survivors; that is the semantically correct merge, since the two refs now point at one file. Any conflict remaining after this step is a genuine invariant violation — fail loudly, never ON CONFLICT DO NOTHING. provider_logo_file_ref / mini_app_logo_file_ref are unique on (sourceId) alone, so a source holds at most one row and a plain update cannot collide.
  • file_entry.id = loser.id → delete.
  • Any downstream consumer of loser.id (future file_upload.fileEntryId, business-service caches keyed by entryId) MUST be enumerated and updated in the same migration. If you add a new table that references file_entry.id, the canonicalization migration procedure expands — document the expansion alongside the table's schema.

Atomicity: the entire re-canonicalize + merge operation runs in one Drizzle migration transaction. On failure the DB rolls back to the pre-migration state and the next startup re-attempts; partial progress is not possible.

Renderer-side cache invalidation: after the migration runs, some React Query caches keyed by the loser's id may be stale. Because migrations execute before the renderer boots, this is self-healing on the first query — no special coordination required.

Duplicate-entry detection on insert

Case-insensitive uniqueness on externalPath is enforced at both layers: the functional UNIQUE index fe_external_path_lower_unique_idx (DB) and ensureExternalEntry's pre-INSERT collision check (application). The two-layer scheme keeps the DB-level guarantee unbreakable while letting the application disambiguate the FS-correct interpretation case-by-case.

typescript
// Inside ensureExternalEntry, AFTER canonicalize, AFTER findByExternalPath miss,
// AFTER fs.stat verifies the new path exists, BEFORE INSERT:
const peers = await fileEntryService.findCaseInsensitivePeers(canonicalPath)
if (peers.length > 0) {
  // `fs.realpath` is the platform-correct probe for "are these the same FS
  // entity": on case-insensitive volumes (macOS APFS default, Windows NTFS
  // default) the FS folds case to its on-disk canonical form, so two case-
  // different inputs resolve to the same string. On case-sensitive volumes
  // (Linux ext4, case-sensitive APFS) they resolve to distinct strings.
  const reusable = await resolveCaseCollisionPeer(canonicalPath, peers)
  if (reusable) return reusable // same FS entity → reuse existing entry
  // No peer is the same FS entity. The DB unique constraint will reject the
  // INSERT, but we throw early with a descriptive error and full peer detail
  // so the caller can decide (rename one of the colliding paths, or surface
  // the conflict to the user) instead of seeing an opaque SQLITE_CONSTRAINT.
  throw new Error(`ensureExternal: case-collision with existing entries…`)
}
// No peers → safe to INSERT; DB unique constraint is now a redundant safety net.

Behavioral matrix (/foo/A.txt already an entry; user invokes ensureExternalEntry('/foo/a.txt')):

Filesystem classfs.realpath('/foo/A.txt')fs.realpath('/foo/a.txt')Outcome
Case-insensitive (macOS APFS default, NTFS default)/foo/A.txt/foo/A.txt (FS folds)Same string → reuse existing entry
Case-sensitive (Linux ext4, case-sensitive APFS)/foo/A.txt/foo/a.txtDistinct strings → throw case-collision
Dangling peer (/foo/A.txt missing on disk)ENOENT/foo/a.txtCannot disambiguate → throw case-collision (caller must permanentDelete the dangling row first)

Scope:

  • Runs only on the insert branch; reuse / update / read branches never invoke peer detection.
  • The lookup is O(log N) (index-backed), so the previous "best-effort, skip above 10k rows" heuristic is removedfindCaseInsensitivePeers now runs unconditionally regardless of table size.
  • The fs.realpath call resolves symlinks too, which is the right semantic for "same logical file"; symlink targets are intentionally NOT canonicalized at storage time (see "Intentionally omitted" above), so two symlinks pointing at the same target each get their own entry, but a case-different reference to one of those entries reuses it.

This subsumes the fs.realpath upgrade that earlier revisions of this section described as "deferred until user feedback" — the same probe is applied at exactly the moment it matters (collision resolution) without paying the FS IO cost on every canonicalize call.

Invariants:

Fieldorigin='internal'origin='external'
nameSoT (user can rename actively)Pure projection of externalPath (basename)
extSoTPure projection of externalPath (extname)
sizeSoT (non-null, ≥ 0)Always null — no DB snapshot; live value via getMetadata
externalPathNULLAbsolute path (the authoritative identity of external)
contentHashDetection substrate {algo}:{hex} (for example xxh3-64:…); maintained on every write; null means unknown, in-flight, or awaiting repairAlways null — content lives outside Cherry (fe_contenthash_external_null CHECK)

For external entries the row stores only identity + stable projections. name / ext do not drift because externalPath is fixed for the lifetime of the entry (external rename by the user surfaces as a dangling entry, not an in-place rewrite of name). size / mtime are served live by File IPC getMetadata(id) on demand — see §3 External Entry Liveness Model.

Content-level dedup detection (contentHash)

contentHash is a detection substrate for content-level deduplication, not an identity or key. It is orthogonal to external-path deduplication: external paths enforce identity and upsert behavior, while internal hashes only surface possible content matches for a consumer-defined policy.

  • Tagged format. Values use {algo}:{lowercase hex}; the current producer is XXH3-64 via native @node-rs/xxhash. Keeping the algorithm in the value allows old and new algorithms to coexist during a future incremental migration.
  • Internal-only and derived by Main. createInternalEntry, write, writeIfUnchanged, and createWriteStream persist the current hash and size. Renderer/callers cannot provide an authoritative hash. Bytes, base64, copy, download, and stream sources calculate both values incrementally from the same byte stream that materializes the durable tmp file; no post-write reread or caller-trusted shortcut is used. External rows never carry either snapshot.
  • Non-null is a trust invariant. A non-null hash describes the current managed blob. Before replacing internal content, FileManager atomically snapshots the old metadata, sets contentHash = NULL, and records the foreground commitAt in updatedAt; after rename it writes the prepared size / contentHash while preserving the row's current updatedAt. Rename failure restores the old metadata (and restores the old timestamp only when no concurrent metadata update advanced it). Rename success followed by DB-finalize failure keeps the hash null and throws ContentCommittedMetadataPendingError; callers refresh the file state and must not retry the same write automatically.
  • Non-unique by design. fe_content_hash_idx is a plain index. Multiple logical entries may contain identical bytes, and a collision must never raise a constraint violation. FileEntryService.findInternalByContentHash, exposed to renderers through DataApi GET /files/entries/by-content-hash, returns active candidates oldest-first without selecting one; FileManager retains the same query for Main-side consumers.
  • Insert-always remains unchanged. createInternalEntry never looks up or reuses candidates. A consumer may hash source bytes, query candidates, apply its own secondary check or user decision, and then either reuse or create; the created entry's persisted hash is still derived independently by Main from the bytes it writes.
  • Startup trust generation + reconciliation. Before File IPC registration, FileManager checks app_state['fileManager:contentMetadataGeneration']. A missing, malformed, or non-v2 marker causes one synchronous transaction to null every non-null internal hash (explicitly preserving updatedAt) and write { version: 2 }. This safely invalidates hashes created by older releases without rewriting an executed migration. After all services are ready, the singleton file.contenthash-backfill job keyset-pages through null rows by id, including trashed rows, and rebuilds size and contentHash from one read. It shares FileManager's per-entry write lock and persists with contentHash IS NULL, so it cannot overwrite a foreground write. Missing or unreadable blobs remain null for the next startup attempt. During reconciliation, dedup queries may temporarily miss candidates but never trust an incorrect non-null hash.

1.3 FileRef (Business Reference)

Business objects associate with FileEntry through source-owned ref tables plus a shared FileRef projection:

chat_message_file_ref / painting_file_ref / ...
├── fileEntryId → FileEntry (FK, CASCADE delete)
├── sourceId → owning source row (FK, CASCADE delete)
├── role: business-semantic reference role (defined by the source module)
└── UNIQUE(fileEntryId, sourceId, role)

FileRefService aggregates these source-owned tables into the shared FileRef discriminated union for DataApi reads, ref counts, and sweep reporting. It does not own persistent ref writes.

When a persistent business object is deleted, SQLite FK cascade removes its association rows. Relationship replacement (for example, replacing a painting's complete file set) is handled directly by the owning business service.

1.4 FileHandle / FileInfo — see architecture.md §2

FileHandle (polymorphic reference crossing IPC), FileInfo (path-indexed data shape), and the full reference-vs-data-shape symmetry are defined at the module-level architecture document, not here. This document concerns FileManager's internal implementation only.

Method applicability inside FileManager:

CategoryMethods
Accept FileHandle (entry + path branches via IPC dispatch)read / getMetadata / getVersion / getContentHash / write / writeIfUnchanged / rename / permanentDelete / copy / open / showInFolder
Accept FileEntryId only (entry-identity operations; no path-handle counterpart)trash / restore / createInternalEntry / ensureExternalEntry / withTempCopy

1.5 FileUpload (AI Provider Upload Cache) — deferred

AI SDK SharedV4ProviderReference integration and the file_upload table are deferred until the Vercel AI SDK Files API stabilises. The module-level DataApi surface (ensureUploaded / buildProviderReference / invalidate) is outlined in architecture.md §3.5; the detailed schema and FileUploadService API are retained here in §9 AI SDK Integration for the eventual landing PR.

1.6 FileManager Implementation Layout (Facade + Private Internals)

FileManager is the sole public entry point of the FileEntry management system but is not a 30-method God class. The implementation uses a facade + private pure-function modules pattern. (Note: "sole public entry" is scoped to FileEntry — the file module also hosts DirectoryTreeBuilder as a parallel top-level primitive for in-memory tree mirrors; see architecture.md §1.2 and directory-tree.md.)

1.6.1 Why It Can Be Split

A method-by-method audit of FileManager's public API for "does it depend on class instance state" concludes: the vast majority of methods do not depend on instance state.

StateUsersOwnership
versionCache (LRU)write / writeIfUnchanged / getVersionclass private field (held by FileManager instance)
fileEntryService / fileRefServiceAll DB operationscontainer singleton (application.get(...))
danglingCacheExternal-related methodsfile-module singleton (module import)
@main/utils/file/*All FS operationspure functions, stateless
IPC handler registration handleslifecyclemanaged by onInit / onStop

Only versionCache and lifecycle artifacts are truly bound to the FileManager instance; business methods themselves are stateless.

1.6.2 Module Layout

src/main/services/file/
├── index.ts              ← barrel: exports FileManager + public entry-facing types/helpers
├── FileManager.ts        ← facade class; lifecycle + legacy IPC + versionCache + inline getMetadata
├── internal/             ← private implementation (not re-exported by index.ts; external imports forbidden)
│     ├── deps.ts              — FileManagerDeps type
│     ├── dispatch.ts          — FileHandle.kind dispatch helper (entry vs path adapter)
│     ├── entry/
│     │    ├── create.ts       — createInternal / ensureExternal
│     │    ├── lifecycle.ts    — trash / restore / permanentDelete + batches
│     │    ├── rename.ts
│     │    └── copy.ts
│     ├── content/
│     │    ├── read.ts         — entry-aware read / createReadStream
│     │    ├── write.ts        — write / writeIfUnchanged / createWriteStream
│     │    └── hash.ts         — getContentHash / getVersion
│     ├── system/
│     │    ├── shell.ts        — open / showInFolder
│     │    └── tempCopy.ts     — withTempCopy
│     └── orphanSweep.ts       — FS-level orphan sweep
├── utils/
│     ├── content.ts           — consistent path read + path conditional write
│     ├── metadata.ts          — path-arm metadata projection
│     └── pathResolver.ts      — FileEntry path resolution + external canonicalization
└── versionCache.ts       ← LRU type definition

getMetadata is the one entry-level read that does NOT live under internal/entry/ — it is implemented inline on the FileManager class because it is a thin wrapper around fs.stat with no entry-flow logic of its own. Adding a future entry-flow concern (e.g. presence event emission on success) would justify extracting it, but until then the inline definition keeps the facade's stat path single-hop.

1.6.3 Dependency Passing Convention

Each internal/* pure function explicitly receives FileManagerDeps:

typescript
// internal/deps.ts (illustrative — see src/main/services/file/internal/deps.ts for the authoritative definition)
export interface FileManagerDeps {
  readonly fileEntryService: FileEntryService
  readonly fileRefService: FileRefService
  readonly danglingCache: DanglingCache
  readonly versionCache: VersionCache
}

// internal/entry/create.ts — two APIs, corresponding to two public methods on the FileManager facade
// Note: CreateInternalEntryParams is a source-discriminated union
//   (source: 'path' | 'url' | 'base64' | 'bytes'); each branch only exposes content
//   that name/ext cannot be derived from. Full matrix in `src/shared/file/types/ipc.ts`.
export async function createInternalEntry(
  deps: FileManagerDeps,
  params: CreateInternalEntryParams
): Promise<FileEntry> {
  // Extract { name, ext, bytes } by source branch → write physical file → DB insert; always produces a new entry
}

export async function ensureExternalEntry(
  deps: FileManagerDeps,
  params: EnsureExternalEntryParams
): Promise<FileEntry> {
  // Upsert by externalPath: reuse the existing row or insert a new one
  // (external entries cannot be trashed, so there is no restore branch)
}

1.6.4 Thin-Delegation Facade

typescript
// FileManager.ts (illustrative — see src/main/services/file/FileManager.ts for the authoritative wiring)
@Injectable('FileManager')
@ServicePhase(Phase.WhenReady)
export class FileManager extends BaseService implements IFileManager {
  private readonly _versionCache = createVersionCacheImpl(2000)
  private readonly deps: FileManagerDeps = {
    fileEntryService,
    fileRefService,
    danglingCache,
    versionCache: this._versionCache
  }

  // Public API: thin delegates. Internal modules export entry-flavoured
  // functions directly (no `*ByEntry` suffix — see §1.6.5); `*ByPath`
  // siblings exist only on the path branch and are NOT exposed here.
  createInternalEntry(params) { return createInternal(this.deps, params) }
  ensureExternalEntry(params) { return ensureExternal(this.deps, params) }
  read(id, opts?) { return read(this.deps, id, opts) }
  trash(id) { return trash(this.deps, id) }
  // ... one line per method

  protected async onInit() {
    await this.deps.danglingCache.initFromDb()
    this.registerIpcHandlers()
    // Auto-policy reaper: a non-awaited startup pass + an idle-gated interval
    // (file-entry-cleanup.md §5.5). The tick also fires the FS orphan sweep
    // concurrently, behind its own weekly floor (`fileSweepTick`). The `runSweep`
    // umbrella is separate — on demand via IPC only.
    void this.runEntryCleanup()
    this.registerInterval(() => this.entryCleanupTick(), CLEANUP_INTERVAL_MS)
  }
}

1.6.5 FileHandle Dispatch Convention (Adapter Responsibility at the IPC Boundary)

Dispatch location: FileHandle.kind dispatch stays at the IPC handler registration site. Rationale:

  • FileHandle is the input shape at the IPC serialization layer—the renderer sends a { kind, ... } tagged union, and post-deserialization kind-dispatch is a "request interpretation" concern—it is the proper responsibility of the IPC adapter layer
  • FileManager's public API remains entry-native (accepts only FileEntryId); main-side business service calls are intuitive without needing a createFileEntryHandle(id) wrapper
  • The FilePathHandle branch only needs the IPC handler; main-side business services hold FileEntries—they have no arbitrary-path scenario

Implementation convention: entry-aware actions stay under internal/* and receive FileManagerDeps; renderer-facing path-arm actions have no entry state to coordinate, live under utils/*, and are re-exported by the module barrel:

typescript
// internal/content/read.ts
export async function read(deps, entryId, opts): Promise<ReadResult<T>>

// utils/content.ts
export async function readByPath(path, opts): Promise<ReadResult<T>>
export async function writeIfUnchangedByPath(path, data, version): Promise<FileVersion>

Naming convention (per the shipped exports): entry-flavoured variants use the bare verb (read, createInternal, ensureExternal, trash, copy, rename, …); path-flavoured siblings carry the *ByPath suffix. The bare entry variant is what FileManager's public method delegates to. Renderer-facing *ByPath variants do not flow through FileManager's public methods — they serve the path-handle branch of the IPC handler and live in utils/* so internal/* remains private.

Unified style for dispatch helper: generic FileHandle routes use the file module's dispatchHandle helper at the renderer transport boundary. Operations whose contract is intentionally path-only call their path helper directly:

typescript
// src/main/ipc/handlers/file.ts
export const fileHandlers = {
  'file.read': async ({ handle, options }) =>
    options.mode === 'range'
      ? dispatchHandle(
          handle,
          id => fileManager.readChunk(id, options.offset, options.length),
          path => readChunkByPath(path, options.offset, options.length)
        )
      : dispatchHandle(
          handle,
          id => fileManager.read(id, { encoding: options.encoding }),
          path => readByPath(path, { encoding: options.encoding })
        ),
  'file.write_if_unchanged': async ({ path, data, expectedVersion }) =>
    writeIfUnchangedByPath(path, data, expectedVersion)
}

Impact of adding a new handle kind (e.g., virtual pointing into archive members, remote pointing to an S3 URI):

  1. src/shared/data/types/file.ts — add variant to handle union
  2. Relevant internal/*/*.ts or utils/*.ts — add the entry-aware or path-like operation
  3. src/main/ipc/handlers/file.ts — extend dispatchHandle; each IPC handler explicitly handles that kind (or throws "unsupported")

The renderer extension surface is concentrated in the File IPC adapter—it is immediately obvious which kinds each IPC method supports, which aids auditing.

1.6.6 External Access Constraints

LocationMay importForbidden to import
Main-side business service (KnowledgeService, MessageService, etc.)@main/services/file (FileManager + selected path helpers) / @main/utils/file/{fs,path,metadata,search,shell} / @main/services/file/watcherDeep imports into @main/services/file/internal/** or @main/services/file/utils/**
Inside the file module itself (internal/*, watcher/*)May reference each other as needed; may also import @main/utils/file/* primitivesExcept FileManager, must not import internal/*
External Node/rendererN/A (file-module is main-side)

Boundary enforcement: the src/main/services/file/index.ts barrel does not re-export general internal/ implementation. dispatchHandle is the temporary documented exception while legacy handlers remain; path-arm operations are implemented in utils/* and re-exported through the barrel. ESLint's barrel/closed rule rejects deep imports.

1.6.7 Design Trade-offs

OptionAdopted?Rationale
Split business methods into 5 lifecycle servicesOverkill—lifecycle registration, dependency ordering, and test mocking costs all 5×, in exchange only for "methods split across files"
FileManager as facade + internal/* pure functionsOnly 1 lifecycle node; pure functions can be unit-tested with stub deps directly; external API surface remains stable
FileAccessor as a standalone class handling FileHandle dispatchDispatch belongs to the IPC adapter layer; the adapter reuses dispatchHandle from internal/dispatch.ts via the file-module barrel, so another class would add pure complexity
FileManager public API switched to handle-nativeIPC and Main-side call contracts need not share shape; main-side business services using entry-native directly is more intuitive, without needing a createFileEntryHandle wrapper
Extract versionCache as a module singletonAs a FileManager private field, it naturally supports test isolation (new instance = fresh cache)

1.6.8 Event Emission & Broadcast (deferred)

FileManager exposes three typed Event<T> on its instance surface and forwards each to every live renderer window. The public contract and queryKey invalidation table live in architecture.md §3.6; this section pins down the emission mechanics that the FileManager implementation must satisfy.

typescript
class FileManager extends BaseService {
  readonly onEntryRowChanged: Event<EntryRowChangedEvent>
  readonly onEntryContentChanged: Event<EntryContentChangedEvent>
  readonly onDanglingStateChanged: Event<DanglingStateChangedEvent>

  private readonly _entryRow = new Emitter<EntryRowChangedEvent>()
  private readonly _entryContent = new Emitter<EntryContentChangedEvent>()
  // onDanglingStateChanged is re-exposed from DanglingCache — see §11.7

  constructor(private readonly windowManager: WindowManager) { super() }

  protected override onInit(): void {
    this.registerIpcHandlers()
    danglingCache.initFromDb()

    // Wire internal events → renderer broadcast. Each disposable auto-cleans on stop.
    this.registerDisposable(this.onEntryRowChanged((e) =>
      this.windowManager.broadcast('file-manager-event', { type: 'entry-row', ...e })))
    this.registerDisposable(this.onEntryContentChanged((e) =>
      this.windowManager.broadcast('file-manager-event', { type: 'entry-content', ...e })))
    this.registerDisposable(this.onDanglingStateChanged((e) =>
      this.windowManager.broadcast('file-manager-event', { type: 'dangling-state', ...e })))
  }
}

Emission point per mutation (all emits happen after the DB transaction commits, never inside):

MethodEmit
createInternalEntry / batchonEntryRowChanged { kind: 'created' } per newly-inserted row
ensureExternalEntry / batchInsert branch → onEntryRowChanged { kind: 'created' }. Reuse branch → no emit (no state change).
write / writeIfUnchangedAfter atomic rename + (internal-only) DB size update commits → onEntryContentChanged { id, version }. StaleVersionError → no emit.
createWriteStreamOn stream 'finish' → emit. On 'abort' / 'error' / .destroy() → no emit.
renameAfter DB commit (and FS rename for external) → onEntryRowChanged { kind: 'updated' }.
trash / restore / batchAfter DB update commits → onEntryRowChanged { kind: 'updated' } per affected id.
permanentDelete / batchAfter DB delete commits (internal: FS unlink runs first; external: FS untouched per §1.2) → onEntryRowChanged { kind: 'deleted' }. CASCADE-dropped association rows emit no extra events — the renderer invalidates ['fileManager', 'entries'] and refetches.
copyCreates a new internal entry → emit onEntryRowChanged { kind: 'created' } for the new id only (source is untouched).

Atomicity & crash semantics: emits are plain Emitter.fire() calls, not part of the DB transaction. A process crash between commit and fire loses the event. This is acceptable because:

  1. The committed DB state is authoritative — user data is durable
  2. The renderer's staleTime contract (see architecture.md §4.1.1) refreshes the query on its natural cadence
  3. Next renderer query loads the latest state — no permanent desync

Events are accelerators, not authoritative notifications. This is an explicit design choice documented in architecture.md §3.6.

Multi-window fan-out: windowManager.broadcast iterates every live webContents in the window pool and sends the IPC payload. Backgrounded / minimized windows still receive the IPC (Electron delivers to paused webContents); they process it on resume via React Query's standard refresh-on-focus behavior, and any brief overlap with the renderer's first query after resume is resolved idempotently by queryClient.invalidateQueries. Windows whose renderer bundle has not finished loading miss the event entirely, but their first query hits a fresh state with no ghost data.

Testing contract: unit tests of mutation methods must not depend on broadcast — windowManager.broadcast is mocked in test fixtures. The emission-point table above is the contract tests should assert against (e.g. "ensureExternalEntry reuse branch does not emit").


2. Storage Architecture

2.1 Physical Path Rules

Physical paths are not persisted; resolved at runtime based on origin:

typescript
function resolvePhysicalPath(entry: FileEntry): string {
  if (entry.origin === 'internal') {
    return application.getPath('feature.files.data', `${entry.id}${entry.ext ? '.' + entry.ext : ''}`)
  }
  return entry.externalPath!
}

internal physical paths are always flat: {userData}/Data/Files/{uuid}.{ext}, and do not change with the FileEntry's name. UUID naming makes internal files invisible and not manually organizable by the user—this is an intentional design choice.

external physical paths are entirely determined by the user; Cherry does not touch them.

2.2 Physical Directory Structure

{userData}/Data/Files/
├── {uuid-1}.pdf
├── {uuid-2}.png
├── ...
└── {uuid-n}.tmp-{uuid}      ← Temporary files for atomic writes (abnormal residues cleaned by `runSweep`)

Cherry creates no subdirectories under {userData}/Data/Files/. All internal files are stored flat.

2.3 Temporary File Handling

Transient processing files (OCR intermediates, PDF pagination, archive extraction, etc.) do not create FileEntry and use @main/utils/file/fs primitives directly under {userData}/temp/ (or process-level os.tmpdir()). After processing, the business side cleans up or relies on OS mechanisms.


3. External Entry Liveness Model

3.1 Design: No DB Snapshot for Drift-Prone Fields

The external file can be modified or moved by the user at any time. Rather than carrying a DB snapshot that silently drifts (and then chasing it with "refresh" paths), file_module stores only the fields that cannot drift while the entry exists:

Field on file_entry (external)Source of truthDrift possible?
id, origin, createdAt, updatedAtDB rowNo
externalPathUser intent at registration timeNo (user-explicit changes go through ensureExternalEntry(newPath))
name / extPure projection of externalPath (path.basename / path.extname)No (stable as long as externalPath is stable)
sizeNot stored — always null (enforced by fe_size_internal_only CHECK)N/A

Live size / mtime for an external entry are obtained via File IPC getMetadata(id) (fs.stat on demand). This makes the freshness cost explicit at the call site rather than hiding a stale snapshot behind the FileEntry.size field.

3.2 Why Size Is Not Stored

The classic "DB snapshot + refresh paths" design produces two symmetric defect classes:

  1. Stale reads — callers consume FileEntry.size assuming freshness, missing the part of the doc that says "snapshot may be stale".
  2. Bookkeeping bugs — every write / read / hash path has to remember to UPDATE the snapshot; forgetting one leaves the snapshot behind.

Making size unavailable on the row eliminates both: the renderer cannot read a stale value (there is nothing to read), and the main-side code has no snapshot to maintain. The cost — one extra fs.stat per external row when size is actually needed — is localized and observable.

Paths that would otherwise need to refresh a snapshot: read / getVersion / getContentHash on external still run fs.stat as part of their own work (and update DanglingCache as a side effect), but they do not write to the DB row — no size column exists to refresh.

Cherry does not track external rename: after a user mv/rename outside of Cherry, the corresponding entry goes dangling. The user must re-@ inside Cherry to establish a new reference at the new path via ensureExternalEntry(newPath).

3.3 Dangling Model

When an external file does not exist on disk (or is inaccessible), the corresponding entry is called dangling. Dangling state is maintained by DanglingCache (a file_module singleton); see §11 for details.

Three states:

StateMeaning
'present'The file was recently observed to exist (watcher event / stat success / ops operation observation)
'missing'The file was recently observed to be absent (watcher unlink / stat ENOENT)
'unknown'No watcher coverage, no prior stat (or cache was actively cleared)

Detection timing:

  • Passive (pull): File IPC getDanglingState / batchGetDanglingStates query → danglingCache.check(entry) (synchronous on cache hit, single fs.stat on cold miss). DataApi never reads this cache.
  • Active push: when a business module creates a watcher via createDirectoryWatcher(), the factory auto-wires add/unlink events into DanglingCache
  • Side effect: FileManager's own read/stat/write operations also update the cache on success/failure

UI semantics: dangling entries show a failed style in the UI (grayscale, icon marker), but dangling state itself never triggers cleanup—the ref association chain is preserved. manual entries require the user to explicitly "Remove from library" or attempt to re-point; delete_when_unreferenced entries are instead reclaimed by the ref-count-driven cleanup pass (file-entry-cleanup.md) regardless of presence.


4. Version Detection and Concurrency Control

4.1 FileVersion

typescript
interface FileVersion {
  mtime: number   // ms epoch
  size: number
}

Used as a fast signal for detecting external changes. Two tiers of usage:

  • Fast path: statVersion(path) (microsecond-level, covers 99% of cases)
  • Deep path: contentHash(path) → tagged XXH3-64 (millisecond-to-second level, used when mtime/size match but further confirmation is needed). This uses the same algorithm as persisted dedup detection, but the OCC value is computed on demand and is not persisted by the comparison itself.

Rationale for mtime + size as a signature:

  • Six scenarios where mtime alone fails—multiple writes within the same ms, clock rewind, backup preserving mtime, user touch, low-precision FS (FAT32), in-place 1-byte edit—are covered by size or hash as fallbacks

4.2 Read API

typescript
interface ReadResult<T> {
  content: T
  mime: string
  version: FileVersion
}

read(id, opts?: { encoding?: 'text' }): Promise<ReadResult<string>>
read(id, opts: { encoding: 'base64' }): Promise<ReadResult<string>>
read(id, opts: { encoding: 'binary' }): Promise<ReadResult<Uint8Array>>

read uniformly returns { content, mime, version }. Returning version comes at near-zero cost—stat is already required on the read path.

4.3 Write vs WriteIfUnchanged

typescript
write(id, data): Promise<FileVersion>
writeIfUnchanged(id, data, expectedVersion: FileVersion): Promise<FileVersion>

Two independent methods that force the caller to explicitly choose whether conflict detection is needed:

CallerWhich to useReason
First-time write, overwrite, migration, preprocessingwriteNo concurrency semantics
Editor save (Notes, Markdown, and other potential future consumers)writeIfUnchangedMust detect external changes

On conflict, writeIfUnchanged throws StaleVersionError, and the caller decides on UX after catching (dialog, three-way merge, keep both versions, etc.). Both entry and raw-path arms fully prepare the tmp payload first, then re-stat immediately before rename; an edit that lands while a large payload is being materialized is therefore detected before replacement.

Behavior on external: write / writeIfUnchanged / createWriteStream / rename / permanentDelete all apply—Cherry supports user-explicitly-triggered external file modifications (editor save, UI rename, user-confirmed delete), delegated to the FS primitives at @main/utils/file/fs (atomic write / rename / remove). Cherry does not perform automatic / watcher-driven external file modifications.

4.4 LRU Version Cache

FileManager maintains Map<FileEntryId, CachedVersion> internally (LRU, ~2000 entries):

TriggerActionPhase
write / writeIfUnchanged completesset(id, new version)Phase 1 (shipped) — see internal/content/write.ts:42, 68, 83
Critical path detects external changeset(id, new version)Deferred — paired with the change-detector that would observe "external change" outside the watcher path
Startup reconcile completesclear()Deferred — no startup reconcile pass exists yet

Trust boundary: the cache only accelerates getVersion queries and is not used for critical decisions. writeIfUnchanged's concurrency comparison must re-stat; it does not trust the cache.


5. Atomic Writes

5.1 tmp + fsync + rename Flow

All writes (entry/internal to userData, entry/external to externalPath, path-handle to any path) use PreparedAtomicWrite and follow the POSIX atomic flow:

1. Create {target}.tmp-{uuid} in the same directory
2. Write data while incrementally deriving size + contentHash
3. fsync(tmp fd)                  ← data flushed to disk; state = prepared
4. rename(tmp, target)            ← atomic replacement (POSIX guarantee)
5. fsync(dir fd)                  ← rename metadata flushed to disk; state = committed

Key rules:

  • fsync on by default. Cherry's write frequency is user-action level, and fsync on SSD costs < 10ms
  • tmp must be in the same directory as target. Cross-filesystem rename is not atomic
  • tmp naming: {target}.tmp-{uuidv7}—UUID avoids concurrent-write conflicts
  • Crash residue: FileManager's on-demand orphan sweep cleans up by ^.+\.tmp-<uuidv7>$
  • 2× disk usage is an inherent cost of POSIX rename semantics, unavoidable
  • commit() and abort() are terminal and idempotent; a prepared write can enter only one terminal state

For an internal entry, steps 3–5 are wrapped by the recoverable metadata protocol described in §1.2: mark hash unknown before rename, finalize exact prepared metadata afterward. A DB failure before rename leaves the target and old metadata untouched; a DB failure after rename leaves a durable null recovery marker instead of a stale non-null hash.

5.2 Stream Variant

typescript
createWriteStream(id): Promise<AtomicWriteStream>

Stream writes also follow the same prepared commit. The returned AtomicWriteStream extends Writable; _final() does not complete until rename, DB finalization, and version-cache update all succeed, so finish is an end-to-end success signal. .abort() / stream errors before commit unlink the tmp and do not replace the target. During shutdown, FileManager aborts streams that have not entered commit and waits for streams that have.

5.3 FS Primitive Access Policy

The atomicWriteFile / atomicWriteIfUnchanged / createAtomicWriteStream primitives exported by @main/utils/file/fs are open to modules outside the file module. BootConfig, MCP oauth storage, and any other main-process service that needs a safe atomic write imports them directly; scattered ad-hoc tmp+rename implementations are not introduced.

Renderer-provided raw paths are a separate trust boundary. Before any renderer-reachable path write/move/rename/delete/mkdir/import, Main rejects lexical or realpath overlap with application.getPath('feature.files.data'), including ancestors and symlinks into the managed tree. Internal FileManager operations use entry ids and are not routed through that raw-path guard.


6. Deletion and Recycle Bin

6.1 deletedAt Model

All soft deletes are implemented via the deletedAt timestamp, without physically moving files:

OperationPhysical impact (internal)Physical impact (external)
trash(id)NoneN/A (fe_external_no_delete CHECK rejects; external rows cannot be trashed)
restore(id)NoneN/A (no trashed external rows to restore)
permanentDelete(id)DB delete + best-effort remove(physicalPath) (@main/utils/file/fs)DB delete only — user's file is never modified (matches architecture.md §3.4)

trash / restore are internal-only. External entries cannot be trashed by definition (fe_external_no_delete CHECK enforces this); the trash semantics make sense only for files Cherry owns.

permanentDelete on internal: DB row is removed first, then the physical file at {userData}/Data/Files/{id}.{ext} is best-effort unlinked. Unlink failures (ENOENT, insufficient permissions, etc.) are logged but do not block — the DB-row-gone outcome is what callers observe; any orphaned blob is later cleaned by the next user-triggered orphan sweep (§10).

permanentDelete on external: DB row is removed; the user's file at externalPath is never modified — Cherry only owns the reference, not the content. This is the only safe contract: silently deleting user files from inside the app would violate the "best-effort external reference" semantics (§1.0.2 in architecture.md). Users who actually want the underlying file gone do so through their OS file manager.

6.2 Auto Expiry (deferred — lands in Phase 2)

Status: design only. Phase 1 ships no expiry timer service, no Preferences key, and no WHERE deletedAt < now() - retentionMs query. Trashed entries persist until the user runs an explicit permanentDelete (or the next user-triggered orphan sweep collects an already-deleted entry's residual blob). The 30-day window below is the proposed retention; the actual default and configurability land with the timer service.

By default trashed entries are cleaned up after 30 days (lifecycle service timer); the user may configure the days or disable it in Preferences.

Query: WHERE deletedAt < now() - retentionMs → batch permanentDelete.

6.3 Edge Cases

ScenarioHandling
unlink fails on permanentDelete internal (file already missing, permission issue)Log warn; the DB row is already gone, so the failure surfaces only as an orphan blob that the next user-triggered orphan sweep will reclaim
permanentDelete on externalDB-only by design; the user's file at externalPath is never touched — Cherry owns only the reference
ensureExternalEntry(path) when an entry for the same path already existsEntry point first calls AbsoluteFilePathSchema.parse(raw); upsert returns the existing row. External entries cannot be trashed, so there is no "restore" branch.
Two entries for the same fileCase differences (macOS APFS, Windows NTFS): rejected at INSERT by the DB lower() functional unique index plus the fs.realpath-based reuse-or-throw probe in ensureExternalEntry (see §1.2 "Duplicate-entry detection on insert"). Unicode (NFD ↔ NFC) differences: deliberately not deduped — externalPath is stored byte-faithful, and the rare, cosmetic macOS case is accepted rather than break reachability on Linux (see §1.2 "Rejected: Unicode (NFC) normalization of externalPath").
External file at original path externally replaced with a different fileCherry does not check content consistency (best-effort). name / ext on the row are derived from externalPath and do not change; size is always served live by getMetadata. DanglingCache flips to 'present' on the next stat, so the UI just renders the new file under the existing reference.
A trashed entry is permanently externally deleted and then restoredAppears dangling (DanglingCache returns missing on next check), UI shows failed style
External write with permission error / disk full on target pathThrow without polluting DB; caller decides retry or user notification

7. Reference Cleanup Mechanism

Three layers of protection, with each layer as a fallback for the next:

+-------------------------------------------------------+
| Layer 1: fileEntryId CASCADE                          |
| FileEntry deleted -> ref rows auto-cascaded           |
| file_upload auto-cascaded                             |
| (DB FK constraint, zero app code)                     |
+-------------------------------------------------------+
| Layer 2: source FK cascade / relationship replacement |
| business entity deleted -> source-FK cascade          |
| relationship replaced -> explicit cleanup+insert      |
+-------------------------------------------------------+
| Layer 3: on-demand DB orphan sweep                    |
| report zero-ref manual entries                        |
| reclaim zero-ref delete_when_unreferenced entries     |
| via the cleanup pass                                  |
+-------------------------------------------------------+

Layer 3 is not a generic persistent-source reconciler. Persistent association rows are FK-constrained and should disappear through Layer 1 / Layer 2 cascades; the sweep reports manual zero-ref entries, and reclaims delete_when_unreferenced zero-ref entries via the cleanup pass described in file-entry-cleanup.md.

7.1 No-Reference Entry Policy

The default stance — FileEntry is preserved even when no business refs point at it — is chosen so the user never loses a file they (or Cherry) bothered to track merely because the original consumer got deleted. A UI surface may show an "unreferenced" marker for user-triggered cleanup.

Automatic deletion applies only to entries whose cleanup_policy = 'delete_when_unreferenced' (see file-entry-cleanup.md); manual entries have no automatic deletion exceptions. Even an external manual entry that is currently missing and has zero refs is still a user-visible library record: it may represent a temporarily unmounted drive, a file the user wants to re-link later, or simply a stale record the user should remove explicitly. The file module may report these rows, but it must not delete them without an explicit user/caller action.

Policy matrix by (origin, dangling state, refs): the rows below describe manual-policy behavior. A delete_when_unreferenced entry is instead reclaimed once it clears the grace window with zero refs — see file-entry-cleanup.md §5.

origindangling staterefsPolicy
internaln/a (always 'present')anyPreserve — user may re-link via UI; only user-initiated cleanup
external'present'anyPreserve — file still exists, fully re-attachable
external'unknown'anyPreserve — not yet observed; treated as still-live until proven otherwise
external'missing'>0Preserve — business objects still reference this entry. Automatic deletion would CASCADE-drop association rows and silently mutate user data (messages' attachment count drops, UI state shifts). The business service owning those refs is the right layer to decide replacement / removal policy, not the file module. Reference-oriented UI surfaces (§3.4 UI convention) show these as "file missing" so the user can act.
external'missing'0Preserve + report — no refs remain, but the row is still user-visible library state. FilesPage / cleanup UI may show "missing" and offer "Remove from library"; no time-based auto-delete.

7.2 No Automatic Dangling-External Cleanup

This section applies to manual-policy entries. Dangling external manual entries are never deleted automatically by a scheduler, startup task, or runSweep() policy pass — deletion of delete_when_unreferenced entries is handled by the separate cleanup pass in file-entry-cleanup.md, and is driven by ref count and grace window, not dangling state. Cleanup for manual entries is explicit:

  • User action: FilesPage or a cleanup UI calls the external-entry deletion path (labelled "Remove from library") for selected rows.
  • Business action: a business service that owns a reference may decide how to handle a missing file in its own workflow (prompt, re-link, remove ref, etc.).
  • Sweep reporting: runDbSweep may report unreferenced entries by origin so a UI can surface candidates, but it does not delete FileEntry rows based on dangling state or ref count.

Rationale:

  • External paths are volatile (USB/NAS/network mounts, permission changes, moved files). A cached or freshly observed 'missing' state is still not sufficient authority to delete a user-visible library record.
  • Automatic deletion would make file rows disappear without a visible initiating action, which is surprising even when refs = 0.
  • Explicit removal keeps product copy accurate: external-entry deletion is "Remove from library" and never claims to delete the user's physical file.

Consequences:

  • No persisted "missing since" timestamp or time-based cleanup query.
  • No cleanup-verification bypass around DanglingCache TTL.
  • No cleanup-specific observability event.
  • No ('external', 'missing', 0) automatic deletion branch keyed on dangling state. Layer 3's reporting sub-path remains zero-ref reporting for manual entries; delete_when_unreferenced entries are instead reclaimed by the policy-driven cleanup pass (file-entry-cleanup.md), independent of dangling state.

8. DirectoryWatcher

8.1 Positioning

DirectoryWatcher is a non-lifecycle general FS primitive (not a service), available for business modules to new themselves. It is merely a chokidar wrapper and binds no business semantics.

Placed in src/main/services/file/watcher/, as a dedicated submodule of the file module distinct from the pure FS primitives at @main/utils/file/*. Rationale for the split:

Aspect@main/utils/file/* primitiveswatcher/
ParadigmPure functions (stateless)Stateful class
LifecycleNone (completes upon call)Has one (start → running → dispose)
Resource holdingNoneFSWatcher instance + pending queues + timers
Consumption contractconst x = await read(path)const w = new DirectoryWatcher(...); ... w.dispose()

Grouping a stateful class with pure-function primitives would break the primitives' stateless contract. This mirrors the layering between Node.js official fs.readFile (function) and fs.watch returning an FSWatcher instance (class): functionally related but fundamentally different consumption shapes, so they live in separate submodules.

8.2 API

Shipped surface mirrors src/main/services/file/watcher/index.ts — a single onEvent(listener) subscriber over a normalized event union. Earlier drafts proposed seven separate event channels (onAdd / onAddDir / onUnlink / onUnlinkDir / onRename / onReady / onError) with file vs directory split and built-in rename detection; the watcher module ships a flat union instead because no current consumer needs the dir-event split, and rename detection is deferred to the same change that lands the first onRename consumer (see §8.3).

typescript
export type WatcherEvent =
  | { readonly kind: 'add'; readonly path: AbsoluteFilePath }
  | { readonly kind: 'addDir'; readonly path: AbsoluteFilePath }
  | { readonly kind: 'unlink'; readonly path: AbsoluteFilePath }
  | { readonly kind: 'unlinkDir'; readonly path: AbsoluteFilePath }
  | { readonly kind: 'change'; readonly path: AbsoluteFilePath }
  | { readonly kind: 'ready' }
  | { readonly kind: 'error'; readonly error: Error }

export type WatcherListener = (event: WatcherEvent) => void

export interface DirectoryWatcher {
  /** Subscribe to normalized FS events. Returns an unsubscribe function. */
  onEvent(listener: WatcherListener): () => void
  /** Stop watching and release OS-level resources. Idempotent. */
  close(): Promise<void>
}

export interface CreateDirectoryWatcherOptions {
  /** Recurse into subdirectories. Default: true. */
  readonly recursive?: boolean
  /** Custom ignore predicate. Built-in OS-junk ignores always apply. */
  readonly ignore?: (path: AbsoluteFilePath) => boolean
  /** Stability window for `awaitWriteFinish` (ms). Default: 200. Set to 0 to disable. */
  readonly stabilityThresholdMs?: number
}

export function createDirectoryWatcher(
  path: AbsoluteFilePath,
  opts?: CreateDirectoryWatcherOptions
): Promise<DirectoryWatcher>

Adding new event kinds: the flat union is additive — extending WatcherEvent with a new kind is non-breaking to existing subscribers (they observe and skip), so the dir-split / rename channels can be reintroduced without a watcher-rewrite if a real consumer surfaces.

8.3 Rename Detection Semantics

When enabled, unlink/add events are delayed windowMs to attempt pairing as a rename:

  • Successful match → only onRename is emitted (the matched unlink/add are suppressed)
  • No match → after timeout, unlink/add are emitted normally

Key guarantee: when enabled, onUnlink/onAdd and onRename do not fire simultaneously, so consumer semantics are clear.

Platform precision:

  • Unix (macOS/Linux): prefers inode matching, falls back to size
  • Windows: size only (NTFS ino is unstable), precision degraded and documented as acceptable

Only file rename is handled. Directory renames are not specially recognized; consumers combine sub-file events themselves.

8.4 Built-in Ignore Rules

OS garbage files are ignored by default (not disable-able):

  • { basename: '.DS_Store' }
  • { basename: '.localized' }
  • { basename: 'Thumbs.db' }
  • { basename: 'desktop.ini' }

Consumers may append ignored; merged after the default rules.

8.5 Usage Pattern

Business modules new + dispose themselves as needed:

typescript
// Illustrative (non-file_module implementation)
const watcher = new DirectoryWatcher({
  path: source.basePath,
  renameDetection: { enabled: true }
})
watcher.onAdd(...)
watcher.onRename(...)
await watcher.start()
// ...
watcher.dispose()

file_module starts no watcher instances. Whether to monitor external directories is the business module's decision.


9. AI SDK Integration (FileUploadService) — Deferred

⚠️ This section is a design record; the corresponding implementation is deferred. Vercel AI SDK's Files Upload API (FilesV4, SharedV4ProviderReference) is still pre-release, and the corresponding dependency is unstable. FileUploadService, the file_upload table, and related IPC methods are all deferred to a separate PR after the SDK reaches stable. This section preserves the design intent for direct landing in the future.

9.1 Motivation

Cherry needs to integrate with the Vercel AI SDK's file upload API. The SDK's SharedV4ProviderReference models "the same logical file may be uploaded to N providers, each with its own fileId".

When it lands, a dedicated file_upload table tracks these uploads, decoupled from fileEntry.

9.2 Schema

sql
CREATE TABLE file_upload (
  id              TEXT PRIMARY KEY,
  file_entry_id   TEXT NOT NULL REFERENCES file_entry(id) ON DELETE CASCADE,
  provider        TEXT NOT NULL,
  remote_id       TEXT NOT NULL,
  content_version TEXT NOT NULL,   -- XXH3-64 at upload time
  uploaded_at     INTEGER NOT NULL,
  expires_at      INTEGER,
  status          TEXT NOT NULL,   -- 'active' | 'expired' | 'failed'
  metadata        TEXT,             -- JSON, provider-specific
  UNIQUE(file_entry_id, provider)
);

9.3 Service API

typescript
interface IFileUploadService {
  ensureUploaded(fileEntryId: FileEntryId, provider: string): Promise<FileUpload>
  buildProviderReference(fileEntryId: FileEntryId): Promise<SharedV4ProviderReference>
  invalidate(fileEntryId: FileEntryId): Promise<void>
}

ensureUploaded logic:

  1. Look up (entryId, provider) in file_upload
  2. Compute current contentHash (internal may trust versionCache; external must recompute)
  3. If contentVersion matches + not expired → reuse
  4. Otherwise: read content → call provider.files().uploadFile() → upsert file_upload

buildProviderReference assembles all active uploads into Record<provider, remoteId>.

9.4 Invalidation and Re-upload

  • Content change (triggered by FileEntry write) → mark all file_upload as stale (or delete)
  • Provider expiry (expires_at < now) → treat as missing on next use; re-upload
  • Manually deleted on the provider side → upstream error on send, catch → invalidate + re-upload

10. Orphan Sweep (scheduled FS pass + on-demand report)

10.1 Positioning

The FS-level pass (§10) runs unattended from FileManager.fileSweepTick — the same idle-gated tick as the entry cleanup, concurrently with it, behind a 7-day floor. That is what reclaims orphan blobs in production. The DB-level report pass (§7 Layer 3) has no scheduled trigger and runs only inside the runSweep umbrella, which is reachable solely via the File_RunSweep IPC channel — a channel with no renderer caller today. FileManager exposes a single runSweep() maintenance method: it first runs the entry-cleanup pass (auto-run separately on init/interval — see file-entry-cleanup.md §5), then runs the FS-level pass and the DB-level report pass concurrently, folding the cleanup pass's own summary into counts.entryCleanup, and returns a single OrphanReport once all three settle. The FS and DB passes each begin with a hasPendingRestore() guard (src/main/data/db/restore/restoreJournal.ts): while a staged backup restore awaits promotion, the sweep stands aside with outcome: 'aborted', abortReason: 'pending-restore' — a staged restore's blobs are on disk but not yet referenced by the live DB, which is exactly what the sweep would otherwise reclaim. (No user-facing UI calls runSweep — the entry cleanup it wraps is silent, and the cleanup mechanism has no user surface; see file-entry-cleanup.md's Decision note.)

typescript
protected override async onInit(): Promise<void> {
  // DanglingCache reverse index built from DB before any IPC accepts
  // a dangling query, so a renderer cannot race the first call.
  await this.deps.danglingCache.initFromDb()
  // IPC handlers, including `File_RunSweep`, are registered here.
  this.registerIpcHandlers()
  // Entry-cleanup pass auto-runs here (previous-session backlog) and on a
  // 30min idle-gated interval — independently of `runSweep`. See
  // file-entry-cleanup.md §5.5.
  void this.runEntryCleanup()
  this.registerInterval(() => this.entryCleanupTick(), FileManager.CLEANUP_INTERVAL_MS)
}

async runSweep(): Promise<OrphanReport> {
  // Three passes, cleanup first:
  //   1. Entry-cleanup pass (file-entry-cleanup.md §5): reclaims zero-ref
  //      `delete_when_unreferenced` entries. No volume abort (spec §5.3).
  //      Runs first so the DB report below doesn't re-report entries it
  //      just reclaimed.
  //   2. FS-level file sweep (§10): scan {userData}/Data/Files/* for
  //      orphans not present in the file_entry snapshot.
  //   3. DB-level entry report (§7 Layer 3): report unreferenced
  //      `manual` entries.
  // Passes 2/3 settle independently with their own error capture. A DB
  // failure dominates as `failed`; FS-side partial/aborted/failed outcomes
  // degrade the umbrella report to `partial` via `fsSweepIssue`. The
  // cleanup pass's own outcome rides in `counts.entryCleanup` and never
  // changes the umbrella `outcome`. Exception: while a staged backup restore
  // is pending promotion, BOTH passes stand aside up front and the umbrella
  // returns `aborted` (abortReason: 'pending-restore') verbatim — expected
  // behavior, never disguised as a degraded `partial` run.
}

Rationale for the split (scheduled FS pass, on-demand report):

  • The FS pass performs reclamation, and the entry-cleanup pass manufactures its input on every run (unlinkFailures, plus crash residue between row-delete and unlink). Reclamation cannot wait on a caller that does not exist, so it rides the idle tick — but behind a coarse floor, because an orphan blob costs disk, never correctness.
  • The DB pass only reports. A report with no consumer has nothing to do, so it stays on demand; when a cleanup UI appears it invokes runSweep and gets both halves plus the entry-cleanup summary in one OrphanReport.
  • No persistent state machine. Each invocation runs end-to-end and returns its own report; FileManager no longer holds lastDbSweepReport / lastDbSweepRanAt. UIs that want "last scan" timing should hold the previously-returned OrphanReport.lastRunAt themselves.

A note on initVersionCache: an earlier draft of this section bundled a synchronous initVersionCache() call into onInit. It didn't survive implementation — version cache is per-FileManager-instance and constructs at field-init time (no boot step), so there is no separate init call to make. registerIpcHandlers() did survive and is the convention used across lifecycle services for the same reason it surfaces in lifecycle-migration-guide.md: keeps onInit a narrow init→register sequence and gives a single spot for Phase 2 channels to land.

10.2 Scan Strategy

The sweep uses a single snapshot of file_entry.id taken at sweep start, not per-file DB lookups:

sql
SELECT id FROM file_entry  -- one query, held in an in-memory Set for the duration of the sweep

It then diffs readdir({userData}/Data/Files/) against the snapshot in memory. Chosen properties:

  • Simple and fast: one SQL round-trip vs N round-trips; for <10k internal entries the cost is sub-10ms.
  • Race window is bounded and explicit: entries inserted after the snapshot but before the sweep reaches their file appear as "not in DB". The mtime > 5min filter in §10.3 is the only thing that protects them — the snapshot strategy makes this reliance explicit rather than hiding it behind DB isolation levels.

Implementations MUST NOT silently switch to per-file DB lookups — the snapshot strategy, the mtime > 5min filter (§10.3), and the safety threshold (§10.4) are co-designed; any change to one requires re-evaluating the other two.

10.3 Heuristic Rationale & FS Prerequisites

The mtime > 5min filter is an engineering heuristic, not a formal guarantee. It is sufficient under the following assumptions:

  • createInternalEntry's "FS write UUID file → DB insert" window is << 5min in practice (microseconds to seconds)
  • atomicWriteFile's "open tmp → write → fsync → rename" window is << 5min, with streaming writes refreshing mtime on each write(2)
  • userData resides on a local POSIX-compliant filesystem with sub-minute mtime resolution (APFS / ext4 / NTFS local)
  • System clock is not discontinuously advanced by > 5min while a write is in flight

Known breakage modes (documented, accepted as residual risk):

ScenarioConsequence
Very slow write (huge file + slow disk/fsync) exceeds 5min between FS write and DB insertNewly-written internal file may be unlinked on the next sweep
Process frozen / suspended > 5min mid-write; then a subsequent sweep runsSame as above
System clock jumps forward > 5min after file creationRecent residue gets mis-aged; usually harmless — those files were orphans anyway
System clock jumps backwardFilter becomes permissive (now < mtime disqualifies the file); cleanup delayed to the next sweep run (safe)
userData on FAT32 / exFAT / SMB / NFS (second-precision or offset-prone mtime)Filter still works at coarse granularity; extreme clock skew between client and server can mis-age files
userData on tmpfs / CoW FS with unusual mtime semanticsOut of contract; user responsibility

The sweep does not defend against deliberately hostile mtime manipulation. These are known limits, not bugs.

10.4 Safety Threshold & Abort

To bound the blast radius of application-layer regressions (e.g. a faulty migration that truncates file_entry, a regression in the sweep predicate, a developer running the app against a stale DB), the sweep MUST evaluate its plan before executing and abort on suspicious scale:

typescript
interface SweepPlan {
  orphanFiles: { path: string; bytes: number }[]
  tmpFiles:    { path: string; bytes: number }[]
}

function shouldAbort(
  plan: SweepPlan,
  totalFilesOnDisk: number,
  totalBytesOnDisk: number
): boolean {
  const toDelete      = plan.orphanFiles.length + plan.tmpFiles.length
  const toDeleteBytes = sumBytes(plan.orphanFiles) + sumBytes(plan.tmpFiles)

  // Normal crash residue: always proceed.
  if (toDelete < 20 && toDeleteBytes < 10 * 1024 * 1024) return false

  // Otherwise check proportion. If the sweep would erase a large fraction of
  // the on-disk UUID population, something upstream is wrong — refuse and
  // warn; the next sweep run re-evaluates after the bug is fixed.
  const countFraction = toDelete      / Math.max(1, totalFilesOnDisk)
  const byteFraction  = toDeleteBytes / Math.max(1, totalBytesOnDisk)
  return countFraction > 0.5 || byteFraction > 0.5
}

Contract:

  • On abort, no files are unlinked.
  • Emits a warn-level structured log (see §10.5) so developers / on-call can diagnose.
  • The service remains available — abort is a controlled outcome, not a failure; no Error is thrown into the .catch() handler.
  • The next sweep run re-evaluates the plan after the upstream issue is resolved.

Scope note: the threshold defends against internal bugs, not user-side manipulation of {userData}/Data/Files/. Users are not expected or encouraged to edit the storage directory (all file operations should go through the in-app entry system). The threshold's job is to ensure "nothing Cherry itself does, internally, silently deletes the bulk of a user's library".

10.5 Observability

Every sweep run emits one structured log record through loggerServiceinfo on normal completion, warn on partial / aborted outcomes, error on unexpected failure caught by .catch():

typescript
{
  event: 'orphan-file-sweep',          // disambiguates from the DB-side 'orphan-sweep' (§7 Layer 3)
  outcome: 'completed' | 'partial' | 'aborted' | 'failed',
  entriesInDb: number,
  direntsScanned: number,              // total readdir entries (informational)
  filesOnDisk: number,                 // UUID files + tmp residue candidates only
  bytesOnDisk: number,                 // bytes of candidates (drives the abort fraction math)
  plannedDeleteCount: number,
  plannedDeleteBytes: number,
  actualDeleteCount: number,           // 0 on aborted / failed
  actualDeleteBytes: number,
  oldestDeletedMtime?: number,         // ms epoch of the oldest file unlinked this run
  statFailedCount: number,             // non-ENOENT stat errors during planning
  scanDurationMs: number,
  // outcome-specific fields (discriminated union):
  // 'partial':  failedDeleteCount: number, failedSamples: readonly string[]  (capped at 5)
  // 'aborted':  abortReason: 'count-fraction' | 'byte-fraction' | 'pending-restore'
  // 'failed':   errorMessage: string
}

The DB-side sweep emits a parallel record under event: 'orphan-sweep'. Its current outcomes are completed, aborted (abortReason: 'pending-restore' — the same stand-aside as the FS pass), or failed: it reports manual entries with zero refs. The shared partial wire branch remains for compatibility, but there is no generic per-source checker pass.

The entry-cleanup pass (§7.1, file-entry-cleanup.md §5.6) emits a third, independent record under event: 'file-entry-cleanup'info on completed and on 'skipped' (the pending-staged-restore stand-aside), error on failed (it has no aborted outcome; the volume abort was removed, spec §5.3) — covering candidate/deleted/gonePinned/failed counts and skip/unlink-failure breakdowns for the delete_when_unreferenced reclaim path. It fires on its own triggers (init, idle-gated interval) in addition to running as the first of runSweep's three passes (§10.1).

These three records are the single source of truth for post-hoc diagnosis. No separate metrics pipeline is needed — at most three records per user-triggered sweep run is a trivial volume for log aggregation.

10.6 DanglingCache Initialization

The reverse index of DanglingCache (Map<path, Set<entryId>>) is built via a single synchronous DB query:

sql
SELECT id, externalPath FROM file_entry
WHERE origin = 'external' AND deletedAt IS NULL

No stat performed—the state field (Map<entryId, DanglingState>) is initially empty; lazy stat on query (see §11).

10.7 Why There Is No Dangling Probe

The old version batch-stat'd all external entries at startup to build the dangling set. The new version cut this step:

  1. Dangling is a pull-only IPC query (getDanglingState / batchGetDanglingStates); most query scenarios don't need it, so it's never computed eagerly
  2. Lazy + Promise.all is fast enough: on the first dangling query, N stats run in parallel, typically <100ms
  3. Watcher-covered paths have zero IOwhere coverage happens to exist. In practice, watcher coverage is emergent and typically sparse (see §11.1): only business modules with their own directory-monitoring needs (NoteService watching the notes directory, etc.) create watchers. Ad-hoc user-@-ed external paths — the main source of external entries — are almost never covered by any watcher. For those paths, the TTL-based cache (§11.6) and ops observations are the dominant freshness mechanisms, not watchers. Do not read this bullet as "watchers cover most paths"; they don't.

10.8 Concurrency Safety

Concurrency scenarioResult
createInternalEntry creates a new internal file during sweepThe mtime > 5min filter (§10.3) prevents the new file from being mistakenly deleted; the snapshot strategy (§10.2) makes this reliance explicit
FileManager.read/write on existing entries during sweepNo mutual exclusion; read/write follow different code paths and are unaffected
Upstream bug causes bulk deletion planSafety threshold (§10.4) aborts the sweep without unlinking
app exits during sweepNo persistent side effect; the next scheduled fileSweepTick re-derives everything

10.9 Crash Consistency

file_module's crash window is very narrow:

OperationOrderCrash mid-operationRecovery
createInternalEntryFS write UUID file → DB insertOrphan fileOrphan sweep
write (internal)prepare tmp → mark hash null → atomic rename → finalize size/hashCrash before rename preserves old bytes; crash after rename leaves new bytes + null hashStartup reconciliation rebuilds null metadata
trash / restore / renameDB onlyNoneNone
permanentDelete (internal)DB delete → best-effort FS unlinkCrash after DB delete leaves an orphan blobOrphan sweep
copy (internal)FS copy → DB insertOrphan fileOrphan sweep
ensureExternalEntryDB insert / reuse (doesn't touch user file)NoneNone
permanentDelete (external)DB delete onlyNone — user's file at externalPath is never modifiedNone needed

No WAL / pending_fs_ops table needed. Orphan sweep covers the internal crash residue; the external side naturally doesn't need it (delete failure just leaves it on disk).


11. DanglingCache (External Presence Tracker)

11.1 Positioning

DanglingCache is a singleton in file_module (not a lifecycle service) that maintains the "latest known on-disk state" for external entries.

typescript
// src/main/services/file/danglingCache.ts
export const danglingCache = new DanglingCache()

Role:

  • Provides a fast query interface for File IPC getDanglingState / batchGetDanglingStates. Cache hit within TTL (§11.2, 30 min) returns synchronously; TTL-expired or cold miss runs a single fs.stat and refreshes the cache. DataApi never reads this cache — DataApi is pure SQL.
  • Consumes add/unlink/rename events from watchers that happen to cover an entry's path (auto-wired via the factory). Watcher coverage is sparse in practice: it exists only where a business module has created a watcher for its own reasons (NoteService watching the notes directory, a future Obsidian-like module, etc.). Arbitrary user-@-ed external paths — which are the bulk of external entries — are typically NOT watcher-covered. Do not assume event-driven updates as the primary freshness mechanism; the TTL (§11.6) is.
  • Consumes observation results from FileManager's own ops (read/stat/write success/failure) — updates cache opportunistically whenever a mutation or lookup touches a path.

11.2 State Model

DanglingCache exposes lazy, query-driven presence checks only. There is no cleanup-only recheck path because dangling state never drives deletion — manual entries are not auto-deleted (§7.2), and the cleanup pass for delete_when_unreferenced entries keys on ref count and grace window, not presence (file-entry-cleanup.md). If a future explicit user workflow needs a strict re-stat escape hatch, add it with that concrete caller and document the user-visible action.

typescript
type DanglingState = 'present' | 'missing' | 'unknown'

interface CachedState {
  state: 'present' | 'missing'
  /** ms epoch of last observation — drives TTL expiry in `check` */
  observedAt: number
  /** Where this observation came from (for diagnostics / log context). */
  source: 'watcher' | 'ops' | 'stat'
}

class DanglingCache {
  /** TTL for cached observations. 30min — external file path moves are rare,
   *  and freshness is bounded by TTL for any entry the user actually queries. */
  private static readonly TTL_MS = 30 * 60 * 1000

  private byEntryId: Map<FileEntryId, CachedState>
  private pathToEntryIds: Map<string, Set<FileEntryId>>  // reverse index

  /** Public event: fires on every genuine state transition (see §11.7). */
  readonly onDanglingStateChanged: Event<DanglingStateChangedEvent>

  // Query (TTL-aware; re-stats when cache entry is stale)
  async check(entry: FileEntry): Promise<DanglingState>

  // Event entry (for watcher factory + FileManager ops) — resets observedAt
  onFsEvent(path: string, state: 'present' | 'missing'): void

  // Index maintenance (for FileManager entry CRUD)
  addEntry(entryId: FileEntryId, externalPath: string): void
  removeEntry(entryId: FileEntryId, externalPath: string): void

  // Startup init
  initFromDb(): void
}

Query strategy for check — lazy TTL expiration:

typescript
async check(entry: FileEntry): Promise<DanglingState> {
  if (entry.origin === 'internal') return 'present'

  // L1: cache hit AND still within TTL → return cached
  const cached = this.byEntryId.get(entry.id)
  if (cached && Date.now() - cached.observedAt < DanglingCache.TTL_MS) {
    return cached.state
  }

  // L2: no cache OR TTL expired → re-stat and update
  return this.doStatAndUpdate(entry, 'stat')
}

private async doStatAndUpdate(
  entry: FileEntry,
  source: CachedState['source']
): Promise<DanglingState> {
  const state = await statToState(entry.externalPath!)
  const prev = this.byEntryId.get(entry.id)
  this.byEntryId.set(entry.id, { state, observedAt: Date.now(), source })
  if (!prev || prev.state !== state) {
    this._onDanglingStateChanged.fire({ id: entry.id, state })
  }
  return state
}

Key design points:

  • Lazy expiration only, no periodic background sweep. FS IO cost scales with query frequency, not total entry count — heavy-user populations (10k+ external entries) consume zero IO when no UI is querying.
  • Watcher events / ops observations reset observedAt to Date.now() — a path with active watcher coverage stays fresh indefinitely and never triggers TTL-driven re-stat.
  • TTL = 30 min: external file path moves are rare in practice (files accumulate, rarely move); a 30-minute worst-case staleness window is acceptable for background UI state, while keeping TTL ≫ React Query's renderer-side staleTime ≤ 5min means most renderer refetches hit cache (desired: the cache adds value).
  • No background or cleanup-only recheck path: presence is refreshed only when a caller queries or an observed operation/watch event supplies a new state. This keeps IO proportional to use and avoids hidden deletion authority.

11.3 Watcher Auto-Wiring

Business modules need not be directly aware of DanglingCache. All watchers must be created via the createDirectoryWatcher() factory; the factory subscribes to its own event stream and mirrors presence transitions into DanglingCache before re-emitting the raw event to external subscribers:

  • add → cache marks present
  • unlink → cache marks missing
  • change → cache untouched (file is still present; mtime drift is not tracked here)

Because externalPath is stored byte-faithful (see §1.2 "Stored form"), the reverse index is keyed by that exact stored form and the watcher matches chokidar events by raw byte equality — no NFC normalization on either leg. (The NFC step that used to sit in the watcher existed only to bridge to the old NFC-canonical keys; it is removed together with them — see §1.2 "Rejected: Unicode (NFC) normalization".) The path forwarded to subscribers is likewise the raw OS form chokidar saw.

check() is the correctness baseline: it stats entry.externalPath (byte-faithful) directly, so an entry's dangling state is always eventually correct regardless of whether any watcher event matched — the watcher leg is an eager-update latency optimization on top of it, never a correctness dependency. On Linux the raw event byte-matches the stored key by construction; on macOS/Windows it matches when the path source and chokidar agree on Unicode form. If they ever diverge (unverified — same open question as §1.2 "Rejected"), the only effect is that a watcher event misses and the badge stays stale until the next check() — benign and self-healing. The platform-conditional fold that could bridge such a divergence, if ever observed, belongs on the reverse-index comparison key (process.platform-aware: NFC on darwin/win32, identity on linux) — never on the stored form.

Note: watcher rename events do not auto-update an external entry's externalPath—Cherry does not track external rename. After a rename, the original entry goes dangling; the user must re-@ to establish a new reference.

11.4 Reverse Index Maintenance

Timing for changes to pathToEntryIds (fully self-governed inside file_module, no DB-FS sync):

EventAction
Startup initFromDb()SELECT id, externalPath FROM file_entry WHERE origin='external' AND deletedAt IS NULL → batch add
ensureExternalEntry creates newaddEntry(id, path)
ensureExternalEntry reuses (upsert hit)No change (path already indexed)
permanentDelete(external)removeEntry(id, path)
rename(external) (explicit user action)removeEntry(id, oldPath) + addEntry(id, newPath)

External entries cannot be trashed (fe_external_no_delete CHECK enforces this at the schema level; trash / restore throw at the entry layer before reaching the reverse-index update). Earlier drafts listed restore(external) and trash(external) rows here — they were dead branches and have been removed.

11.5 Handler-Side Parallelization

The File IPC batchGetDanglingStates handler fans out over the requested ids in parallel:

typescript
async function batchGetDanglingStates(ids: FileEntryId[]): Promise<Record<FileEntryId, DanglingState>> {
  const entries = await fileEntryService.batchGetById(ids)
  const pairs = await Promise.all(
    entries.map(async (e) => [e.id, await danglingCache.check(e)] as const)
  )
  return Object.fromEntries(pairs)
}
  • Cache-hit entries return synchronously (microtask)
  • Only cache-miss external entries go through stat, all in parallel
  • 1000 entries cold-start typically <100ms (libuv threadpool parallel stat)
  • Handler lives behind File IPC, not DataApi — the FS side effect is contained to the IPC channel where side effects are expected

11.6 State Invalidation Policy

TTL-based lazy expiration. A cached entry is considered fresh while now - observedAt < TTL_MS (30 min); once stale, the next check() call re-stats and updates the cache. observedAt is refreshed by any of:

  • Watcher add/unlink/rename events (where coverage exists — see §11.1 caveat)
  • Observation side effects of FileManager ops (stat ENOENT → missing; create / ensureExternal / rename / write success → explicit 'present' commit through onFsEvent(..., 'ops')). Read / hash / getMetadata / getVersion do not flip the cache to 'present' on success — they only commit 'missing' on ENOENT through the observeExternalAccess chokepoint. The watcher-led design deliberately keeps presence learning out of the passive-read path; see internal/observe.ts for the contract.
  • Cold-path or TTL-driven fs.stat from check() / getMetadata / getDanglingState

Freshness guarantee: for any path the caller queries, cached state is never older than the TTL. Paths that are never queried may stay stale indefinitely — but by construction, no consumer is looking at them, so the staleness has no user-visible impact.

Why no background sweep: a periodic background re-validation across all cached entries was considered and rejected. See §12 Key Design Decisions. The short version: FS IO cost would scale with total entry count instead of query frequency, and deletion is never driven by dangling state (manual entries require explicit action; auto-policy cleanup keys on ref count), so stale presence state should be corrected at use/query time rather than by a hidden global scanner.

Known residual case — stale 'present' with refs > 0: if an external file is deleted outside Cherry, without any watcher or ops observation to signal it, and no UI ever queries getDanglingState for that entry, the cache stays 'present' past TTL boundaries (first query after TTL will re-stat and fix). Business services that depend on referenced files MUST re-validate at use time (read will surface ENOENT anyway); DanglingCache is a UI/presence helper, not a correctness boundary.

11.7 Reactivity — Event Emission (deferred)

DanglingCache exposes onDanglingStateChanged: Event<DanglingStateChangedEvent> fired on every genuine state transition (watcher event, cold-path fs.stat observation after a cache miss, explicit ops observation):

typescript
export interface DanglingStateChangedEvent {
  id: FileEntryId
  state: 'present' | 'missing'
  // Note: 'unknown' is never broadcast — transitions FROM unknown TO concrete
  // fire; 'unknown' itself is the default pre-observation state, not a signal.
}

FileManager subscribes in onInit (see §1.6.8) and fans the event out to all renderer windows via the shared file-manager-event IPC channel, closing the "main-side FS observation → renderer React Query invalidation" loop. The public queryKey invalidation target is ['fileManager', 'dangling', id]; see architecture.md §3.6 for the full dispatch table.

Emission rules:

  • Transitions only — if onFsEvent(path, 'present') arrives and the cached state is already 'present', no emit. Prevents broadcast floods on watcher-chatty filesystems.
  • Fan-out via reverse index — when multiple entries share the same externalPath (allowed by schema — one row per path per non-trashed state, but historical / edge cases may produce more), §11.4's reverse index yields all affected ids. Emit one event per id. Renderer invalidates ['fileManager', 'dangling', id] per id.
  • Internal-origin entries never fire — they are always 'present' by construction (§3.3); DanglingCache never tracks them.

Staleness backstop retained: the staleTime ≤ 5min contract from architecture.md §4.1.1 still applies. Events accelerate refresh; a lost event is bounded by React Query's natural refetch cadence. DanglingCache's emission is therefore an optimization over pure pull, not a replacement for the pull path.

11.8 Observability (deferred)

DanglingCache emits a structured info-level log record at a fixed cadence (every 10 minutes, driven by a simple timer in onInit) summarizing its recent activity. This mirrors the orphan sweep observability contract (§10.5) — one periodic record plus opportunistic warn / error on anomalies, no separate metrics pipeline.

typescript
{
  event: 'dangling-cache-snapshot',
  cachedEntries: number,           // total entries currently in byEntryId
  pathIndexSize: number,           // total keys in pathToEntryIds
  // counters since the last snapshot record (reset each emit)
  checkCalls: number,              // total check() invocations
  checkCacheHits: number,          // returned cached within TTL
  checkTtlExpiredReStats: number,  // re-stat triggered by TTL
  checkColdStats: number,          // re-stat triggered by cache miss
  watcherEvents: number,           // onFsEvent calls from watcher factory
  opsObservations: number,         // onFsEvent calls from FileManager ops
  transitionsFired: number,        // onDanglingStateChanged fires
  statErrors: number,              // fs.stat threw (permission, I/O error, etc.)
  windowMs: number,                // interval this snapshot covers (≈ 600_000)
}

Emission cadence: every 10 minutes while the service is active. Upon onStop, one final snapshot flushes any outstanding counters. Snapshot volume at steady state is 6 records/hour × 24 = 144 records/day through loggerService — trivial for log aggregation even on long-running installs.

Anomaly triggers (emitted out-of-band at warn level, independent of the snapshot cadence):

  • statErrors / checkCalls > 0.1 sustained across two consecutive snapshot windows → likely a systemic FS issue (unmounted drive, permission regression)
  • cachedEntries > 50_000 → memory-budget anomaly; suggests either a runaway caller or a bug in removeEntry cleanup
  • transitionsFired > 1000 within one 10-minute window → likely a watcher feedback loop or mass unmount event

These thresholds are heuristic starting points — tune based on real-world telemetry once available. IPC latency is not instrumented here; loggerService is a log pipeline, not a metrics system, and Cherry has no telemetry backend. Per-IPC latency concerns should be diagnosed ad-hoc via performance.now() in the affected handler during investigation, not baked into a permanent counter.


12. Key Design Decisions

DecisionConclusionCore rationale
Tree vs flatFlatFileEntry manages "user-submitted independent files"; directory organization is not a file_module responsibility
Mount abstractionRemovedAll internal files live flat under {userData}/Data/Files/ (via the feature.files.data path key); external is reached directly via externalPath; no mount needed
Origin two-stateinternal/externalExpress "Cherry-owned" and "user-owned, Cherry-referenced" respectively; clear semantics
External read/write permissionsExplicit user ops may change; Cherry doesn't auto-changeVS Code-style behavior model—change when told to; don't modify behind the scenes
External operation symmetrywrite/rename/permanentDelete all delegate to the FS primitives and take effect; trash/restore touch DB onlySoft delete preserves reversibility (doesn't touch FS); hard delete is the terminal action (really deletes FS)
External identityexternalPath unique(where not trashed)At most one active entry at a time for the same path; ensureExternalEntry upserts by path
Cherry tracks external renameNot trackedBest-effort semantics; external rename → dangling → user re-@
Snapshot vs realtime statExternal row stores only identity + stable projections (name / ext from externalPath); live size / mtime via getMetadata on demandEliminates stale-snapshot bug class at the type level; cost of the extra fs.stat is explicit at the call site instead of hidden behind a DB field
Dangling state carrierIn-memory singleton DanglingCacheNot in DB (avoids bidirectional DB-FS sync); three states present/missing/unknown; TTL-based lazy expiration (§11.6, 30 min); refreshed on query / FS observation / watcher; no periodic background sweep — IO cost scales with query frequency, not entry count
Dangling exposure methodFile IPC getDanglingState / batchGetDanglingStates (never DataApi)DataApi is pure SQL; FS probe lives in IPC where side effects are expected; zero cost by default; parallel stat on demand
Watcher → DanglingCache wiringFactory auto-wiresBusiness modules unaware of DanglingCache; a single watcher instance serves business events + dangling tracking
Content hash algorithmXXH3-64 via native @node-rs/xxhash, stored as {algo}:{hex}One implementation serves persisted dedup detection and on-demand OCC comparison. The native package avoids the previous WASM throughput ceiling and provides incremental XXH3-64 hashing; its XXH3-128 API is one-shot only, which would require buffering streamed files. The 64-bit value is a candidate signal, never an identity: collisions only produce a candidate that the consumer must verify, and the algorithm tag preserves an incremental upgrade path.
Does write carry versionSplit into write / writeIfUnchangedForce the caller to explicitly choose; avoid silent degradation to blind write when version is forgotten
Atomic write fsyncOn by defaultCorrectness guarantee takes precedence over performance; Cherry is not a high-throughput scenario
Trash modeldeletedAt timestampparentId unchanged; naturally supports expiry; no system_trash entries
pending_fs_opsRemovedAfter extreme simplification, orphan sweep suffices to cover crashes
Startup dangling probeRemovedChanged to lazy + Promise.all; stat only when an IPC caller explicitly requests dangling state
Is Watcher a lifecycle serviceNoDirectoryWatcher is a primitive; business modules new it via the factory; file_module doesn't actively watch
Directory import / bidirectional syncMoved out of file_moduleBusiness modules (Knowledge, etc.) implement this with DirectoryWatcher + their own mapping tables
AI SDK upload cacheStandalone file_upload table (deferred)Decoupled from mount / remote; naturally aligns with SharedV4ProviderReference
NotesFile tree is an independent domain, not mirrored to FileEntryIf other modules need to reference Notes files, they use the origin of their choice via the corresponding path
CacheService integration for DanglingCache / versionCacheNot integrated; both stay bespokeCacheService (src/main/data/CacheService.ts) is a general TTL KV + cross-window sync primitive. DanglingCache needs a path → Set<entryId> reverse index (§11.4), transition-aware event emission (§11.7 — fire only on genuine state change by comparing old vs new), and observedAt-based "TTL expired → re-stat then update" semantics (§11.6 — CacheService's TTL is "expired → deleted", which would destroy the prev-state comparison needed for transition detection). versionCache needs size-bounded LRU (§4.4), not TTL — a fundamentally different eviction policy; and lives as a per-FileManager-instance field for test isolation, not as a BeforeReady singleton. Wrapping either in CacheService would flatten the value schema, bolt on the secondary structures separately, and bypass the TTL layer — no logic shed, only domain expression lost. CacheService remains the right tool for future scenarios that genuinely match "simple per-id TTL cache" or "cross-window cache" shape (e.g. a short-lived batchGetMetadata result cache, a future FileUploadService provider-upload cache).

13. Adding a New origin Variant — Developer Checklist

The origin field ('internal' | 'external' today) is the single most cross-cutting axis in the file module: its value implicitly drives storage layout, DB constraints, mutability policy, dangling semantics, UX language, and cleanup scope. Adding a new variant (e.g. 'archived', 'shared', 'synced') requires coordinated changes across the layers below. Missing any row silently breaks an invariant — TypeScript catches discriminated-union exhaustiveness but not CHECK constraints, not ad-hoc policy branches, and not documentation tables.

This checklist is the canonical addition procedure. A PR introducing a new origin MUST tick every row it touches (or explicitly justify a skip).

13.1 Type & Schema Layer

LocationChange required
src/shared/data/types/file/fileEntry.tsFileEntryOriginSchemaAdd the new enum value
Same file → new XxxEntrySchemaDefine the row shape for the new variant (which columns are nullable / required / branded)
Same file → FileEntrySchema discriminated unionAdd the new schema as a union member
Same file → any type guard helpers (isInternalEntry, etc.)Add isXxxEntry helper if code needs to narrow

13.2 DB Schema Layer

LocationChange required
src/main/data/db/schemas/file.tsReview every CHECK constraint naming originfe_origin_consistency, fe_size_internal_only, fe_external_no_delete, fe_external_path_unique, etc. — and decide whether the new variant honors / violates / is exempt from each
Drizzle migrationShip the constraint updates in the same migration as the enum expansion. Partial unique indexes on externalPath may need a new branch
Existing rowsNo migration should run for existing rows unless the new variant has a natural subset mapping (unlikely)

13.3 Path Resolution Layer

LocationChange required
src/main/services/file/utils/pathResolver.tsresolvePhysicalPathAdd the new entry.origin branch; decide storage layout
src/shared/utils/file/canonicalize.tscanonicalizeFilePath()If the new variant is path-based and distinct from 'external', decide whether it shares the canonical form or needs its own normalization + brand

13.4 Behavior Policy Matrix

Every ad-hoc if (entry.origin === 'internal') / === 'external' in the codebase is a policy decision that must be re-evaluated. Grep for both and review:

PolicyLocation
Trash-ability (who can soft-delete)trash / restore in FileManager; DB CHECK fe_external_no_delete
Size snapshot storagewrite / writeIfUnchanged internal-DB-update branch; toFileInfo projection
Name / ext as SoT vs projectionrename mutation; toFileInfo projection; FileEntrySchema field docs
DanglingCache participationDanglingCache.check returns 'present' for internal; consider where the new variant falls on the present/missing/unknown axis
permanentDelete semanticsDoes it touch physical files? Just DB? Refer to §6 and architecture.md §3.4
Orphan sweep scope§10 scans origin='internal' UUID files; does the new variant have a sweepable disk presence?
Explicit cleanup semantics§7.2 forbids dangling-state-driven automatic deletion of manual entries (auto-policy entries are reclaimed by the ref-count-driven cleanup pass — file-entry-cleanup.md); decide whether the new origin is preserved, reported, or removable only through an explicit user/caller action
IPC dispatch applicabilityarchitecture.md §3.3 tables per method — does each method make sense for the new variant?

13.5 UX Layer

LocationChange required
architecture.md §3.4 UX labeling convention tableAdd a row for the new origin's permanentDelete user-facing label + confirmation copy
Product-side component copyConcrete button labels / menu items that branch on origin

13.6 Event & Observability Layer

LocationChange required
architecture.md §3.6 event payloadsonEntryRowChanged.origin field value domain expands — TS catches via discriminated-union narrowing in the renderer binding
Observability logsdangling-cache-snapshot and orphan-sweep records may need per-origin breakdowns if the new variant is material to diagnostics

13.7 Documentation Layer

LocationChange required
architecture.md §1.0.1 Semantics of OriginAdd a paragraph describing the new variant
file-manager-architecture.md §1.2 Origin tableAdd the row — physical location, ownership, mutability
file-manager-architecture.md §1.2 Invariants tableAdd the column — name / ext / size / externalPath behavior per-origin
file-manager-architecture.md §12 Key Design DecisionsIf the addition surfaces a notable new trade-off, record it

13.8 Gate

PR description MUST list each ticked row with a one-line justification, and each explicitly-skipped row with a reason. A reviewer MUST spot-check §13.2 (DB CHECK constraints) and §13.4 (ad-hoc policy grep) before approval — these are the two layers where silent bugs most often hide.