FileMigrator

FileMigrator migrates the legacy v1 Dexie files table into the v2 file_entry SQLite table.

Data Sources

Data	Source	File/Path
File metadata	Dexie files table	files.json

The table is streamed via createStreamReader('files') in batches of BATCH_SIZE (default 500) to keep peak memory bounded even for large file collections — 500 is large enough to amortize the per-batch round-trip but small enough that one batch's worth of FileMetadata rows fits comfortably in memory.

Target Tables

• file_entry

Outputs

• file_entry rows — one row per valid source file

No cross-migrator shared state is published: per migration-plan §2.9 the v1 file id is preserved verbatim into v2, so downstream migrators (ChatMigrator, KnowledgeMigrator, …) reference files by the same id they already have without needing a translation map.

Key Transformations

ID Preservation

• The v1 file id is carried into v2 file_entry.id unchanged (no translation, no remap)
• FileEntryIdSchema = z.uuid() accepts both legacy v4 and v2-native v7 ids
• New entries created in v2 still receive a v7 id via uuidPrimaryKeyOrdered(); the column allows both shapes to coexist

Origin Discrimination

Condition	origin	externalPath	size
path starts with {userData}/Data/Files/	internal	null	row.size (≥0)
any other absolute path	external	row.path	null

Ext Normalization

• Legacy v1 ext field may include a leading dot (.pdf, .txt) or be empty
• Leading dot is stripped before writing (pdf, txt)
• Empty / whitespace-only / missing ext → null in file_entry.ext (matches the SafeExtSchema whitespace guard in shared file common.ts so the migrated rows pass the same validation as v2-native writes)

Timestamp Conversion

• created_at (ISO 8601 string) is parsed to ms epoch integer
• Missing / empty created_at → Date.now() silently (valid v1 case)
• Non-empty but unparseable → Date.now() plus a warning recorded against the row id (surfaced through PrepareResult.warnings). Falling back to "now" (not 0) keeps migrated rows sortable next to v2-native rows; the warning is the diagnostic trail for users whose v1 data carried corrupted dates.
• Both createdAt and updatedAt are set to the same parsed value

Name Derivation

• Internal rows: name = origin_name basename without extension (preserves the user-visible filename)
• External rows: name = path basename without extension

Field Mappings

Source (v1 FileMetadata)	Target (file_entry)	Notes
id	id	Preserved verbatim
(derived from path)	origin	internal or external
origin_name / name	name	Basename without ext
ext	ext	Leading dot stripped; empty/whitespace-only → null
size	size	Non-null for internal; null for external
path (external only)	externalPath	null for internal
(always null)	deletedAt	No v1 soft-delete state
created_at	createdAt	ISO → ms epoch; fallback Date.now() + warning on parse failure
created_at	updatedAt	Same as createdAt

Dropped v1 fields: count, tokens, purpose, type, origin_name (stored as-is in name derivation only)

Idempotency

The migrator is safe to re-run. MigrationEngine.verifyAndClearNewTables clears the file association tables and file_entry before each run, so execute() always starts from empty tables. The v1 id is preserved verbatim, so the engine-layer clear is the sole invariant — no onConflict guard or per-row pre-check is needed at the migrator layer.

Validate Behavior

validate() performs:

1. Count check: asserts SELECT count(*) FROM file_entry >= preparedEntries.length
2. Physical file sampling: up to VALIDATE_SAMPLE_LIMIT = 10 internal entries are checked for their physical file at {userData}/Data/Files/{id}.{ext} via fs.existsSync. 10 is small enough to keep validate cheap on large migrations and large enough to catch a systematic "Files directory moved/missing" issue early; per-row I/O is intentionally bounded since the migration's own physical-copy step is the authoritative integrity boundary. Missing physical files go through this.recordWarning (not validation errors) — v1 routinely leaves dangling file_entry rows behind (deleted attachments, interrupted uploads), and the DB row keeps the historical reference even when bytes are gone.

External entries are not sampled in validate (physical files are user-owned and may have moved).

Failure Handling

Issue	Detection	Handling
Malformed row (missing id/path/name)	toFileEntry() returns null	Skipped; skippedCount++; warn logged
Duplicate id in v1 source	seenIds set in prepare()	Second occurrence skipped; warn logged
Insert error (DB constraint, disk full)	Transaction throws	execute() returns success=false with error message
Missing files table	tableExists('files') returns false	Prepare returns success with 0 items and a warning

Implementation Files

• FileMigrator.ts — main migrator class
• __tests__/FileMigrator.test.ts — unit tests