Knowledge Schema Notes (V2)

This document records the current V2 knowledge target schema, migration constraints, and temporary scope boundaries.

Scope Clarification

• video items are out of scope for V2 knowledge data migration and should be skipped.
• memory items belong to the memory module, not the knowledge module, and should be skipped in knowledge migration.

Current Target Schema

knowledge_base

• Persisted columns:
  • id
  • name
  • groupId
  • dimensions
  • embeddingModelId
  • status
  • error
  • rerankModelId
  • fileProcessorId
  • chunkSize
  • chunkOverlap
  • threshold
  • documentCount
  • createdAt
  • updatedAt

knowledge_item

• Persisted columns:
  • id
  • baseId
  • groupId
  • type
  • data
  • status
  • error
  • createdAt
  • updatedAt
• New app-created knowledge items use ordered UUID generation for id.

Fields Removed From The V2 SQLite Schema

• video is not a target knowledge_item.type.
• memory is not a target knowledge_item.type.
• sitemap is not a target knowledge_item.type; legacy sitemap entries are migrated as url items.
• Legacy runtime-only item fields are not stored as standalone SQLite columns:
  • uniqueId
  • uniqueIds
  • processingProgress
  • retryCount
  • isPreprocessed
• remark is not part of the V2 SQLite schema.
• sourceUrl is not a standalone knowledge_item column:
  • for notes, it may exist inside data.sourceUrl
  • for url items, the URL is stored inside the typed data payload
• Official v1 legacy exports do not contain groupId.

groupId Semantics

• knowledge_item is modeled as a flat same-base item collection.
• groupId is an optional stable grouping key inside one knowledge base.
  • Typical usage: items from the same imported source/container
  • Examples: one directory import, one URL collection
  • When one item is the logical container/owner of a group, downstream items use groupId = containerItem.id
  • The schema enforces same-base ownership:
    • (baseId, groupId) must reference (baseId, id) in knowledge_item
    • deleting the owner cascades to grouped members
• Current runtime read flows use:
  • GET /knowledge-bases/:id/items for flat item listing
  • optional query filters: type, groupId
• Current runtime write workflows use KnowledgeService IPC, not DataApi endpoints:
  • add items: normalize caller-friendly inputs, create SQLite rows, and enqueue prepare/index tasks
  • delete items: interrupt runtime work, delete vectors, then delete SQLite roots
  • reindex items: interrupt runtime work, delete old vectors, rebuild expanded children when needed, then enqueue indexing
  • search and chunk mutation: execute against the per-base vector store through runtime IPC
• DataApi remains limited to SQLite-backed reads and knowledge base metadata PATCH.
• Migration from official v1 data does not preserve or infer grouping metadata:
  • official v1 exports are flat
  • migrated items are inserted with groupId = null

Current type / data Integrity Boundary

• knowledge_item.type and knowledge_item.data are intended to stay aligned by controlled UI flows.
• In the current V2 scope, knowledge item create/edit operations are expected to come from strongly associated UI forms or controlled write paths for each item type.
• The current implementation does not add an extra DB-level cross-structure constraint that re-validates data against the stored type on every write.
• At the DataApi/service layer:
  • create flows still rely on controlled write paths to keep type and data aligned
  • update flows re-validate data against the stored type before persisting changes
• Downstream knowledge code may therefore treat the stored type + data pair as a trusted contract produced by the app's controlled write path.
• If future write paths are added outside the current controlled UI flow, such as import tools, scripts, sync jobs, or public/external APIs, this assumption must be revisited and explicit boundary validation should be added at that time.

Current Non-Goals

• This phase does not reconstruct hierarchy from legacy v1 exports.
• This phase does not infer directory child relationships during migration.
• This phase does not introduce a first-class knowledge_group table.
• This phase does not preserve temporary processing lifecycle states beyond the uniqueId-based status rule below.
• This phase does not migrate video or memory into V2 knowledge tables.

dimensions Resolution Rule

• dimensions is treated as a required field for target V2 knowledge_base.
• Migration does not trust legacy Redux dimensions as the source of truth.
• Migration must resolve dimensions from the legacy vector database by inspecting:
  • the per-base legacy vector DB file
  • the vectors table
  • a non-null vector blob whose byte length can be converted to a positive dimension count
• Resolution is considered failed when the legacy vector DB is missing, empty, invalid, or its vector blob length cannot be parsed into a valid positive dimension count.
• When resolution fails, the knowledge base is considered unusable in V2 migration:
  • skip the entire base
  • skip all items under that base
  • record a warning for diagnostics
• Migration does not apply fallback or auto-fix for unresolved dimensions.

Item Status Migration Rule

• Legacy processingStatus is treated as runtime state and is not used as the migration source of truth.
• Migration infers target V2 knowledge_item.status from legacy uniqueId:
  • non-empty uniqueId -> completed
  • otherwise -> idle
• Temporary legacy states such as in-progress or failed processing are not preserved as V2 status during migration.

Runtime Status Boundary

• knowledge_item.status and knowledge_item.error remain part of the official V2 business schema.
• The runtime queue implementation is not part of the schema contract:
  • no separate task table
  • no persisted queue record
  • no persisted task run id
• Runtime currently uses an in-memory p-queue based pipeline in KnowledgeRuntimeService.
• The schema-level status set is:
  • idle
  • preparing
  • processing
  • reading
  • embedding
  • completed
  • failed
  • deleting
• Current runtime writes:
  • preparing while a directory root or nested directory is being expanded
  • reading while a leaf item is reading source documents
  • embedding while a leaf item is embedding / writing vectors
  • processing while a container has active descendants but is not itself expanding
  • completed after successful leaf indexing, or when a container has no active children
  • failed on runtime failure, interrupt cleanup failure, or shutdown interruption
• fileProcessorId is persisted in base config, but it does not participate in runtime execution yet.
• In other words:
  • queue structure is implementation detail
  • status is business lifecycle and coarse runtime progress
  • container status is reconciled from its own status and child item statuses
  • these concerns must not be conflated

Current Runtime Consumption Notes

• Runtime entrypoint:
  • src/main/services/knowledge/runtime/KnowledgeRuntimeService.ts
• Reader dispatch code still exists for stored knowledge_item.type values:
  • file -> file reader by extension
  • url -> fetch markdown through Jina Reader
  • note -> inline note content
  • directory -> currently treated as a container placeholder and returns no documents
• sitemap is no longer a valid persisted V2 knowledge_item.type. Legacy v1 sitemap items are mapped to url during migration and indexed through the URL path.
• Runtime add flow accepts new item payloads:
  • leaf payloads create knowledge_item rows and enqueue index-leaf
  • directory payloads create root rows and enqueue prepare-root
• prepare-root expands a directory owner inside the runtime queue, creates child rows, and enqueues concrete leaf children as index-leaf.
• Callers must not create user-supplied nested directory items under another item. Nested directory rows may still be created internally by directory expansion to preserve filesystem hierarchy.
• Runtime embedding model resolution currently expects knowledge_base.embeddingModelId in providerId::modelId format and only supports ollama as the active provider.

Implementation Status

• video and memory items are skipped during migration.
• The target schema uses optional groupId, but migration from official v1 data still writes it as null.
• The current DataApi contract exposes flat item read/listing only; write operations go through runtime orchestration.
• Group ownership is represented implicitly by groupId = ownerItem.id; there is no standalone group table in the current phase.
• dimensions resolution failure skips the entire base and all nested items, with warnings recorded in migration output.
• Knowledge item status migration uses uniqueId instead of processingStatus.
• The current runtime service is KnowledgeRuntimeService, not the old KnowledgeService name used in earlier notes.
• Current runtime queue behavior is a single in-memory PQueue({ concurrency: 5 }) shared across knowledge bases; there is no per-base serial queue yet.
• Current runtime queue entries are prepare-root and index-leaf; preparation and leaf indexing share interrupt / wait / shutdown cleanup semantics.