Virtual Sidecar System (VSS)

The Virtual Sidecar System (VSS) manages derivative data associated with your files - thumbnails, OCR text, video transcripts, embeddings, and other metadata-rich artifacts.

Key Principles

Content-Scoped: Sidecars are linked to file content, not paths. Multiple copies of the same file share one set of sidecars.

Non-Destructive: Spacedrive never modifies your original files during indexing.

Portable: All managed sidecars live in .sdlibrary/sidecars, making your entire setup portable.

Sidecar Types

The VSS handles two types of sidecars:

Managed Sidecars - Spacedrive-generated derivatives stored in .sdlibrary/sidecars/:

• Thumbnails (.webp)
• Video proxies (.mp4)
• OCR text (.json)
• Subtitles and lyrics (.srt)

Reference Sidecars - Pre-existing files treated as sidecars:

• Live Photo video components
• RAW + JPEG pairs
• Tracked by source_entry_id in database
• Files stay in original location until user converts them to managed sidecars

Filesystem Layout

All sidecars are stored in a sidecars directory within the .sdlibrary. The path to a sidecar is deterministic and is derived from the content UUID of the file it is associated with, the kind of sidecar, and a variant.

A typical path looks like this:

.sdlibrary/
  sidecars/
    content/
      {h0}/{h1}/{content_uuid}/
        thumbs/{variant}.webp
        proxies/{profile}.mp4
        transcript/{variant}.srt
        ocr/default.json

• {h0}/{h1}: These are the first two byte-pairs of the content UUID, used for sharding the directory structure to ensure good filesystem performance at scale.
• {content_uuid}: The unique identifier of the content the sidecar is associated with.
• thumbs/{variant}.webp: An example of a thumbnail sidecar.

Database Schema

The VSS uses two tables in the Spacedrive database:

• sidecars: This table tracks all the sidecars that have been generated or are pending generation. It stores information like the content UUID, kind, variant, format, path, size, and status.
• sidecar_availability: This table tracks which devices in your library have a copy of a particular sidecar. This is used to avoid regenerating sidecars that already exist on another device.

Sidecar Lifecycle

1. Identification

• Managed: Indexer queues generation job, creates "pending" record in sidecars table
• Reference: Indexer creates sidecar record with source_entry_id pointing to original file

2. Generation

• Managed: Background job generates derivative, stores in deterministic path
• Reference: No generation - already exists

3. Recording

• Managed: Job marks sidecar "ready", records size and metadata
• Reference: Created as "ready" immediately
• Both update sidecar_availability table for current device

4. Syncing

• Devices exchange availability information
• Missing sidecars can be transferred from peers instead of regenerating

Current Implementation Status

The VSS is partially implemented. Here is a summary of the current status:

• Implemented:
  • The core SidecarManager service.
  • Filesystem layout and path generation.
  • Database schema and queries for sidecars and sidecar_availability.
  • Enqueueing of sidecar generation jobs.
  • A bootstrap_scan function to synchronize the database with the filesystem.
  • Reference sidecars, which allow tracking files as sidecars without moving them.
• To-Do:
  • Job Dispatch and Execution: The system for actually executing the generation jobs is not yet implemented. This is the most critical missing piece.
  • Filesystem Watcher: A watcher for the sidecars directory is needed for real-time updates.
  • Checksumming: Checksumming of sidecar files for integrity verification is not yet implemented.

How to Use the VSS (API)

The SidecarManager provides a set of APIs for interacting with the VSS. These are primarily for internal use by other Spacedrive services, but they can also be used for development and debugging.

• sidecars.presence(content_uuids, kind, variants): Checks for the presence of sidecars for a given set of content UUIDs.
• sidecars.path(content_uuid, kind, variant): Gets the local path to a sidecar, or enqueues it for generation if it doesn't exist.
• sidecars.reconcile(): Triggers a bootstrap scan to reconcile the database with the filesystem.

Benefits

• Non-Destructive: Reference sidecars preserve original file locations
• Portable: All managed sidecars in .sdlibrary can be moved/backed up as one unit
• Efficient: One sidecar per content hash, shared across devices
• Extensible: New sidecar kinds can be added without schema changes
• AI-Ready: Provides structured data for semantic search and intelligent organization