Navidrome Scanner: Technical Overview

This document provides a comprehensive technical explanation of Navidrome's music library scanner system.

Architecture Overview

The Navidrome scanner is built on a multi-phase pipeline architecture designed for efficient processing of music files. It systematically traverses file system directories, processes metadata, and maintains a database representation of the music library. A key performance feature is that some phases run sequentially while others execute in parallel.

flowchart TD
    subgraph "Scanner Execution Flow"
        Controller[Scanner Controller] --> Scanner[Scanner Implementation]
        
        Scanner --> Phase1[Phase 1: Folders Scan]
        Phase1 --> Phase2[Phase 2: Missing Tracks]
        
        Phase2 --> ParallelPhases
        
        subgraph ParallelPhases["Parallel Execution"]
            Phase3[Phase 3: Refresh Albums]
            Phase4[Phase 4: Playlist Import]
        end
        
        ParallelPhases --> FinalSteps[Final Steps: GC + Stats]
    end
    
    %% Triggers that can initiate a scan
    FileChanges[File System Changes] -->|Detected by| Watcher[Filesystem Watcher]
    Watcher -->|Triggers| Controller
    
    ScheduledJob[Scheduled Job] -->|Based on Scanner.Schedule| Controller
    ServerStartup[Server Startup] -->|If Scanner.ScanOnStartup=true| Controller
    ManualTrigger[Manual Scan via UI/API] -->|Admin user action| Controller
    CLICommand[Command Line: navidrome scan] -->|Direct invocation| Controller
    PIDChange[PID Configuration Change] -->|Forces full scan| Controller
    DBMigration[Database Migration] -->|May require full scan| Controller
    
    Scanner -.->|Alternative| External[External Scanner Process]

The execution flow shows that Phases 1 and 2 run sequentially, while Phases 3 and 4 execute in parallel to maximize performance before the final processing steps.

Core Components

Scanner Controller (controller.go)

This is the entry point for all scanning operations. It provides:

• Public API for initiating scans and checking scan status
• Event broadcasting to notify clients about scan progress
• Serialization of scan operations (prevents concurrent scans)
• Progress tracking and monitoring
• Error collection and reporting

type Scanner interface {
    // ScanAll starts a full scan of the music library. This is a blocking operation.
    ScanAll(ctx context.Context, fullScan bool) (warnings []string, err error)
    Status(context.Context) (*StatusInfo, error)
}

Scanner Implementation (scanner.go)

The primary implementation that orchestrates the four-phase scanning pipeline. Each phase follows the Phase interface pattern:

type phase[T any] interface {
    producer() ppl.Producer[T]
    stages() []ppl.Stage[T]
    finalize(error) error
    description() string
}

This design enables:

• Type-safe pipeline construction with generics
• Modular phase implementation
• Separation of concerns
• Easy measurement of performance

External Scanner (external.go)

The External Scanner is a specialized implementation that offloads the scanning process to a separate subprocess. This is specifically designed to address memory management challenges in long-running Navidrome instances.

// scannerExternal is a scanner that runs an external process to do the scanning. It is used to avoid
// memory leaks or retention in the main process, as the scanner can consume a lot of memory. The
// external process will be spawned with the same executable as the current process, and will run
// the "scan" command with the "--subprocess" flag.
//
// The external process will send progress updates to the main process through its STDOUT, and the main
// process will forward them to the caller.

sequenceDiagram
    participant MP as Main Process
    participant ES as External Scanner
    participant SP as Subprocess (navidrome scan --subprocess)
    participant FS as File System
    participant DB as Database
    
    Note over MP: DevExternalScanner=true
    MP->>ES: ScanAll(ctx, fullScan)
    activate ES
    
    ES->>ES: Locate executable path
    ES->>SP: Start subprocess with args:
scan --subprocess --configfile ... etc.
    activate SP
    
    Note over ES,SP: Create pipe for communication
    
    par Subprocess executes scan
        SP->>FS: Read files & metadata
        SP->>DB: Update database
    and Main process monitors progress
        loop For each progress update
            SP->>ES: Send encoded progress info via stdout pipe
            ES->>MP: Forward progress info
        end
    end
    
    SP-->>ES: Subprocess completes (success/error)
    deactivate SP
    ES-->>MP: Return aggregated warnings/errors
    deactivate ES

Technical details:

1. Process Isolation

   • Spawns a separate process using the same executable
   • Uses the --subprocess flag to indicate it's running as a child process
   • Preserves configuration by passing required flags (--configfile, --datafolder, etc.)

2. Inter-Process Communication

   • Uses a pipe for bidirectional communication
   • Encodes/decodes progress updates using Go's gob encoding for efficient binary transfer
   • Properly handles process termination and error propagation

3. Memory Management Benefits

   • Scanning operations can be memory-intensive, especially with large music libraries
   • Memory leaks or excessive allocations are automatically cleaned up when the process terminates
   • Main Navidrome process remains stable even if scanner encounters memory-related issues

4. Error Handling

   • Detects non-zero exit codes from the subprocess
   • Propagates error messages back to the main process
   • Ensures resources are properly cleaned up, even in error conditions

Scanning Process Flow

Phase 1: Folder Scan (phase_1_folders.go)

This phase handles the initial traversal and media file processing.

flowchart TD
    A[Start Phase 1] --> B{Full Scan?}
    B -- Yes --> C[Scan All Folders]
    B -- No --> D[Scan Modified Folders]
    C --> E[Read File Metadata]
    D --> E
    E --> F[Create Artists]
    E --> G[Create Albums]
    F --> H[Save to Database]
    G --> H
    H --> I[Mark Missing Folders]
    I --> J[End Phase 1]

Technical implementation details:

1. Folder Traversal

   • Uses walkDirTree to traverse the directory structure
   • Handles symbolic links and hidden files
   • Processes .ndignore files for exclusions
   • Maps files to appropriate types (audio, image, playlist)

2. Metadata Extraction

   • Processes files in batches (defined by filesBatchSize = 200)
   • Extracts metadata using the configured storage backend
   • Converts raw metadata to MediaFile objects
   • Collects and normalizes tag information

3. Album and Artist Creation

   • Groups tracks by album ID
   • Creates album records from track metadata
   • Handles album ID changes by tracking previous IDs
   • Creates artist records from track participants

4. Database Persistence

   • Uses transactions for atomic updates
   • Preserves album annotations across ID changes
   • Updates library-artist mappings
   • Marks missing tracks for later processing
   • Pre-caches artwork for performance

Phase 2: Missing Tracks Processing (phase_2_missing_tracks.go)

This phase identifies tracks that have moved or been deleted.

flowchart TD
    A[Start Phase 2] --> B[Load Libraries]
    B --> C[Get Missing and Matching Tracks]
    C --> D[Group by PID]
    D --> E{Match Type?}
    E -- Exact --> F[Update Path]
    E -- Same PID --> G[Update If Only One]
    E -- Equivalent --> H[Update If No Better Match]
    F --> I[End Phase 2]
    G --> I
    H --> I

Technical implementation details:

1. Track Identification Strategy

   • Uses persistent identifiers (PIDs) to track tracks across scans
   • Loads missing tracks and potential matches from the database
   • Groups tracks by PID to limit comparison scope

2. Match Analysis

   • Applies three levels of matching criteria:
     • Exact match (full metadata equivalence)
     • Single match for a PID
     • Equivalent match (same base path or similar metadata)
   • Prioritizes matches in order of confidence

3. Database Update Strategy

   • Preserves the original track ID
   • Updates the path to the new location
   • Deletes the duplicate entry
   • Uses transactions to ensure atomicity

Phase 3: Album Refresh (phase_3_refresh_albums.go)

This phase updates album information based on the latest track metadata.

flowchart TD
    A[Start Phase 3] --> B[Load Touched Albums]
    B --> C[Filter Unmodified]
    C --> D{Changes Detected?}
    D -- Yes --> E[Refresh Album Data]
    D -- No --> F[Skip]
    E --> G[Update Database]
    F --> H[End Phase 3]
    G --> H
    H --> I[Refresh Statistics]

Technical implementation details:

1. Album Selection Logic

   • Loads albums that have been "touched" in previous phases
   • Uses a producer-consumer pattern for efficient processing
   • Retrieves all media files for each album for completeness

2. Change Detection

   • Rebuilds album metadata from associated tracks
   • Compares album attributes for changes
   • Skips albums with no media files
   • Avoids unnecessary database updates

3. Statistics Refreshing

   • Updates album play counts
   • Updates artist play counts
   • Maintains consistency between related entities

Phase 4: Playlist Import (phase_4_playlists.go)

This phase imports and updates playlists from the file system.

flowchart TD
    A[Start Phase 4] --> B{AutoImportPlaylists?}
    B -- No --> C[Skip]
    B -- Yes --> D{Admin User Exists?}
    D -- No --> E[Log Warning & Skip]
    D -- Yes --> F[Load Folders with Playlists]
    F --> G{For Each Folder}
    G --> H[Read Directory]
    H --> I{For Each Playlist}
    I --> J[Import Playlist]
    J --> K[Pre-cache Artwork]
    K --> L[End Phase 4]
    C --> L
    E --> L

Technical implementation details:

1. Playlist Discovery

   • Loads folders known to contain playlists
   • Focuses on folders that have been touched in previous phases
   • Handles both playlist formats (M3U, NSP)

2. Import Process

   • Uses the core.Playlists service for import
   • Handles both regular and smart playlists
   • Updates existing playlists when changed
   • Pre-caches playlist cover art

3. Configuration Awareness

   • Respects the AutoImportPlaylists setting
   • Requires an admin user for playlist import
   • Logs appropriate messages for configuration issues

Final Processing Steps

After the four main phases, several finalization steps occur:

1. Garbage Collection

   • Removes dangling tracks with no files
   • Cleans up empty albums
   • Removes orphaned artists
   • Deletes orphaned annotations

2. Statistics Refresh

   • Updates artist song and album counts
   • Refreshes tag usage statistics
   • Updates aggregate metrics

3. Library Status Update

   • Marks scan as completed
   • Updates last scan timestamp
   • Stores persistent ID configuration

4. Database Optimization

   • Performs database maintenance
   • Optimizes tables and indexes
   • Reclaims space from deleted records

File System Watching

The watcher system (watcher.go) provides real-time monitoring of file system changes:

flowchart TD
    A[Start Watcher] --> B[For Each Library]
    B --> C[Start Library Watcher]
    C --> D[Monitor File Events]
    D --> E{Change Detected?}
    E -- Yes --> F[Wait for More Changes]
    F --> G{Time Elapsed?}
    G -- Yes --> H[Trigger Scan]
    G -- No --> F
    H --> I[Wait for Scan Completion]
    I --> D

Technical implementation details:

1. Event Throttling

   • Uses a timer to batch changes
   • Prevents excessive rescanning
   • Configurable wait period

2. Library-specific Watching

   • Each library has its own watcher goroutine
   • Translates paths to library-relative paths
   • Filters irrelevant changes

3. Platform Adaptability

   • Uses storage-provided watcher implementation
   • Supports different notification mechanisms per platform
   • Graceful fallback when watching is not supported

Edge Cases and Optimizations

Handling Album ID Changes

The scanner carefully manages album identity across scans:

• Tracks previous album IDs to handle ID generation changes
• Preserves annotations when IDs change
• Maintains creation timestamps for consistent sorting

Detecting Moved Files

A sophisticated algorithm identifies moved files:

1. Groups missing and new files by their Persistent ID
2. Applies multiple matching strategies in priority order
3. Updates paths rather than creating duplicate entries

Resuming Interrupted Scans

If a scan is interrupted:

• The next scan detects this condition
• Forces a full scan if the previous one was a full scan
• Continues from where it left off for incremental scans

Memory Efficiency

Several strategies minimize memory usage:

• Batched file processing (200 files at a time)
• External scanner process option
• Database-side filtering where possible
• Stream processing with pipelines

Concurrency Control

The scanner implements a sophisticated concurrency model to optimize performance:

1. Phase-Level Parallelism:

   • Phases 1 and 2 run sequentially due to their dependencies
   • Phases 3 and 4 run in parallel using the chain.RunParallel() function
   • Final steps run sequentially to ensure data consistency

2. Within-Phase Concurrency:

   • Each phase has configurable concurrency for its stages
   • For example, phase_1_folders.go processes folders concurrently: ppl.NewStage(p.processFolder, ppl.Name("process folder"), ppl.Concurrency(conf.Server.DevScannerThreads))
   • Multiple stages can exist within a phase, each with its own concurrency level

3. Pipeline Architecture Benefits:

   • Producer-consumer pattern minimizes memory usage
   • Work is streamed through stages rather than accumulated
   • Back-pressure is automatically managed

4. Thread Safety Mechanisms:

   • Atomic counters for statistics gathering
   • Mutex protection for shared resources
   • Transactional database operations

Configuration Options

The scanner's behavior can be customized through several configuration settings that directly affect its operation:

Core Scanner Options

Playlist Processing

Performance Options

Persistent ID Options

These options can be set in the Navidrome configuration file (e.g., navidrome.toml) or via environment variables with the ND_ prefix (e.g., ND_SCANNER_ENABLED=false). For environment variables, dots in option names are replaced with underscores.

Conclusion

The Navidrome scanner represents a sophisticated system for efficiently managing music libraries. Its phase-based pipeline architecture, careful handling of edge cases, and performance optimizations allow it to handle libraries of significant size while maintaining data integrity and providing a responsive user experience.