Path Locks and Crash Recovery

OpenViking uses two simple primitives — path locks and redo log — to protect the consistency of core write operations (rm, mv, add_resource, session.commit), ensuring that VikingFS, VectorDB, and QueueManager remain consistent even when failures occur.

Design Philosophy

OpenViking is a context database where FS is the source of truth and VectorDB is a derived index. A lost index can be rebuilt from source data, but lost source data is unrecoverable. Therefore:

Better to miss a search result than to return a bad one.

Design Principles

1. Write-exclusive: Path locks ensure only one write operation can operate on a path at a time
2. On by default: All data operations automatically acquire locks; no extra configuration needed
3. Lock as protection: LockContext acquires locks on entry, releases on exit — no undo/journal/commit semantics
4. Only session_memory needs crash recovery: RedoLog re-executes memory extraction after a process crash
5. Queue operations run outside locks: SemanticQueue/EmbeddingQueue enqueue operations are idempotent and retriable

Architecture

Service Layer (rm / mv / add_resource / session.commit)
    |
    v
+--[LockContext async context manager]--+
|                                       |
|  1. Create LockHandle                 |
|  2. Acquire path lock (poll+timeout)  |
|  3. Execute operations (FS+VectorDB)  |
|  4. Release lock                      |
|                                       |
|  On exception: auto-release lock,     |
|  exception propagates unchanged       |
+---------------------------------------+
    |
    v
Storage Layer (VikingFS, VectorDB, QueueManager)

Two Core Components

Component 1: PathLockEngine + LockManager + LockContext (Path Lock System)

PathLockEngine implements file-based distributed locks with two lock types — EXACT and TREE — using fencing tokens to prevent TOCTOU races and automatic stale lock detection and cleanup.

LockHandle is a lightweight lock holder token:

@dataclass
class LockHandle:
    id: str          # Unique ID used to generate fencing tokens
    locks: list[str] # Acquired lock file paths
    created_at: float # Handle creation time
    last_active_at: float # Last successful acquire/refresh time

LockManager is a global singleton managing lock lifecycle:

• Creates/releases LockHandles
• Background cleanup of leaked locks (in-process safety net)
• Executes RedoLog recovery on startup

LockContext is an async context manager encapsulating the lock/unlock lifecycle:

from openviking.storage.transaction import LockContext, get_lock_manager

async with LockContext(get_lock_manager(), [path], lock_mode="exact") as handle:
    # Perform operations under lock protection
    ...
# Lock automatically released on exit (including exceptions)

Component 2: RedoLog (Crash Recovery)

Used only for the memory extraction phase of session.commit. Writes a marker before the operation, deletes it after success, and scans for leftover markers on startup to redo.

/local/_system/redo/{task_id}/redo.json

Memory extraction is idempotent — re-extracting from the same archive produces the same result.

Consistency Issues and Solutions

rm(uri)

Locking strategy (depends on target type):

• Deleting a directory: lock_mode="tree", locks the directory and its subtree
• Deleting a file: lock_mode="exact", locks the file path itself

Operation flow:

1. Check whether target is a directory or file, choose lock mode
2. Acquire lock
3. Delete VectorDB index -> immediately invisible to search
4. Delete FS file
5. Release lock

VectorDB deletion fails -> exception thrown, lock auto-released, file and index both intact. FS deletion fails -> VectorDB already deleted but file remains, retry is safe.

mv(old_uri, new_uri)

Locking strategy (handled automatically via lock_mode="mv"):

• Moving a directory: TreeLock on the source path and ExactPathLock on the destination path
• Moving a file: EXACT lock on both source path and destination path

Operation flow:

1. Check whether source is a directory or file, set src_is_dir
2. Acquire mv lock (internally chooses TreeLock or ExactPathLock based on src_is_dir)
3. Copy to new location (source still intact, safe)
4. If directory, remove the lock file carried over by cp into the copy
5. Update VectorDB URIs
   - Failure -> clean up copy, source and old index intact, consistent state
6. Delete source
7. Release lock

add_resource

First-time add (target does not exist) — handled in ResourceProcessor.process_resource Phase 3.5:

1. Acquire TreeLock on final_uri
   - If final_uri does not exist, check ancestor/descendant/same-path conflicts first
   - If there is no conflict, create final_uri and write final_uri/.path.ovlock as a T lock
2. Keep temp as the source directory and enqueue SemanticMsg(uri=temp, target_uri=final_uri, lifecycle_lock_handle_id=...)
3. DAG runs on temp and syncs temp content into final_uri after completion
   - Do not use raw agfs.mv(temp -> final_uri), because final_uri already exists for the lock file
4. Clean up temp directory
5. DAG starts lock refresh loop (refreshes the lock token and updates handle activity every lock_expire/2 seconds)
6. DAG complete + all embeddings done -> release TreeLock

If summarization and indexing are both disabled, no downstream DAG takes over. In that case ResourceProcessor copies temp directory content into final_uri under the same TreeLock, deletes temp, then releases the lock. It does not call VikingFS.mv(temp, final_uri, lock_handle=handle), because move cleanup can remove the directory lock file.

During this period, rm attempting to acquire a TreeLock on the same path will fail with ResourceBusyError.

Incremental update (target already exists) — temp stays in place:

1. Acquire TreeLock on target_uri (protect existing resource)
2. Enqueue SemanticMsg(uri=temp, target_uri=final, lifecycle_lock_handle_id=...)
3. DAG runs on temp, lock refresh loop active
4. DAG completion triggers sync_diff_callback or move_temp_to_target_callback
5. Callback completes -> release TreeLock

Note: DAG callbacks do NOT wrap operations in an outer lock. Each VikingFS.rm and VikingFS.mv has its own lock internally. An outer lock would conflict with these inner locks causing deadlock.

Both first-time add and incremental update hold only TreeLock(resource_dir). There is no ExactPathLock(resource_dir) -> TreeLock(resource_dir) handoff, so the two modes cannot accidentally release the same .path.ovlock file in the wrong scope.

Automatic naming is handled by the resource layer, not the lock service: ResourceProcessor checks exists(candidate_uri) first; occupied candidates try _1, _2, and so on. Only a non-existing candidate attempts TreeLock, without waiting. If that candidate is busy, the next suffix is tried.

Server restart recovery: SemanticMsg is persisted in QueueFS. On restart, SemanticProcessor detects that the lifecycle_lock_handle_id handle is missing from the in-memory LockManager and re-acquires a TreeLock.

Derived Semantic Files (.abstract.md / .overview.md)

.abstract.md and .overview.md are generated sidecar files, not regular user source files. Their concurrency protection has two layers:

Example: concurrent writes to docs/a.md, docs/b.md, and docs/c.md hold separate ExactPathLocks and do not block each other. Background refresh may start multiple docs/ summary tasks, but only the latest version writes docs/.overview.md and docs/.abstract.md; stale tasks drop their results before writeback.

Memory directory summaries use the same rule. Concurrent writes to:

viking://user/default/memories/preferences/theme.md
viking://user/default/memories/preferences/editor.md

hold separate ExactPathLocks for the two source files. Refreshing preferences/.overview.md and preferences/.abstract.md no longer needs a long TreeLock; stale background tasks are filtered by coalesce_version, and final sidecar writes briefly acquire ExactPathLock.

session.commit()

LLM calls have unpredictable latency (5s~60s+) and cannot be inside a lock-holding operation. The design splits into two phases:

Phase 1 — Archive (no lock):
  1. Generate archive summary (LLM)
  2. Write archive (history/archive_N/messages.jsonl + summaries)
  3. Clear messages.jsonl
  4. Clear in-memory message list

Phase 2 — Memory extraction + write (RedoLog):
  1. Write redo marker (archive_uri, session_uri, user identity)
  2. Extract memories from archived messages (LLM)
  3. Write current message state
  4. Write relations
  5. Directly enqueue SemanticQueue
  6. Delete redo marker

Crash recovery analysis:

LockContext

LockContext is an async context manager that encapsulates lock acquisition and release:

from openviking.storage.transaction import LockContext, get_lock_manager

lock_manager = get_lock_manager()

# Exact lock (write operations, semantic processing)
async with LockContext(lock_manager, [path], lock_mode="exact"):
    # Perform operations...
    pass

# Tree lock (directory delete and lifecycle protection)
async with LockContext(lock_manager, [path], lock_mode="tree"):
    # Perform operations...
    pass

# MV lock (move operations)
async with LockContext(lock_manager, [src], lock_mode="mv", mv_dst_path=dst):
    # Perform operations...
    pass

Lock modes:

Exception handling: __aexit__ always releases locks and does not swallow exceptions. Lock acquisition failure raises LockAcquisitionError.

Lock Types (EXACT vs TREE)

The lock mechanism uses two lock types to handle different conflict patterns:

• EXACT (E): Locks one concrete path. It can protect files, directory names, and not-yet-created target paths. Blocks if any ancestor holds a TreeLock.
• TREE (T): Used for directory delete, directory move, resource lifecycle protection, and similar subtree-level operations. Logically covers the entire subtree but only writes one lock file at the root. Before acquiring, scans all descendants and ancestor directories for conflicting locks. If the target directory is missing, conflicts are checked first; only then is the directory created and locked. If a later double-check finds a new conflict, the acquire fails or retries without rolling back the empty directory.

Lock Mechanism

Lock Protocol

Lock file paths:

TreeLock(path)                  -> {path}/.path.ovlock
ExactPathLock(existing dir path) -> {path}/.path.ovlock
ExactPathLock(file or missing path) -> {parent}/.exact.ovlock.<name>.<hash>

Lock file content (Fencing Token):

{handle_id}:{time_ns}:{lock_type}

Where lock_type is E (EXACT) or T (TREE).

Lock Acquisition (EXACT mode)

loop until timeout (poll interval: 200ms):
    1. Check if target path is locked by another operation
       - Stale lock? -> remove and retry
       - Active lock? -> wait
    2. Check all ancestor directories for TREE locks
       - Stale lock? -> remove and retry
       - Active lock? -> wait
    3. Ensure the lock file's parent directory exists; create it if missing
    4. Write EXACT (E) lock file
    5. TOCTOU double-check: re-scan target path and ancestors for TREE locks
       - Conflict found: compare (timestamp, handle_id)
       - Later one (larger timestamp/handle_id) backs off (removes own lock) to prevent livelock
       - Wait and retry
    6. Verify lock file ownership (fencing token matches)
    7. Success

Timeout (default 0 = no-wait) raises LockAcquisitionError

Lock Acquisition (TREE mode)

loop until timeout (poll interval: 200ms):
    1. Check if target directory is locked by another operation
       - Stale lock? -> remove and retry
       - Active lock? -> wait
    2. Check all ancestor directories for TREE locks
       - Stale lock? -> remove and retry
       - Active lock? -> wait
    3. Scan all descendant directories for any locks by other operations
       - Missing target directory? -> treat as no descendant locks
       - Stale lock? -> remove and retry
       - Active lock? -> wait
    4. Ensure the target directory exists; create it if missing
    5. Write TREE (T) lock file (only one file, at the root path)
    6. TOCTOU double-check: re-scan descendants and ancestors
       - Conflict found: compare (timestamp, handle_id)
       - Later one (larger timestamp/handle_id) backs off (removes own lock) to prevent livelock
       - Wait and retry
    7. Verify lock file ownership (fencing token matches)
    8. Success

Timeout (default 0 = no-wait) raises LockAcquisitionError

Missing Directory Creation

The lock system may create directories so it can place lock files, but it checks for conflicts first:

1. Ancestor TreeLock / same-path lock / descendant lock conflict -> do not create the directory
2. No current conflict -> create the directory and write the lock
3. A post-write double-check finds a new conflict -> remove our own lock and fail or retry
4. Step 3 does not roll back the empty directory

Lock Expiry Cleanup

Stale lock detection: PathLockEngine checks the fencing token timestamp. Locks older than lock_expire (default 300s) are considered stale and are removed automatically during acquisition.

In-process cleanup: LockManager checks active LockHandles every 60 seconds. Handles that still own lock files but have been inactive for longer than lock_expire are force-released.

Orphan locks: Lock files left behind after a process crash are automatically removed via stale lock detection when any operation next attempts to acquire a lock on the same path.

Crash Recovery

LockManager.start() automatically scans for leftover markers in /local/_system/redo/ on startup:

Defense Summary

Configuration

Path locks are enabled by default with no extra configuration needed. The default behavior is no-wait: if the path is locked, LockAcquisitionError is raised immediately. To allow wait/retry, configure the storage.transaction section:

{
  "storage": {
    "transaction": {
      "lock_timeout": 5.0,
      "lock_expire": 300.0
    }
  }
}

QueueFS Persistence

The lock mechanism relies on QueueFS using the SQLite backend to ensure enqueued tasks survive process restarts. This is the default configuration and requires no manual setup.

Related Documentation

• Architecture - System architecture overview
• Storage - AGFS and vector store
• Session Management - Session and memory management
• Configuration - Configuration reference