Runtime property reindex

Change inverted-index configuration on a live class — without restarting the cluster, without losing writes that arrive during the rebuild, and without dropping query availability on the affected property.

This document is the entry-point for anyone reviewing or extending the feature. It covers the full surface area: REST API, architecture layer by layer, the migration-strategy catalogue, crash-safety contract, concurrency model, multi-tenancy, tokenization overlay, the deferred-finalize / per-migration generation design that closes the load-bearing #10675 family of data-loss bugs, and the test map.

The package-level godocs of the touched packages are the per-symbol source of truth; this doc is the navigable overview that ties them together. If a section here disagrees with a godoc in source, the godoc wins — and that's a bug in this doc.

1. Overview

A runtime reindex rebuilds one or more inverted-index buckets on a property in place while the class stays open for reads and writes. The typical journeys it unlocks:

• Change a text property's tokenization (word → trigram, etc.) with no downtime — both the searchable and filterable buckets are rewritten on every replica and the schema's tokenization field is flipped only after every shard has committed its swap.
• Add a missing inverted index after the fact: enable-filterable, enable-searchable, enable-rangeable.
• Migrate a searchable index from WAND (Map) to BlockMax (Inverted): change-algorithm.
• Repair a bucket suspected of corruption: repair-filterable, rebuild-searchable (rebuild an existing BlockMax bucket in place), repair-rangeable.
• Upgrade a searchable index from the deprecated WAND format to BlockMax: change-algorithm.
• Cancel a migration that has not yet entered its cluster-wide coordination phase; the cluster cleans up the partial state and the property is back to its pre-submit on-disk shape — unless a node had already merged its rewritten buckets, which needs an operator rebuild instead (§12).

The whole feature is built on top of three substrates:

• The LSM bucket-swap primitives (adapters/repos/db/lsmkv/store.go SwapBucketPointer + FinalizeBucketSwap) — the atomic in-memory pointer flip and its deferred on-disk counterpart.
• The Distributed Task Manager (DTM) (cluster/distributedtask/) — RAFT-backed task state, per-unit assignment, group barriers, per-node PREPARING and SWAPPING coordination states, two-phase PreparationComplete + PostCompletion ack barriers (§6.3).
• The schema FSM mutation guard — rejects external mutations on classes / properties / tenants while a reindex is in flight, so the bucket↔schema invariant cannot be broken by a concurrent DeleteClass or UpdateProperty.

The reindex feature itself is the orchestration that ties these together into a small REST surface rooted at /v1/schema/{className}/properties/{propertyName}/index/{indexType}.

2. REST surface

Runtime reindex is off by default. Set RUNTIME_REINDEX_ENABLED=true to enable it. While it is off, submitting a migration returns 400 Bad Request with runtime reindex is disabled; enable with RUNTIME_REINDEX_ENABLED=true. The cancel verb and GET .../indexes keep working, so a task that was already running stays observable and, for as long as it is still STARTED, cancellable (§12). Everything below describes behavior with the flag on.

With the flag off, a replica move can kill a running migration. The backup path's reindex check is skipped, so a replica move — or any other operation that closes the shard — is admitted even while a migration is still writing to that shard. Closing the shard takes the storage away from the migration, and the migration fails loudly: the task ends in FAILED, the schema stays at its pre-migration value, and the copy itself completes normally. Nothing is corrupted, but the migration is lost and has to be resubmitted once the flag is back on. With the flag on, the move is refused and the replication engine retries it until the migration finishes.

PUT /v1/schema/{class}/indexes/{property}

Declaratively upsert one inverted index. {indexType} is one of filterable, searchable, rangeFilters (rangeable, the former Preview name, is accepted as a write-only alias; GET returns rangeFilters).

The body (IndexUpsertRequest) carries only configuration, never a verb: tokenization (searchable / filterable) and algorithm (searchable only). rangeFilters takes no fields. An empty body {} is valid and means "ensure the index exists with its current/default config". At most one of tokenization / algorithm may change per call. The server diffs the body against current state and picks the migration:

A PUT whose desired config already matches current state AND has no reindex task in flight submits no task and returns 200 with {"status":"NO_OP"}. A PUT that converges on a task still in flight also submits no task, but the config is not in place yet, so it returns 202 with that task's taskId.

202 with {"status":"STARTED", "taskId": ...} means a task is running for the requested configuration and taskId is the handle to poll. It does not assert that this call created the task: a request that converges on an already-running migration joins it and receives that task's ID.

POST /v1/schema/{className}/properties/{propertyName}/index/{indexType}/rebuild

Rebuild the index from stored objects with unchanged configuration (repair / format refresh). No request body. A WAND searchable index is rejected — migrate to Blockmax first via PUT {"algorithm":"blockmax"}.

POST /v1/schema/{className}/properties/{propertyName}/index/{indexType}/cancel

Cancel the in-flight task on (class, property, indexType). No request body. Idempotent: 202 + Status: CANCELLED (with taskId) when a STARTED task is cancelled, 202 + Status: NO_OP (no taskId) when nothing matches (already finished, never submitted, or already cancelled), 409 for every other in-flight status — a coordination phase (PREPARING / SWAPPING), or one this build does not recognize. See §12. Never 404 for "nothing to cancel".

Query parameters (PUT and rebuild):

• ?tenants=t1,t2 — scope to named tenants on a multi-tenant class. Rejected on single-tenant classes. On PUT, allowed only when the operation is format-only (rangeFilters creation); rebuild allows it for every index type. Rejected on every semantic migration (IsSemanticMigration: enable-searchable, enable-filterable, change-tokenization, change-tokenization-filterable, change-algorithm) because the cluster-wide schema flip cannot be sub-scoped — all tenants must migrate together.

Cancel takes no query parameters; a tenants value on the URL is silently ignored. Cancel targets the task itself, which already carries the tenant set of the rebuild that submitted it, so POST .../cancel after rebuild?tenants=t1,t2 stops only that work, leaving other tenants untouched.

Response shapes (PUT / rebuild / cancel):

• 200 OK — PUT only: desired config already in place AND no task in flight, so no task submitted ({"status":"NO_OP"}).
• 202 Accepted — a task is running for the requested configuration (PUT / rebuild, status: STARTED, taskId naming that task), or cancel processed; body is an IndexUpdateResponse (taskId + status). STARTED asserts that the task is running, not that this call created it: a PUT converging on an already-running migration joins it and receives that task's ID. Cancel answers CANCELLED + taskId when it cancelled a cancellable task, or NO_OP (no taskId) when nothing matched; it is idempotent and never returns 404 for "no task to cancel".
• 400 Bad Request — validation failure with an actionable hint (e.g. "property X has no searchable index; PUT /v1/schema/{className}/properties/X/index/searchable with a tokenization to add one first").
• 404 Not Found — class or property doesn't exist.
• 409 Conflict — two distinct meanings. On PUT / rebuild, an in-flight task already touches this property; the error names the offending task ID. On cancel, either arm can refuse. The pre-flight refuses a status other than STARTED (PREPARING or SWAPPING, where nodes may already have written merged state or renamed bucket directories, or one this build does not recognize) and names the task ID and that status. The apply refuses a target that left STARTED after that read, and names no status, since it cannot tell a coordination phase from a task that has already finished. Neither refusal cancelled anything, but the apply refusal cannot tell you what did: a concurrent cancel may have won the race, so re-read the index status to see where it landed — CANCELLED included. A pre-flight refusal leaves the task in flight, and a status this build does not recognize keeps reading as indexing here until it reaches a terminal state on the nodes that do (§12).
• 422 Unprocessable Entity — {indexType} outside the enum.
• 429 / 503 — per-collection in-flight cap reached (default 32), or the cluster service is unavailable / an in-flight task's payload cannot be parsed so conflict-freedom cannot be proven.

DELETE /v1/schema/{className}/properties/{propertyName}/index/{indexType}

Drop a configured inverted index. {indexType} is one of filterable, searchable, rangeFilters (rangeable accepted as an alias). Flips the corresponding schema flag to false, drops the bucket dir, removes the sidecar buckets, and scrubs the stale migration trackers — a generation-suffixed tracker carrying tidied.mig / merged.mig is live deferred-finalize state and is deliberately kept — so a subsequent re-enable starts from a clean slate. Subject to the same MutationGuard as UpdateProperty — rejected while a reindex on this property is in flight.

GET /v1/schema/{className}/indexes

Per-property, per-index-type snapshot:

{
  "collection": "Article",
  "properties": [{
    "name": "body",
    "dataType": "text",
    "indexes": [
      { "type": "filterable", "status": "ready", "tokenization": "word" },
      { "type": "searchable", "status": "indexing", "progress": 0.42,
        "tokenization": "word", "targetTokenization": "trigram",
        "algorithm": "blockmax" }
    ]
  }]
}

Status values: ready, pending, indexing, failed, cancelled. The status is synthesized in mergeReindexStatus from the DTM task list crossed with the schema flags. Both values are read from the node serving the request, and the task list is read first, so the schema is never the older of the two: a class read at a later applied index carries every flag flip the tasks it is compared against have already committed. A FINISHED task surfaces nothing of its own — the schema flag alone decides whether an entry is emitted.

The apply edge

PUT .../index/{indexType} returns 202 as soon as the leader applied the add-task entry (Raft.Execute). On a follower the call does not wait for that node's own apply; a leader-served PUT has already applied locally when the 202 returns. Cancel works the same way: it decides on the leader-routed task list and applies through Raft.Execute.

A GET against a lagging follower right after that 202 finds no task and answers from the flags alone, in one of two shapes:

• enable-*, flag still off: the entry is dropped. The indexes array reads null only when every applicable type on the property is off; with indexFilterable still on you get a non-null array that is merely missing one entry.
• change-tokenization / change-algorithm, flag already on at submit time: the entry emits as ready with the pre-migration tokenization, reporting the property done and unchanged.

For a client pinned to one node the window is bounded by that node's replication lag, and the entry does not flap. It is left open deliberately: closing it means blocking a mutation on a local apply, which turns a 202 into a latency-bound call that can hang on a partitioned follower, to remove a transient polling artifact.

A client that round-robins across nodes does see it flap. Nodes that have applied the add-task entry and nodes that have not answer differently, so the pill alternates between indexing and no entry at all, and algorithm between blockmax and wand for a property whose stamp only one of them has.

This endpoint is node-local while GET /v1/tasks is leader-routed, so a UI polling both against a lagging follower can render the task FINISHED and the index pill indexing at the same moment.

RBAC

Authorization is symmetric across every index write verb: PUT (upsert), POST .../rebuild, POST .../cancel, and DELETE all require UPDATE on Collections, which resolves to both update_collections (metadata) and update_data. A per-property index write rewrites indexed data, so metadata permission alone is insufficient — a caller with only update_collections gets 403.

GET .../indexes requires READ on CollectionsMetadata, i.e. read_collections.

Mutations authorize before any conflict/in-flight pre-flight, so an unprivileged caller never learns whether a reindex is running (no 404/409 leak). The REST handler and the manager (DeleteClassPropertyIndex) enforce the same permission.

3. End-to-end lifecycle

The diagram below tracks state across two independent status surfaces — keep the distinction in mind reading top-to-bottom:

• UnitStatus (PENDING → IN_PROGRESS → COMPLETED/FAILED) — per-unit. One unit = one shard × one node. COMPLETED means this single replica has finished its piece of the reindex work; the per-shard swap and cluster-wide schema flip are still ahead.
• TaskStatus — per-task aggregate. The transition sequence depends on whether the task opts into the PREP barrier (NeedsPreparationBarrier, set automatically for semantic migrations by the submit handler; full mechanics in §6.3):
  • Semantic migrations (change-tokenization, change-tokenization-filterable, enable-filterable, enable-searchable, change-algorithm): STARTED → PREPARING → SWAPPING → FINISHED. PREPARING and SWAPPING are both reached only after every unit across the cluster is at terminal status. The FSM gates PREPARING → SWAPPING on every node's PreparationCompleteAck landing successfully, and gates SWAPPING → FINISHED on every node's PostCompletionAck landing successfully.
  • Format-only migrations (enable-rangeable, repair-filterable, repair-rangeable, rebuild-searchable): STARTED → SWAPPING → FINISHED. PREPARING is skipped because there is no cross-replica state alignment to bound — each shard's RunOnShard completes the full lifecycle locally and there is no cluster-wide schema flip.
  • Either path may terminate at FAILED (any per-node ack Success=false) or CANCELLED (operator-initiated). FINISHED is the correct signal for "fully done".

These are different Go types in the source — cluster/distributedtask/types.go defines UnitStatus and TaskStatus distinctly — so the COMPLETED-before-PREPARING ordering in the diagram is not a sequence on the same field; it's the terminal value of the per-unit field preceding a transition on the per-task field. Annotations (← UnitStatus / ← TaskStatus) mark which surface each box lives on.

                ┌────────────────────────────────────────────────────────┐
                │         PUT .../properties/{prop}/index/{type}         │
                └──────────────────────────┬─────────────────────────────┘
                                           │
        ┌──────────────────────────────────▼─────────────────────────────┐
        │  REST handler (adapters/handlers/rest/handlers_indexes.go)    │
        │   • Per-(class, prop) ReindexSubmitLock                       │
        │   • Validate body, classify migration type, dispatch          │
        │   • checkReindexConflict (read-side mirror of FSM check)      │
        │   • Pre-submit DB.NewStalePartialReindexSweep                 │
        │   • AddDistributedTaskWith[Groups]Barrier("reindex", payload) │
        └──────────────────────────────────┬─────────────────────────────┘
                                           │  RAFT-replicated AddTask
        ┌──────────────────────────────────▼─────────────────────────────┐
        │  cluster/distributedtask FSM (Manager)              ← TaskSt. │
        │   • ConflictDetector.CheckConflict (FSM-deterministic)        │
        │   • Append task to FSM state at STARTED                       │
        └──────────────────────────────────┬─────────────────────────────┘
                                           │  Scheduler tick on each node
        ┌──────────────────────────────────▼─────────────────────────────┐
        │  ReindexProvider.StartTask (per node, per task)     ← UnitSt. │
        │   • processOneUnit per local unit, in a bounded worker pool   │
        │     (per unit: PENDING → IN_PROGRESS → COMPLETED on success)  │
        │   • Build ShardReindexTaskGeneric per (strategy, unit)        │
        │   • persistRecoveryRecord (payload.mig)                       │
        │   • RunReindexOnlyOnShard — iterate objects, write to         │
        │     __reindex_<N>/ bucket; install double-write callbacks     │
        │   • markReindexed → UnitStatus = COMPLETED  ← per-Unit status │
        └──────────────────────────────────┬─────────────────────────────┘
                              All units terminal across the cluster
                                           │
                          (semantic only — format-only skips to SWAPPING)
                                           │
                                  ┌────────▼────────┐
                                  │ PREPARING       │ ← TaskStatus
                                  └────────┬────────┘    (semantic only;
                                           │             format-only skips)
                                           │  scheduler fires per-node
        ┌──────────────────────────────────▼─────────────────────────────┐
        │  Provider.OnGroupCompleted (per-node, semantic only, barrier) │
        │   PHASE A — PREP per local shard, idempotent at merged.mig:   │
        │    1. PREP (background): FlushAndSwitch + Prepend             │
        │      (disk-I/O proportional to bucket size — minutes at       │
        │       billion-scale; this is what the barrier closes)         │
        │   RecordPreparationCompleteAck(success bool) — RAFT (per node)       │
        └──────────────────────────────────┬─────────────────────────────┘
                  Every node's PreparationCompleteAck landed (success on all)
                                           │   FSM gates the transition
                                  ┌────────▼────────┐
                                  │ SWAPPING        │ ← TaskStatus
                                  └────────┬────────┘   (the barrier
                                           │            bounded cross-node
                                           │            swap window is
                                           │            now ≈ tens of ms)
                                           │  scheduler fires per-node
        ┌──────────────────────────────────▼─────────────────────────────┐
        │  Provider.OnSwapRequested (per-node, semantic only, barrier)  │
        │   PHASE B — OVERLAY + SWAP per local shard:                   │
        │    2. OVERLAY SET: per-shard tokenization resolver            │
        │    3. ATOMIC SWAP: SwapBucketPointer per prop (microseconds)  │
        │    + post-atomic inline tidy (Shutdown + rename old → backup) │
        │    + OnMigrationComplete (per-strategy hook)                  │
        │   RecordPostCompletionAck(success bool) — RAFT (per node)     │
        │                                                                │
        │  (Format-only path: Provider.OnGroupCompleted runs the       │
        │   inline PREP+OVERLAY+SWAP body in a single callback; no      │
        │   PreparationCompleteAck barrier; SWAPPING fires directly from       │
        │   AllUnitsTerminal.)                                          │
        └──────────────────────────────────┬─────────────────────────────┘
              Every node's PostCompletionAck landed (success on all)
                                           │
        ┌──────────────────────────────────▼─────────────────────────────┐
        │  Provider.OnTaskCompleted (per-node, semantic only)           │
        │   flipSemanticMigrationSchema — RAFT UpdatePropertyInternal   │
        │   (idempotent: every node fires, first commit wins)           │
        │   ClearTokenizationOverlay on every loaded local shard        │
        └──────────────────────────────────┬─────────────────────────────┘
                                           │  scheduler marks finalized
                                  ┌────────▼────────┐
                                  │ FINISHED        │ ← TaskStatus
                                  └─────────────────┘   (operator-visible
                                           │            "fully done" signal)
                                           │  next process startup
        ┌──────────────────────────────────▼─────────────────────────────┐
        │  FinalizeCompletedMigrations (pre-LSM init)                   │
        │   For each (prop, indexType) with tidied.mig:                 │
        │     rename property_p_<idx>__<ingestSuffix>_<N>/              │
        │       → property_p_<idx>/                                     │
        │     remove backup dirs, lower-gen sidecars, tracker dir       │
        └────────────────────────────────────────────────────────────────┘

Format-only migrations (enable-rangeable, repair-filterable, repair-rangeable, rebuild-searchable) skip the OnGroupCompleted barrier — each shard runs the full lifecycle inside its own RunOnShard and there is no cluster-wide schema flip. The flow is otherwise identical.

What goes through RAFT vs. what is local-only

The diagram is easy to misread as "PREPARING and SWAPPING are cluster-wide synchronization moments; every node atomically swaps in lock-step". The truth is more nuanced — the FSM transitions ARE cluster-wide RAFT commits, but each node's local PREP and SWAP work runs on its own timeline. The barrier (PREPARING → SWAPPING) bounds the cross-node window where local timelines diverge.

What goes through RAFT (cluster-wide commits):

• AddTask — task created at STARTED.
• UpdateUnitProgress / RecordUnitCompletion — per unit, on the node that owns the unit.
• The transition STARTED → PREPARING (semantic) or STARTED → SWAPPING (format-only) — happens once the cluster-wide AllUnitsTerminal predicate becomes true on the Manager's FSM state. Routing depends on Task.NeedsPreparationBarrier.
• RecordPreparationCompleteAck — one per node, with Success=bool. Semantic-migration path only — fires after each node's local PREP body (OnGroupCompleted) returns.
• The transition PREPARING → SWAPPING — committed inside RecordPreparationCompleteAck's apply once every expected ack has landed with Success=true.
• RecordPostCompletionAck — one per node, with Success=bool. Fires after each node's local SWAP body (OnSwapRequested for barrier tasks, OnGroupCompleted for non-barrier tasks) returns.
• The transition SWAPPING → FINISHED — committed by MarkTaskFinalized once every expected PostCompletionAck has landed successfully.
• UpdateProperty from OnTaskCompleted — the cluster-wide schema flip (semantic migrations only).

What does NOT go through RAFT (local-only):

• PREP (FlushAndSwitch, Prepend, Shutdown the reindex bucket).
• OVERLAY SET (install per-shard tokenization resolver).
• ATOMIC SWAP (SwapBucketPointer per prop, in microseconds).
• Post-atomic inline tidy (Shutdown old main, rename old → backup).
• OnMigrationComplete per-strategy hook (in-memory shard-local state mutations).

All five of those are pure in-process operations on each node's local LSM store. There is no single instant where every node atomically swaps. Different nodes fire OnGroupCompleted at slightly different times (scheduler-tick jitter, typically sub-second to a few seconds). Each node runs its own three-phase dance on its own local timeline.

What "atomic" actually means

Two different scopes of atomicity carry weight here:

• Per-node atomic — inside RunSwapOnShard's Phase 2a, the per- prop SwapBucketPointer tight loop holds the mixed-state subwindow ("some props swapped, others not") to microseconds. A query landing on this node during this subwindow that hits a not-yet-swapped prop would tokenize input with the new value against an old-tokenized bucket; the loop staying microseconds ensures the probability is negligible. This is the meaning of "ATOMIC SWAP" in the diagram.
• Cluster-wide convergence (NOT instantaneous synchronization) — the system guarantees that either every node converges to the new bucket+schema state, or the migration fails cleanly and every node stays at the old state. It does NOT guarantee that nodes flip at the same wall-clock instant.

The cluster-wide convergence guarantee is enforced by two independent mechanisms:

1. Per-shard tokenization overlay (the per-node bridge). Between the local SwapBucketPointer (bucket is now NEW) and the eventual cluster-wide schema flip (schema flag is now NEW), this node's queries need to tokenize input matching the new bucket content. The overlay installs the new tokenization at the per-shard query path so the per-node window between "swap committed locally" and "schema flip committed cluster-wide" is correct on this node's reads. The overlay is cleared from OnTaskCompleted after the schema flip lands. See §10.
2. Two-phase ack barrier (the cross-node handshake). For semantic migrations each node submits RecordPreparationCompleteAck(Success=bool) after its local PREP returns, then RecordPostCompletionAck(Success=bool) after its local SWAP returns. The FSM gates PREPARING → SWAPPING on every node's PreparationCompleteAck (success on all), then gates SWAPPING → FINISHED on every node's PostCompletionAck (success on all). Any Success=false on either ack flips the task to FAILED, which makes OnTaskCompleted skip the cluster-wide schema flip. So the schema never moves to NEW unless every node has successfully prepared AND swapped. Format-only migrations skip the PreparationCompleteAck barrier (no cross-replica tokenization alignment to bound) and gate SWAPPING → FINISHED on PostCompletionAck only. See §6.3.

Why PREP runs inside PREPARING (not earlier)

Two reasons it can't move earlier into the STARTED phase:

• PREP requires the reindex bucket to exist on disk with complete segments. PrependSegmentsFromBucket(reindex → ingest) needs the reindex bucket fully populated. The reindex bucket is filled by RunReindexOnlyOnShard — which is the work that drives UnitStatus → COMPLETED. So PREP can't start before the unit hits COMPLETED on this node.
• Running PREP at per-unit completion time (still during STARTED on the cluster level) would mean each node starts heavy disk I/O the moment its OWN units finish, while other nodes are still reindexing. That doubles the cluster's I/O profile during the long phase. Concentrating PREP inside the per-node OnGroupCompleted (PHASE A) keeps the resource curve well-defined: STARTED = "everyone reindexing", PREPARING = "everyone running PREP locally then acking", SWAPPING = "everyone running atomic SWAP locally then acking".

The role PREPARING plays is twofold: (a) signaling that every unit is terminal, so every node has the right on-disk state to start its local PREP; and (b) gating the cross-replica SWAP barrier — only after every node has acked PreparationCompleteAck does any node proceed to PHASE B (OnSwapRequested). This is what bounds the cross-replica stagger window to RAFT propagation latency instead of per-node PREP duration variance. The synchronize-or-fail-cleanly invariant lives in the ack barriers that follow.

4. Architecture by layer

4.1 REST layer — adapters/handlers/rest/handlers_indexes.go + handlers_reindex.go

Validation, dispatch, response shaping. Two structural details worth calling out:

Per-(collection, property) submit lock. Held across class read, validation, conflict check, and the RAFT AddDistributedTaskWithBarrier (or AddDistributedTaskWithGroupsBarrier for a multi-tenant semantic migration) call. Without this, a parallel DELETE /properties/{p}/index/{name} could win the lock between the PUT's ReadOnlyClass snapshot and its RAFT task-add, leaving the PUT validating against a schema that no longer matches the on-disk bucket. The lock is process-wide; the multi-node version of the same race is closed by the FSM-side MutationGuard (see §4.3). Stored on state.ReindexSubmitLocks so the PUT handler and the DELETE-property-index handler share the same entries. See adapters/handlers/rest/state/reindex_submit_locks.go.

Pre-submit DB.NewStalePartialReindexSweep sweep. Defense in depth against the CANCEL→retry silent-failure family (same shape as DELETE→re-enable): if a previous cancelled run left stale .migrations/<dir>/started.mig + sidecar dirs on disk, the new task would resume against them, finish in <1s with a no-op, flip the schema flag, and report success against an empty bucket. The cancel handler already runs this synchronously, but the wait can time out or the node can crash mid-cancel. Submit-time cleanup catches that gap. Critically, for change-tokenization the sweep wipes BOTH searchable AND filterable migration dirs — cleaning only one of them was the root cause of a Sev 1 (see indexTypesFromMigrationType's godoc).

4.2 Distributed Task Manager — cluster/distributedtask/

The substrate. Package doc at cluster/distributedtask/doc.go documents the four "journey" shapes; reindex uses Journey 3 for semantic migrations and Journey 2 for format-only ones, plus Journey 4 for per-tenant grouped MT.

Key types & contracts:

• Manager — the RAFT FSM. Owns task state, applies AddTask, UpdateUnitProgress, RecordUnitCompletion, RecordPreparationCompleteAck, RecordPostCompletionAck, MarkTaskFinalized, CancelTask, CleanUpTask. Every mutation is FSM-deterministic.
• Scheduler — per-node loop. Polls Manager for current task list, starts/stops local work via Provider, fires OnGroupCompleted (PHASE A: PREP for barrier tasks, PREP+SWAP for non-barrier), fires OnSwapRequested (PHASE B: SWAP for barrier tasks), OnTaskCompleted (cluster-wide schema flip), submits MarkTaskFinalized when the local callbacks succeed.
• Provider / UnitAwareProvider — the extension point. ReindexProvider implements the latter and provides both OnGroupCompleted and OnSwapRequested for barrier-aware semantic migrations.
• ConflictDetector — pluggable interface implemented per namespace (here, by ReindexProvider.CheckConflict). Called under Manager's lock from the AddTask apply path, BEFORE the task is appended. FSM-deterministic.
• SchemaMutationDetector — the other half of the conflict surface: gates external schema mutations (UpdateProperty/DeleteClass/DeleteTenants/UpdateTenants) while a reindex is in flight. Implementation: ReindexProvider.CheckPropertyUpdate / CheckClassMutation / CheckTenantMutation. Uses TaskStatus.IsActive(), so every non-terminal status counts as "in flight" for mutation gating (see Task status classification below).
• TaskStatusPreparing and TaskStatusSwapping — the post-units, pre-FINISHED coordination states that split per-node PREP from per-node SWAP with a cluster-wide PreparationCompleteAck barrier in between. Semantic migrations transit STARTED → PREPARING → SWAPPING → FINISHED; format-only migrations skip PREPARING and transit STARTED → SWAPPING → FINISHED. The task is NOT safe to act on from the API surface in either coordination state; callers polling for "fully done" must wait for FINISHED. Format-only journeys pass through SWAPPING in essentially zero time. These states close the schema-flip-lag race where a task would otherwise be FINISHED at the FSM layer before every node's post-completion callback had committed its bucket-pointer flip, and the two-phase split additionally bounds the cross-replica stagger window to RAFT propagation latency (tens of ms) instead of per-node PREP duration variance (which scales with bucket size and reaches minutes at billion-scale).
• Two-phase ack barrier — RecordPreparationCompleteAck (semantic only)
  • RecordPostCompletionAck (all paths). Every node's scheduler records its phase outcome (success or error) on the task before the cluster is allowed to advance: PREPARING → SWAPPING is gated on every PreparationCompleteAck landing with Success=true; SWAPPING → FINISHED is gated on every PostCompletionAck landing with Success=true. A Success=false on EITHER ack flips the task to FAILED, which makes OnTaskCompleted skip the cluster-wide schema flip — the load-bearing invariant that prevents a per-node swap failure from leaving the schema pointing at not-yet-swapped buckets. Acks idempotent and rehydrate over restart.
• Permanent-rejection sentinels — ErrTaskNotRunning/ErrTaskDoesNotExist/ErrUnitAlreadyTerminal/ ErrUnitWrongNode/ErrTaskNotInFinalizingState, all matched by the umbrella ErrPermanentRejection. Encoded over gRPC with a stable codes.FailedPrecondition + [dtm-perm/<id>] ... message marker so the sentinel identity survives wire transit and gets re-attached on the receiving side by RehydratePermanentRejection. Mixed-version friendly (pre-sentinel peers return plain-text phrasing; the classifier substring-matches as a fallback). See cluster/distributedtask/errors.go.

Task status classification

TaskStatus is classified by five predicates in cluster/distributedtask/types.go:

IsCancellable() is a literal comparison rather than a classification, which is what makes it different from the other four. Manager.CancelTask is an FSM apply, so every binary that will ever replay the entry has to reach the same verdict, including one that cannot name the status. Classifying would let a node that has never heard of a status cancel a migration a newer node is still coordinating, and follower apply errors are discarded, so the divergence would be silent.

Because IsActive() is the negation of IsTerminal() rather than a list of known in-flight statuses, a status this build does not recognize counts as in flight. That is the case an older node hits when a newer release introduces a status and the cluster is mid rolling-upgrade. IsRecognized() deliberately has no default case, so the exhaustive linter fails until a newly added status is classified here.

The default is fail-closed on purpose: reading an unknown status as "done" would admit a second migration onto a property a newer node is still migrating, and would let the orphan audit and the TTL sweep delete live state. It costs availability instead. For as long as such a task is in the node's state, the node:

• rejects schema mutations overlapping the task's properties, and rejects new reindex submits that overlap them (CheckPropertyUpdate, CheckClassMutation, CheckTenantMutation, CheckConflict);
• refuses backups on the shards the task lists (db.NewShardReindexActivityLookup feeding DB.AnyLiveReindexForShard);
• reports the property's index as indexing on GET .../indexes rather than ready or pending, since the per-unit progress does not prove that no shard has started;
• keeps the task's on-disk tracker dirs, because the orphan audit reads the task as live (db.NewLiveReindexTrackerLookup);
• refuses a cancel with 409, at the REST pre-flight and again at Manager.CancelTask. IsCancellable() is STARTED only, so an unrecognized status is not cancellable. The mutation-refusal messages say so directly: for this case they drop the usual "cancel it" advice and tell the operator the task has to reach a terminal state on the nodes that do recognize the status (MutationRemedy in reindex_conflict.go).

The two lookups both route through db.IsLiveReindexTaskStatus, so the rule for a new status is answered in one place.

TTL cleanup is the one exit. Manager.CleanUpTask refuses a task that is both active and recognized, so an unrecognized-status task is deleted once its completedTaskTTL has elapsed. That is sound because the Scheduler TTL sweep is the only proposer of CLEAN_UP (pinned by TestStructuralInvariant_TTLSweepIsTheOnlyCleanUpProposer) and it filters on IsActive() against the list it read from the leader. So a CLEAN_UP naming a status this build cannot classify can only come from a node that does recognize it and reads it as terminal, which is exactly when deleting the local copy is right. Without the exit the entry is unreachable forever: no transition can advance it (MarkTaskFinalized refuses every status but FINISHED and SWAPPING), no later sweep on this node proposes it, and it keeps blocking schema mutations and backups on its collection through the local map.

Scheduler.warnOnUnrecognizedStatuses names every task in an unrecognized status on one aggregated warn line, sampled at three lines an hour. The condition holds until an operator acts on it, so the log is the diagnostic and not the signal: the per-namespace gauge weaviate_distributed_tasks_unrecognized_status is written on every tick, which is what makes the state alertable. The line does not offer a cancel — the FSM refuses one for exactly this task set.

The cost of the two directions is asymmetric, which is what governs how new statuses get introduced. Adding a new terminal status is only safe once every version in the supported upgrade and rollback range recognizes it: a node that does not reads a finished task as in flight, dropping schema mutations the rest of the cluster has already committed. That window normally closes at completedTaskTTL — a node that does recognize the status proposes the clean-up, the entry replicates, and the unrecognized arm of CleanUpTask above deletes the old node's copy. Three cases have no such bound: a full rollback, where no node recognizes the status so nothing ever proposes a clean-up; a node that misses the clean-up entry and is caught up by a snapshot instead, because Manager.Restore merges into the existing task map rather than replacing it (weaviate/0-weaviate-issues#245); and a node whose completedTaskTTL is longer than the proposer's, because CleanUpTask re-checks the age against that node's own clock and TTL inside the apply, and an apply that refuses is logged and dropped rather than retried. Adding a new non-terminal status is cheaper, because the fail-closed reading is the correct one, but it is not free: such a node still refuses backups on the collection and reports the index as indexing.

4.3 Schema FSM — cluster/schema/ + cluster/proto/api/

Three changes here serve the reindex feature:

BucketGeneration counter on Property. Bumped on every reindex that touches a property. Lets unrelated machinery (notably backup) tell "this is the same bucket I saw before" apart from "this property has been rebuilt since I last looked."

UpdateProperty fieldmask. applyPerPropertySchemaUpdate passes a fields []string mask all the way down to the FSM apply path (MergePropsMasked). Two strategies running in parallel on the same property — each touching different fields — no longer clobber each other on RAFT apply: the FSM merges only the listed fields onto the live class state. An empty mask falls back to "replace every field" semantics for callers that don't care.

FromInFlightMigration flag. Routed via UpdatePropertyInternalFromMigration. Bypasses the MutationGuard described next: migration-driven schema flips are exactly the kind of "in-flight" mutation the guard would otherwise reject, so they need an explicit opt-out signal that's set only by the provider's OnTaskCompleted path.

MutationGuard (cross-FSM). The schema FSM consults distributedtask.SchemaMutationDetector on every UpdateProperty, DeleteClass, UpdateTenants(FROZEN), DeleteTenants apply. ReindexProvider's implementation rejects any mutation overlapping the properties of an in-flight reindex task on the same collection. In flight means any non-terminal status, admitted via TaskStatus.IsActive() (see §4.2). The motivating failure mode is documented verbatim on CheckPropertyUpdate's godoc: a change-tokenization migration spawns separate per-shard sub-tasks for searchable and filterable; a DELETE arrives mid-flight; cleanStaleMigrationDirs wipes the searchable sub-task's working dir; the searchable sub-unit FAILs; the filterable sub-unit commits its local swap; per-shard ack barrier sees mixed acks; task FAILED; flipSemanticMigrationSchema skipped; schema stays at OLD tokenization while the filterable bucket on disk holds NEW-tokenized data. Bucket↔schema inversion. Same family as the ack-barrier failure mode but triggered by an external schema mutation instead of a crash.

The guard is intentionally blanket: any overlap rejects. Migration- driven flips bypass via FromInFlightMigration=true so the migration's own scheduled completion flip still works. Class-wide (CheckClassMutation) is stricter — any reindex on the class (regardless of property) blocks DeleteClass.

4.4 Reindex orchestration — adapters/repos/db/reindex_* & inverted_reindex_*

ReindexProvider (reindex_provider.go) — the DTM UnitAwareProvider implementation. Lifecycle hooks:

• StartTask → per-node bootstrap. Unmarshals the payload, identifies this node's local units, hands them to processUnits. Bounded concurrency via ConcurrencyLimiter.
• processOneUnit → per-(unit, shard) bootstrap. Constructs the strategy instance(s) at the right per-node generation (nextMigrationGeneration), writes the recovery payload, runs the reindex iteration via ShardReindexTaskGeneric. For semantic migrations it stops at markReindexed (barrier); for format-only it runs the full lifecycle including the swap.
• OnGroupCompleted (semantic only) → the swap phase, per local shard. Three-phase: PREP → OVERLAY SET → ATOMIC SWAP. See §6.
• OnTaskCompleted (semantic only) → flipSemanticMigrationSchema via RAFT, then ClearTokenizationOverlay on every loaded local shard (an unloaded shard holds no in-memory overlay).
• CheckConflict / CheckPropertyUpdate / CheckClassMutation / CheckTenantMutation — see §4.3 & §7.

ShardReindexTaskGeneric (inverted_reindex_task_generic.go) — the strategy-parameterized lifecycle. State machine, merge / swap / tidy, object iteration, progress tracking, sentinel writes. The file-level godoc documents the three-phase contract (PREP / ATOMIC / DEFERRED) that every code path must preserve.

MigrationStrategy (inverted_reindex_strategy.go) — the per-migration extension surface. Each strategy supplies bucket naming, the per-key transform, source/target/backup LSM strategies, the Add/Delete double-write callbacks, the optional AnalyzerOverlay (see §8), the PreReindexHook, and the OnMigrationComplete hook (see §4.5 phase contract).

reindex_conflict.go — CheckConflict (FSM-deterministic), CheckPropertyUpdate, CheckClassMutation, CheckTenantMutation, plus the predicates ReindexPropsOverlap, TouchesSearchable, TouchesFilterable, TypesConflictReason. The exhaustive switches in the Touches* predicates intentionally panic on unknown migration types so a new ReindexMigrationType cannot silently be treated as "doesn't touch X" — it surfaces on the first request.

reindex_recovery.go — DiscoverInFlightReindexTasks, buildRecoveryTasks, NewShardReindexerV3FromRecovered, SeedReindexProviderFromRecovery. Called from MakeAppState BEFORE DB.WaitForStartup, so reconstructed ShardReindexTaskGeneric instances are registered before any post-restart write can reach the shard. See §6 crash safety.

reindex_cancel_cleanup.go — DB.NewStalePartialReindexSweep() returns a func(ctx, collection, prop, indexType) error that fans out to Shard.CleanStalePartialReindexState per shard. Called from the cancel handler (after WaitForLocalTaskDrain), from the submit handler (defense in depth), and from autoCleanupAfterTerminal on every node once a task reaches FAILED or CANCELLED. Per-shard failures don't stop iteration so a stuck shard can't permanently wedge a (collection, prop, indexType) tuple. One sweep serves a whole call so an unloaded shard's directory listing is read once across the index types a single migration touches; a loaded shard's sweep always reads the filesystem directly and never acts on the cached snapshot.

inverted_reindex_finalize.go — startup-time deferred dir rename (see §9), nextMigrationGeneration, maxMigrationGeneration, completedMigrationGens (parseMigrationDirName lives in inverted_reindex_strategy_dir_names.go). The finalize algorithm handles every shape defensively: tidied / merged-but-not- tidied / lower-gen sidecars / in-flight gens left alone for DiscoverInFlightReindexTasks to pick up.

4.5 Strategy implementations — inverted_reindex_strategy_*.go

Eight strategy implementations, one file each:

change-tokenization spawns TWO strategy instances per unit (SearchableRetokenizeStrategy + FilterableRetokenizeStrategy) so the searchable + filterable buckets retokenize in lock-step, with their per-shard swaps inside the same tokenization-overlay window. Per-shard cleanup (indexTypesFromMigrationType) must wipe BOTH tracker dirs — see §4.1.

These instances are constructed and parameterized by ShardReindexTaskGeneric (the generic V3 task lifecycle), so all strategies share a single state machine, sentinel writer, callback manager, and progress tracker.

The strategy interface itself documents the per-method contract; see inverted_reindex_strategy.go. Of particular note is OnMigrationComplete's phase contract — it fires in Phase 2c, AFTER the per-prop SwapBucketPointer tight loop and AFTER the inline oldMain.Shutdown + oldMain → backup rename loop, but still INSIDE the per-shard tokenization-overlay window for migrations that use one. The godoc enumerates what's allowed and forbidden in that position and is the authoritative spec for adding a new strategy.

Semantic vs format-only. IsSemanticMigration is the predicate: change-tokenization, change-tokenization-filterable, enable-filterable, enable-searchable, and change-algorithm are semantic. Every shard must reindex before any shard swaps (Journey 3 barrier), and the schema flip happens cluster-wide from OnTaskCompleted. The rest are format-only: each shard runs the full lifecycle independently (Journey 2), with no cluster-wide schema dependency.

enable-rangeable is intentionally format-only. Range queries' correctness during the migration is gated by the per-shard rangeableLocalReady flag — falling back to the filterable bucket walk on shards that haven't completed locally is slow but correct. The barrier dance would be over-engineering for a journey that has a correct (if slow) per-shard fallback.

4.6 LSM primitives — adapters/repos/db/lsmkv/store.go

Two primitives carry the load:

Store.SwapBucketPointer(ctx, targetName, sourceName). Atomic in-memory pointer flip — all future Store.Bucket(targetName) calls return the bucket currently registered as sourceName. The source name is removed from the map; the source bucket's on-disk path is released from GlobalBucketRegistry. The caller is responsible for shutting down the returned old bucket, persisting any crash-safety markers around this call, and finalizing directory renames at a later point (typically next restart).

The registry release is what makes back-to-back migrations in the same process work: a second migration's ingest bucket can claim the same canonical path after the on-disk dir has been cleaned by cleanStaleSidecarDirs. Without the release, the second cycle aborts at OnAfterLsmInit with "bucket already registered".

Store.FinalizeBucketSwap(ctx, bucketName, canonicalDir, currentDir, backupDir). The deferred-finalize counterpart: flush memtable, remove backup dir, os.Rename(currentDir → canonicalDir), rewrite bucket.dir + bucket.disk.dir + every segment's in-memory .path, create a fresh active memtable. MUST only be called during startup, before the bucket serves any queries. Calling on a live bucket creates a race where step 3 (os.Rename) and step 4 (updateBucketDir) form a non-atomic window; a concurrent compaction reading bucket.disk.dir sees the old path and post-rename writes fail with ENOENT. See §9 for the full history.

The atomic-phase contract in the orchestrator file enforces this rule: a unit test fails if SwapBucketPointer is preceded by any disk-I/O or compaction-wait op inside Phase 2 (ShardReindexTaskGeneric.processOneSwapPropFn + wall-clock budget assertion).

5. Migration strategies — quick map

                               semantic?   tokenization    analyzer
                                           overlay (§10)   overlay (§11)
change-tokenization              ✓              ✓
change-tokenization-filterable   ✓              ✓
enable-filterable                ✓                              ✓
enable-searchable                ✓                              ✓
change-algorithm                 ✓
enable-rangeable                                                ✓
repair-rangeable                                                ✓
repair-filterable
rebuild-searchable

The five semantic migrations take the cluster-wide barrier and the cluster-wide schema flip; the four format-only ones take neither. The two overlay columns are independent of that split. The tokenization overlay covers the per-shard window on a migration that changes tokenization (IsTokenizationChangingMigration); the analyzer overlay lets a from-scratch build see a property whose schema flag is still false (MigrationStrategy.AnalyzerOverlay). change-algorithm takes neither: it only swaps the searchable bucket strategy.

6. Crash safety

6.1 Sentinel files

Every per-shard migration owns a tracker dir under <lsm>/.migrations/<strategy-prefix>_<propname-suffix>_<gen>/. Phase transitions write fsync'd sentinel files:

Per-prop variants exist for swapped.mig (one per property) so a crash mid Phase 2 can resume from the last successfully-swapped prop.

6.2 Recovery — startup DB init

Sequence in MakeAppState:

1. DiscoverInFlightReindexTasks(rootPath) walks every shard's .migrations/ dir. For each tracker dir with started.mig + reindexed.mig present and tidied.mig absent, loads the persisted payload.mig and reconstructs a per-shard ShardReindexTaskGeneric at the correct generation. The narrow window — "terminal but not yet tidied" — is exactly the recovery gap the design exists to close.
2. NewShardReindexerV3FromRecovered wires the recovered tasks into a stripped-down recovery-only ShardReindexerV3 that fires OnAfterLsmInit per shard load — re-installing the double-write callbacks BEFORE any post-restart write can reach the shard. Without this, writes that arrive between shard init and the swap that completes a deferred reindex go only to the old main bucket and are lost when the swap replaces it with the ingest bucket.
3. SeedReindexProviderFromRecovery pre-populates the provider's per-descriptor task cache so OnGroupCompleted reuses the same instances rather than creating fresh ones and calling OnAfterLsmInit a second time (which would attempt to load already-loaded ingest buckets).
4. FinalizeCompletedMigrations runs per shard before LSM init reloads any buckets. For each (prop, indexType), finds the highest tidied (or merged-but-not-tidied) generation, promotes its ingest dir to the canonical name, deletes lower-gen sidecars, deletes the tracker dir. See §9 for the multi-gen algorithm.

6.3 The two-phase ack barrier (PREPARING + SWAPPING)

The post-completion barrier is split into two phases.

Semantic migrations (NeedsPreparationBarrier=true):

1. OnGroupCompleted (PHASE A) runs PREP per local shard. Returns a non-nil error iff any task in the group failed to merge. The scheduler emits RecordPreparationCompleteAck(Success=bool) per node.
2. FSM transitions PREPARING → SWAPPING only when every expected PreparationCompleteAck has landed with Success=true. The transition is committed inside the FSM apply path (atomic) so no node can advance to SWAPPING before every node has finished PREP.
3. OnSwapRequested (PHASE B) runs OVERLAY+SWAP per local shard. Returns a non-nil error iff any task's RunSwapOnShard failed. The scheduler emits RecordPostCompletionAck(Success=bool) per node.
4. OnTaskCompleted fires → cluster-wide schema flip commits. The scheduler runs this phase (runCompletedCallbackPhase) before it proposes finalization.
5. FSM transitions SWAPPING → FINISHED only when every expected PostCompletionAck has landed with Success=true AND this node's OnTaskCompleted has fired — runFinalizePhase skips any task whose completedCallbackFired is unset.

Format-only migrations (NeedsPreparationBarrier=false): PHASE A is skipped; the FSM goes STARTED → SWAPPING directly. OnGroupCompleted runs the inline PREP+OVERLAY+SWAP body and the scheduler emits RecordPostCompletionAck. SWAPPING → FINISHED is gated on the PostCompletionAck barrier only.

Failure handling (both paths):

• Any Success=false on EITHER ack → task flips to FAILED immediately → OnTaskCompleted early-returns, schema flip is SKIPPED.
• Per-node swap failure can no longer let the cluster-wide schema flip propagate while one replica's bucket pointer never moved.
• On node restart, the recovery path re-fires the appropriate phase callback (PHASE A for PREPARING, PHASE B for SWAPPING) via the rehydrate branch; if the per-shard work succeeds the second time, the ack lands and the task completes normally.

Cross-replica window: The time during which different nodes' buckets are in mixed-tokenization state (some swapped, some not) is bounded by RAFT propagation latency between PHASE B firing on the fastest-node and PHASE B firing on the slowest-node — tens of milliseconds regardless of PREP duration. Decoupling SWAP from PREP across nodes is the reason this window stays bounded even when PREP runs minutes per shard at billion-scale.

Why two phases instead of one — the empirical anchor: at 1M-object scale on a 3-node × RF=1 × 5-shard cluster, QA observed a ~1-second total in-flight window with a 248 ms within-window mixed-state captured between two probe samples on adjacent shards (see weaviate/0-weaviate-issues#225). At that scale the per-node PREP duration variance dominates: with no barrier, the cross-replica mixed-state window scales with PREP duration (minutes at billion-scale); with the barrier, it scales with RAFT propagation (milliseconds). The cost is one extra cluster-wide RAFT roundtrip per semantic migration. Without this anchor, future maintainers reading the FSM table see "two RAFT-coordinated phases" and may propose collapsing it back to one — the bug it closes is not visually obvious from the code alone.

Acks idempotent (first ack per (task, node) wins); rehydrate over restart (the scheduler re-fires on the next tick); silent on already- terminal tasks (a late-arriving ack from a slow follower must not produce a noisy apply failure).

6.4 The three-phase swap

OnGroupCompleted runs a strict prep / atomic / defer split per local shard. The contract is enumerated verbatim at the top of inverted_reindex_task_generic.go; in short:

• Phase 1 — PREP (background, NOT inside the overlay window): FlushAndSwitch reindex bucket, Shutdown it, PrependSegmentsFromBucket per property, removeReindexBucketsDirs, sentinel writes. Disk-I/O- heavy. Schema = OLD, bucket = OLD throughout — safe with live queries.
• Phase 2a — ATOMIC SWAP (microseconds, inside overlay window): per prop, Store.SwapBucketPointer(ctx, mainName, ingestName) followed by markSwappedProp. Tight loop. Bounds the per-shard "mixed-state" subwindow (some props swapped, others not) to a few microseconds total — queries during this subwindow that hit not-yet-swapped props would tokenize input with the new value against an old-tokenized bucket. Must stay microseconds.
• Phase 2b — POST-ATOMIC INLINE TIDY (slow but correctness-safe): oldMainBucket.Shutdown(ctx) + os.Rename(oldMainDir, backupDir) per property, then markSwapped + markTidied. After every prop has flipped in 2a, the mixed-state subwindow is closed; queries during 2b see all-new buckets with the overlay still active. The oldMain.Shutdown is REQUIRED inline (not deferred) because Bucket.Shutdown is the only call that removes the bucket's path from GlobalBucketRegistry; deferring leaks the path entry process-wide.
• Phase 2c — POST-ATOMIC INLINE FINALIZE: OnMigrationComplete + trimOlderGenerationsLocked. Outside the mixed-state subwindow. Strategy-specific hook for in-memory shard-local query-path state mutations (e.g. setRangeableLocallyReady) or for non-semantic RAFT calls.
• Phase 3 — DEFERRED LIVE-BUCKET RENAME (next process startup): FinalizeCompletedMigrations runs the ingest → canonical dir rename before LSM init reloads any bucket. See §9.

The PREP / OVERLAY-SET / ATOMIC-SWAP ordering is what closes the SWAPPING-window misalignment. Setting the overlay before prep would expose the very gap it closes (NEW-tokenized analyzer input against OLD-tokenized bucket content for hundreds of ms). Setting it between prep and atomic swap means the window is bounded by the in-memory pointer flip (microseconds).

Under the two-phase barrier (semantic migrations), PREP and OVERLAY SET + ATOMIC SWAP fire from two different scheduler callbacks (OnGroupCompleted for PREP, OnSwapRequested for OVERLAY+SWAP) with the cluster-wide RecordPreparationCompleteAck barrier in between. The phase ordering on a single node is preserved; the barrier additionally bounds the cross-node skew between fastest-node SWAP and slowest-node SWAP to RAFT propagation latency instead of per-node PREP duration variance.

7. Concurrency model

Per-(class, property) REST submit lock. Closes the same-process PUT/DELETE race. See §4.1.

Cluster-wide FSM conflict check (ConflictDetector.CheckConflict). Called under Manager.mu from the RAFT-apply AddTask path BEFORE the new task is appended to FSM-stored state. FSM-deterministic: every node applies the same RAFT log entry, sees the same existingTasks snapshot, runs the same predicate, reaches the same accept/reject decision. Survives leader re-election.

Conflict rule: any two reindex migrations on overlapping properties of the same collection conflict, regardless of which bucket type they primarily write to. Earlier versions allowed parallel migrations as long as they wrote to different bucket types; that turned out to be a real Sev 1, because the completing migration's OnMigrationComplete fires an UpdateProperty whose MergeProps preserved the still-false sibling flag (the other migration hadn't flipped its flag yet), and the FSM apply path then ran cleanStaleMigrationDirs for every index whose flag was now false — wiping the in-flight migration's working directory. The closure happens at submit time: reject any new task whose property set overlaps an in-flight task's, so the caller gets a clean conflict error and can serialize. Every non-terminal status counts as in-flight (via TaskStatus.IsActive()), PREPARING and SWAPPING included; see §4.2.

Cluster-wide FSM schema-mutation check (SchemaMutationDetector). See §4.3. Blanket reject any external mutation overlapping an in-flight task's properties on the same collection. Migration-driven flips bypass via FromInFlightMigration.

Per-collection in-flight cap: 32 (maxConcurrentReindexPerCollection). Bumped from the original DTM default of 4 specifically for this feature — wide schemas where every property needs a semantic migration must complete in operator-tractable time, and serializing them at 4 made multi-hour migrations into multi-week migrations. 32 was chosen empirically as the point where LSM compaction throughput saturates on a single shard's disk for the typical migration mix; the REST handler returns 429 once the cap is reached (reindexCapExceededResponder; on this verb 503 instead means an in-flight task carries a payload this build cannot decode).

Per-collection worker pool in processUnits. Bounded by the concurrency function passed to the provider (typically a runtime.DynamicValue), so operators can throttle on overloaded clusters.

8. Multi-tenancy

?tenants=t1,t2 scopes the task to the named tenants on a multi- tenant class. The handler validates:

• tenants on a single-tenant class → 400.
• No tenants on a multi-tenant class → defaults to all tenants, for semantic and format-only migrations alike. A multi-tenant semantic migration takes the per-tenant group barrier (AddDistributedTaskWithGroupsBarrier); see below.
• tenants with a semantic migration → 400 ("all tenants must be targeted").
• Tenant in OFFLOADED / FROZEN → 400 with the named tenant.

Per-tenant unit groups (Journey 4 from the DTM doc): one UnitSpec.GroupID per tenant, so OnGroupCompleted fires per-tenant as each tenant's replicas all finish. Tenant A starts serving new data immediately even while tenant B is still reindexing.

Status after cancelling a tenant-scoped task is reported at the collection level: GET /v1/schema/{className}/indexes renders cancelled for the property as a whole. The task is the unit of identity, so there is no per-tenant index-status surface.

FROZEN tenants are rejected by the MutationGuard. A reindex on a tenant that transitions to FROZEN mid-flight fails the affected unit; the post-completion ack barrier propagates that failure to the task. Operator can resume by un-freezing the tenant (transition back to HOT) and re-submitting.

9. Per-migration generation + deferred live-bucket rename

The original design problem this section answers:

The natural finalize step after a successful in-memory swap is to rename the new bucket's on-disk dir from …__<ingestSuffix>_<N>/ to canonical property_<p>_<index>/. That rename is not safe at runtime (segments mid-flight hold path snapshots; renaming under them produces ENOENT on the next write).

The solution has two parts: defer the rename to next startup, and give every migration a per-node generation suffix so back-to-back migrations on the same property don't collide.

9.1 Why the rename is deferred

After a successful runtimeSwap on each node:

• bucketsByName[mainName] → the ingest bucket instance.
• That bucket's dir still points at property_<p>_<index>__<ingestSuffix>_<N>/; no filesystem rename.
• The old main has been Shutdown and its dir renamed to …__<backupSuffix>_<N>/.

Store.FinalizeBucketSwap would rename ingest → canonical, but as its godoc spells out, it MUST only be called during startup, before the bucket serves any queries. Calling on a live bucket creates a non-atomic window between os.Rename (step 3) and updateBucketDir (step 4) where a concurrent compaction reading bucket.disk.dir sees the old path; the next write fails with ENOENT (rename ...l0.s5.db ...deleteme: no such file is the production-observed shape).

Pausing compactions doesn't fix it cleanly: reads in flight also hold paths from the consistent-view-of-segments snapshot, and any defer- callback that touches bucket.disk.dir mid-update would crash or scribble onto the wrong path.

The cost-benefit is wrong anyway: the rename is purely a tidiness step. The bucket already serves correct data from the ingest-named dir; the rename only matters across a process restart, so deferring to next-startup FinalizeCompletedMigrations (when no buckets are loaded) is correct and zero-risk.

9.2 Why per-migration generation suffixes

Once the rename is deferred, a second problem appears: back-to-back migrations on the same property. After T_N tidied, bucketsByName[mainName] points at the gen-N ingest bucket and that bucket's dir is …__<ingestSuffix>_<N>/. The next migration T_(N+1) would, without a generation suffix, try to create a new ingest bucket at the same path the live main is currently serving from → GlobalBucketRegistry collision or two *Bucket structs pointing at the same physical directory.

Each migration takes its own gen-suffixed sidecars:

T_(N+1) on prop=text:
  reindex dir : property_text_searchable__retokenize_reindex_<N+1>
  ingest dir  : property_text_searchable__retokenize_ingest_<N+1>
  backup dir  : property_text_searchable__retokenize_backup_<N+1>
  tracker dir : .migrations/searchable_retokenize_text_<N+1>

No path collision with the gen-N state still on disk. T_(N+1)'s runtimeSwap replaces the gen-N pointer with the gen-N+1 one; the old gen-N bucket is shut down and renamed to its gen-N+1 backup.

9.3 Generation is per-node, not in the RAFT payload

Each node computes its own gen by scanning its own disk. The RAFT payload does NOT carry the gen. Different nodes may use different gens for the same RAFT task — and that's correct: gen is purely a per-node implementation detail of the deferred-finalize. A node that restarted between T_N and T_(N+1) will have promoted gen N to canonical at startup, so on that node T_(N+1) picks gen 1; a node that didn't restart picks gen N+1. The cluster-wide logical state still converges via the regular swap-then-flip pipeline.

9.4 Trim at end of swap keeps depth bounded

After T_N tidies, the per-shard runtimeSwap does an in-process trim that deletes every older gen's sidecar dirs (reindex / ingest / backup) and tracker dir. The invariant: at any time, on disk for any (prop, indexType), there is at most one tidied generation plus at most one in-flight generation (the trim runs at end of swap, so two tidied gens can only coexist if the trim crashed between markTidied and its os.RemoveAll loop).

9.5 FinalizeCompletedMigrations handles every shape

Per namespace (strategy-prefix + props-suffix):

1. Find the highest gen T with tidied.mig.
2. Find the highest gen M with merged.mig (regardless of tidied).
3. effective = max(T, M).
4. If effective exists:
   • effective == T: standard path. Rename …_<ingestSuffix>_<T>/ → property_<prop>_<index>/, remove …_<backupSuffix>_<T>/.
   • effective == M > T: recovery path. The in-process swap on this node crashed AFTER markMerged but BEFORE markTidied. The ingest dir at gen M holds the target-tokenization data the schema expects (the cluster-wide schema flip in flipSemanticMigrationSchema likely already committed via RAFT, since the DTM task was FINISHED before this node died). Write swapped.mig + tidied.mig sentinels and promote the same way. CRITICAL: otherwise this node serves the old data under the new schema → divergence vs other replicas → #10675-shape bug.
5. Remove every dir on disk with gen < effective.
6. Remove the tracker dir for effective.
7. Leave gens > effective alone (in-flight; DiscoverInFlightReindexTasks handles them).

9.6 Hard rules

• Do not call Store.FinalizeBucketSwap at runtime. Single biggest landmine in the reindex code path; every previous attempt to "clean up the cosmetic naming" has produced a Sev 1.
• Do not rename the live main bucket's physical directory while loaded, by any means. The mmap'd segments survive os.RemoveAll (POSIX unlink-while-open) and can serve reads from cached pages, but new segment writes will land in a missing dir and silently lose data. weaviate/weaviate#10675 is exactly this failure mode.
• Do not put the gen in the RAFT payload. The whole point of the deferred-finalize design is that each node's on-disk state is its own — forcing cluster-wide agreement on a per-node implementation detail would re-introduce the collisions the per-node gen was created to avoid.

10. Per-shard tokenization overlay

Defined in adapters/repos/db/inverted/tokenization.go.

The problem: on each replica, the bucket pointer flips to NEW-tokenized data at the end of RunSwapOnShard (Phase 2a), but the cluster-wide schema flip (OnTaskCompleted's RAFT commit) doesn't propagate to the local FSM for tens to hundreds of milliseconds afterward. Queries that arrive in that window tokenize their input against the still-OLD schema value and search a NEW-tokenized bucket — wrong results.

The overlay closes this. Shard.SetTokenizationOverlay(prop, newTok) installs a per-shard map; every query path that needs a property's tokenization calls inverted.ResolveTokenization(b.tokResolver, prop.Name, prop.Tokenization), which consults the overlay before falling back to the schema value. The searcher's tokResolver is Shard.TokenizationFor, attached via WithTokenizationResolver wherever the shard builds a Searcher or BM25Searcher.

Lifecycle:

1. Set — maybeWirePerPropOverlaySet runs in Phase 2 of OnGroupCompleted, between PREP and ATOMIC SWAP, installing a per-prop swapPropAtomic hook on each task (plus an onPropSwapped fallback the recovery/resume path still uses). The overlay for a prop is then SET inside the swap's Phase 2a tight loop, ATOMICALLY with that prop's bucket-pointer flip — one critical section via Shard.SwapBucketAndSetOverlay — not once-for-the-whole-shard up front (which opens a disk-I/O-sized overlay=NEW, bucket=OLD window across RunSwapOnShard's preamble). Only for tokenization-changing migrations (change-tokenization, change-tokenization-filterable).
2. Cover — the entire Phase 2 (atomic swap + post-atomic tidy + OnMigrationComplete) runs with the overlay active. Queries see NEW-tokenized analyzer input against NEW-tokenized bucket content.
3. Clear — OnTaskCompleted clears the overlay on every local shard AFTER flipSemanticMigrationSchema's RAFT commit succeeds. The live schema's prop.Tokenization is now NEW, so subsequent queries hit the right answer via the regular schema-lookup path.
4. Self-clear backstop — Shard.TokenizationFor has a defensive self-clear: if the live schema's tokenization for a prop now matches the overlay value, the entry is removed. Catches the case where the explicit clear in (3) was skipped (schema-flip failure) — the next query touching the prop after the schema eventually catches up will lazily clean up.

For migrations that DON'T change tokenization (the AnalyzerOverlay-driven enable-filterable etc., or format-only enable-rangeable / repair-*), no tokenization overlay is needed — the per-shard local-ready flag and the OnMigrationComplete hook handle the gap.

11. The analyzer overlay

A different but related mechanism. Defined per-strategy via MigrationStrategy.AnalyzerOverlay(props) → map[string]inverted.PropertyOverlay.

Applied by the inverted analyzer during the backfill scan. Used by "from-scratch" strategies (EnableFilterableStrategy, EnableSearchableStrategy, FilterableToRangeableStrategy for both enable-rangeable and repair-rangeable) that build a brand-new inverted bucket while the corresponding schema flag is still false in the RAFT-stored schema. Without this override the analyzer would skip the targeted property (see HasAnyInvertedIndex in inverted/objects.go) and the new bucket would come out empty.

Strategies that don't need an overlay embed noAnalyzerOverlay to get the nil-return default: both retokenize strategies, MapToBlockmaxStrategy, and RoaringSetRefreshStrategy. The live schema flag is already true for the properties they target. RebuildSearchableStrategy returns nil directly, for a different reason: a rebuild must not change tokenization, which is a separate verb.

Different from the tokenization overlay: the analyzer overlay is read by the backfill iterator (write-side); the tokenization overlay is read by the query analyzer (read-side). They operate on different phases of different concerns and don't share state.

12. Cancel + DELETE-property-index

Cancel (POST .../index/{indexType}/cancel):

1. Find the non-terminal task targeting (collection, prop, indexType). If none matches (already finished, never submitted, or already cancelled), return 202 with Status: NO_OP and no taskId. The verb is idempotent: caller's (collection, property) was already verified to exist by the outer handler, so "nothing to cancel" is surfaced as a no-op rather than overloading 404 with two distinct meanings.

2. If that task is not STARTED, return 409 and stop. STARTED is the only cancellable status (TaskStatus.IsCancellable, §4.2). Two cases reach this branch:

   • a coordination phase (PREPARING / SWAPPING). Nodes may already have written merged state or renamed bucket directories; stopping the rest would leave the cluster serving migrated buckets under the pre-migration schema, repairable only by an operator following the guidance the provider logs.
   • a status this build does not recognize (§4.2). This build cannot tell whether the task has swapped anywhere, so it refuses rather than guess. The mutation-refusal guards agree: for this case they drop the "cancel it" advice and point the operator at the nodes that do recognize the status (MutationRemedy).

   Either way the task has to run through to FINISHED or FAILED. The REST pre-flight and Manager.CancelTask apply the same predicate, so a cancel that loses the race to a status transition is refused at the FSM too, and cancelApplyFailureResponder renders that refusal as the same 409 the pre-flight would have.

3. RAFT CancelDistributedTask.

4. Wait for the local reindex goroutine to drain (WaitForLocalTaskDrain, 10s timeout). Bounded so a stuck goroutine doesn't turn the HTTP request into a hang.

5. DB.NewStalePartialReindexSweep() — wipe sidecars + migration dir so the next submit starts from a clean slate. Runs for every index type the migration touches, not just the one named in the URL: change-tokenization spawns a searchable and a filterable strategy under one task, so cleaning only one leaves the sibling orphaned. Tracker generations a swap already merged or tidied are preserved, so a property that got that far needs an operator rebuild rather than a resubmit.

6. 202 with Status: CANCELLED + the cancelled task ID.

If the drain times out, return 202 anyway — the next submit's defense-in-depth cleanup will pick up the work. If the node crashes mid-cancel, the on-disk state survives; the next submit's pre-cleanup catches that gap.

DELETE /properties/{propertyName}/index/{indexType}:

1. Acquire the same ReindexSubmitLocks entry as PUT (closes the race described in §4.1).
2. Schema FSM applies the UpdateProperty flipping the flag to false. The MutationGuard rejects if a reindex is in flight on this property (FromInFlightMigration=false on this path).
3. removeBucket drops the canonical bucket dir, cleanStaleSidecarDirs removes the leftover __reindex / __ingest / __backup dirs, and cleanStaleMigrationDirs removes the stale tracker dirs — preserving any generation-suffixed tracker carrying tidied.mig / merged.mig, which is live deferred-finalize state, not stale partial state.
4. Subsequent re-enable starts from clean state.

13. Known limitation: schema migration

Schema migration across runtime-reindex state does not work, and the v1.38 Preview merge does not fix it. The fixes live on backup-runtime-reindex-fixes and land as a follow-up PR. Tracking: weaviate/0-weaviate-issues#215.

Backups are covered: DB.AnyLiveReindexForShard (reindex_inflight.go) refuses a backup on any shard DTM reports a live reindex on, and test/acceptance/reindex_backup/ covers it. Operators should not rely on schema migration interacting cleanly with an in-flight or recently-completed reindex while running v1.38 Preview.

14. Upgrade / operations: DISTRIBUTED_TASKS_COMPLETED_TASK_TTL_HOURS

This knob sets how long FINISHED distributed tasks (completed reindex tasks included) linger in the RAFT task list before garbage collection. 0 means immediate GC on the next scheduler tick. It is now accepted because the per-property blockmax truth is durable in the schema stamp, so no reader depends on FINISHED reindex tasks lingering.

Keep DISTRIBUTED_TASKS_COMPLETED_TASK_TTL_HOURS at its default until every node runs the durable-stamp version. During a mixed-version (v1.38/v1.39) rolling upgrade window 0 is unsafe two ways:

• A stamp-blind older node that does not hold the shard can transiently mis-resolve a partial-class property once the completed task is GC'd. This self-heals once every node is on the new version.
• On rollback, 0 fails an older node's startup (fail-loud, not data corruption).

Set 0 (immediate GC of completed tasks) only once every node runs the durable-stamp version.

One narrower residual remains even at the default TTL. Between node startup and the first RunSearchableBlockmaxRepair pass, a pre-stamp residual class (blockmax on disk, nil stamp, class flag false) is not yet seeded, so it still resolves through the legacy class-flag/task-list path. A client that issues a change-tokenization on such a class inside that window can mis-resolve its searchable bucket. This is client-triggered and transient: it self-heals the moment the repair seeds the stamp, and it is narrower than the automatic post-restart corruption the stamp closed. The TTL does not govern it; the window is bounded by the repair's first pass, not by task GC.

The repair seeds the stamp from two sources — a loaded shard's on-disk blockmax bucket, or a still-present FINISHED blockmax task — so a cold (unloaded) shard is still covered as long as its completed task is in the list. TTL=0 breaks that: it can GC the FINISHED task before any repair pass observes it, and with no on-disk evidence either (cold shard) the stamp is never seeded until that shard next loads. Keeping the default TTL avoids this residual.

Downgrade: the stamp is an additive searchableBlockmax property field; an older binary ignores it and derives blockmax from the class flag / task list as before, so a downgrade does not corrupt data. The residual is only that a partial-class property seeded solely by the stamp (its FINISHED task already GC'd) resolves as WAND on the older binary until a re-migration.

15. Files of interest

REST

• adapters/handlers/rest/handlers_indexes.go — GET / PUT / DELETE.
• adapters/handlers/rest/handlers_reindex.go — validation helpers, status synthesis.
• adapters/handlers/rest/state/reindex_submit_locks.go — per-(class, prop) submit lock.
• entities/models/index_*.go — swagger DTOs (IndexStatus, IndexUpdateRequest, PropertyIndexStatus).
• openapi-specs/ — source of truth for the DTOs.

Orchestration

• adapters/repos/db/reindex_provider.go — DTM provider, three-phase swap, flipSemanticMigrationSchema.
• adapters/repos/db/reindex_provider_payload.go — ReindexTaskPayload, migration type constants.
• adapters/repos/db/reindex_conflict.go — CheckConflict, CheckPropertyUpdate, CheckClassMutation, CheckTenantMutation, Touches* predicates, MutationRemedy.
• adapters/repos/db/reindex_activity_lookup.go — NewShardReindexActivityLookup, the backup gate's snapshot.
• adapters/repos/db/reindex_orphan_audit.go — NewLiveReindexTrackerLookup, the orphan audit's snapshot.
• adapters/repos/db/reindex_recovery.go — DiscoverInFlightReindexTasks, buildRecoveryTasks, recovery-only ShardReindexerV3.
• adapters/repos/db/reindex_cancel_cleanup.go — DB.NewStalePartialReindexSweep.
• adapters/repos/db/reindex_inflight.go — DB.AnyLiveReindexForShard, the backup gate.

Strategy + finalize

• adapters/repos/db/inverted_reindex_strategy.go — MigrationStrategy interface, applyPerPropertySchemaUpdate, reindexTaskConfig.
• adapters/repos/db/inverted_reindex_strategy_*.go — one per strategy.
• adapters/repos/db/inverted_reindex_strategy_dir_names.go — genSuffix, parseMigrationDirName, strategy dir prefix constants.
• adapters/repos/db/inverted_reindex_task_generic.go — ShardReindexTaskGeneric, the phase-contract godoc at the top of the file is the authoritative spec.
• adapters/repos/db/inverted_reindex_finalize.go — FinalizeCompletedMigrations, nextMigrationGeneration, maxMigrationGeneration, completedMigrationGens.

LSM primitives

• adapters/repos/db/lsmkv/store.go — SwapBucketPointer, FinalizeBucketSwap, updateBucketDir. The two function godocs document the in-memory-vs-disk split.
• adapters/repos/db/lsmkv/segment_group_prepend.go — SegmentGroup.PrependSegmentsFromBucket; the wrapper the reindex path calls, Bucket.PrependSegmentsFromBucket, is in bucket.go.

Inverted analyzer / overlay

• adapters/repos/db/inverted/tokenization.go — TokenizationResolver, ResolveTokenization.
• adapters/repos/db/shard.go — SetTokenizationOverlay, ClearTokenizationOverlay, TokenizationFor (with the self-clear backstop).

DTM

• cluster/distributedtask/doc.go — package-level architecture + the four "journey" shapes.
• cluster/distributedtask/types.go — Task, Unit, UnitSpec, TaskStatusPreparing, TaskStatusSwapping, NeedsPreparationBarrier, the TaskStatus.IsTerminal() / IsActive() / IsCompleted() / IsRecognized() classification helpers, and IsCancellable() (§4.2).
• cluster/distributedtask/manager.go — FSM. RecordPostCompletionAck, MarkTaskFinalized godocs are essential reading.
• cluster/distributedtask/scheduler.go — per-node loop, callback dispatch.
• cluster/distributedtask/errors.go — permanent-rejection sentinels + gRPC wire encoding.

Schema FSM

• cluster/proto/api/ — PropertyField* constants used by the fieldmask, FromInFlightMigration flag.
• cluster/schema/meta_class.go — MergePropsMasked fieldmask apply path.
• usecases/schema/ — UpdatePropertyInternal, UpdatePropertyInternalFromMigration.

16. Tests of interest

Tests are layered: unit close to the symbol; per-package integration (build tag integrationTest) where multi-component interaction matters; acceptance under test/acceptance/reindex_* with the modern testcontainer style.

Acceptance — single node (test/acceptance/reindex_singlenode/):

• happy_path per migration type (one file each: enable_filterable_test, enable_searchable_test, enable_rangeable_test, change_tokenization_test, change_tokenization_filterable_test, blockmax_test, roaring_set_test).
• delete_then_reenable_test / delete_reenable_multicycle_test / delete_reenable_indexing_bleed_test / delete_reenable_shortcircuit_test — the #10675 family.
• change_tok_delete_journeys_test — the cross-strategy clobber + cleanStaleMigrationDirs family.
• cancel_test / cancel_then_retry_test — cancel + the defense-in-depth cleanup.
• torn_resume_test / restart_during_swap_test — crash recovery in every phase boundary.
• property_state_migration_matrix_test — exhaustive matrix (~510 cells × 6 data types × 15 body shapes).
• scope_assertion_test — blast-radius bounds: a migration on one property never mutates buckets on any other property.
• api_validation_test — REST contract.
• repair_rangeable_test / finished_race_test.

Acceptance — multi-node (test/acceptance/reindex_multinode/):

• happy_path_test — 3-node baseline.
• finalizing_window_test — per-shard tokenization overlay coverage.
• finalizing_crash_test — ack barrier rehydrate over restart.
• restart_matrix_test — rolling restart × migration type.
• restart_test — full restart at every sentinel boundary.
• post_restart_test — recovery + finalize after planned restart.
• round_trip_test — the original word → field → word pin for #10675.
• round_trip_adjacent_test — adjacent journeys (multi-round, different tokenizations, filterable-only, searchable-only, enable-then-change, MT, concurrent-different-props).
• in_flight_rangeable_test — query correctness during enable- rangeable mid-flight (relies on rangeableLocalReady).
• migration_journeys_test — full lifecycle coverage of every semantic migration.
• concurrent_migrations_test — parallel non-conflicting submits.

Acceptance — concurrency (test/acceptance/reindex_concurrent/):

• concurrent_test — same-collection concurrent submits.
• parallel_conflict_matrix_test — every (migrationA × migrationB) pair on overlapping vs disjoint properties.
• parallel_same_property_test — race-condition coverage on the per-(class, prop) submit lock + FSM conflict check.

Acceptance — multi-tenant (test/acceptance/reindex_mt/):

• reindex_mt_test — ?tenants= filtering, per-tenant repair, FROZEN-tenant resume, per-tenant OnGroupCompleted barrier.

Acceptance — backup gate (test/acceptance/reindex_backup/):

• suite_test — backup refused while a reindex is live on the shard.

Acceptance — rangeable (test/acceptance/reindex_rangeable/):

• concurrent_writes_test — writes landing during an enable-rangeable build.

Distributed task framework (test/acceptance/distributed_tasks/):

• unit_tracking_test — end-to-end unit creation under reindex provider on 3-node cluster.

Shared helpers — test/acceptance/helpers/reindex/helpers.go. HTTP-level helpers (SubmitIndexUpdate, AwaitReindexFinished, GetIndexes, AwaitReindexViaIndexes, BoolPtr, IdsMatchUnordered) with WithTenants / WithTimeout functional options. Plus the fixture API: SetupClass, SetupClassWithConfig, ImportObjects, WithEnv — consolidated from prior duplicated copies across the four test packages.

Unit tests of interest

• mergeReindexStatus — handlers_indexes_edge_test.go / handlers_indexes_gaps_test.go.
• checkReindexConflict / ReindexPropsOverlap / TypesConflictReason / Touches* — reindex_conflict_test.go.
• failUnit and recovery — reindex_provider_failunit_test.go, reindex_provider_recovery_test.go, reindex_provider_repair_guidance_test.go.
• parseMigrationDirName, nextMigrationGeneration, multi-gen FinalizeCompletedMigrations paths — inverted_reindex_finalize_test.go.
• OnGroupCompleted cache + rehydrate — reindex_provider_on_group_completed_test.go.
• Tokenization overlay set/clear/self-clear — reindex_provider_tokenization_overlay_test.go, shard_tokenization_overlay_test.go.
• Shard CoW callbacks — shard_callbacks_test.go.
• DTM finalizing + ack barrier — cluster/distributedtask/manager_test.go, scheduler_multinode_test.go, errors_test.go.
• LSM swap primitives — lsmkv/store_bucket_swap_test.go.
• Atomic-phase regression guard — inverted_reindex_task_generic_test.go (the processOneSwapPropFn wall-clock budget assertion).

17. Deferred simplifications

docs/proposals/deferred_reindex_simplifications.md catalogues two refactors that the scout pass identified as worthwhile but that were deliberately not applied autonomously, because each touches either a crash-safety path or the hottest write hook. Re-evaluated for the v1.38 Preview merge and kept deferred. Source of truth for follow-up work.