docs/design-docs/design_docs/20260313-scalar_index_version_management.md
Introduce scalar index version management capabilities on par with vector indexes, including target version override, force rebuild, and MaximumVersion propagation and clamping for both vector and scalar indexes.
Milvus already has a complete vector index version management system:
[MinimalIndexVersion, CurrentIndexVersion] from knowhere at startup via etcd Session.IndexEngineVersionManager aggregates all QN versions to determine cluster-wide safe build versions.autoUpgradeSegmentIndex triggers compaction to rebuild indexes when versions lag behind.targetVecIndexVersion provides temporary flexibility to override the default build version.forceRebuildSegmentIndex forces all vector indexes to rebuild to a specified version.However, scalar indexes lack equivalent version management. Currently, scalar index versions only have hardcoded MinimalScalarIndexEngineVersion and CurrentScalarIndexEngineVersion constants, with no target override, force rebuild, or MaxVersion validation.
Additionally, the vector index side's MaximumIndexVersion (the upper bound knowhere supports) exists in C++ but was never propagated to the Go layer, and targetVecIndexVersion was not validated against the upper bound.
| Version | Meaning |
|---|---|
MinimalIndexVersion | Lowest version knowhere can build and search |
CurrentIndexVersion | Hardcoded default build version |
MaximumIndexVersion | Highest version knowhere can build and search (non-beta) |
Knowhere guarantees it can build and search any index version in [MinimalIndexVersion, MaximumIndexVersion].
| Parameter | Intent |
|---|---|
autoUpgradeSegmentIndex | After rolling upgrade, automatically upgrade old indexes to the cluster's current version |
targetVecIndexVersion | Temporary flexibility: override default build version without code changes (default -1, inactive) |
forceRebuildSegmentIndex | Emergency measure: force all indexes to rebuild to the target version (rebuilds already-built indexes too) |
| Parameter | Key | Default | Description |
|---|---|---|---|
TargetScalarIndexVersion | dataCoord.targetScalarIndexVersion | -1 | Target scalar index version, -1 means unspecified |
ForceRebuildScalarSegmentIndex | dataCoord.forceRebuildScalarSegmentIndex | false | Force rebuild scalar index toggle |
Semantics symmetric with the vector side:
forceRebuildScalarSegmentIndex=true + targetScalarIndexVersion=N: Force all scalar indexes to rebuild to version NforceRebuildScalarSegmentIndex=false + targetScalarIndexVersion=N: New scalar indexes use max(cluster aggregated value, N)targetScalarIndexVersion=-1: Inactive, follows cluster aggregated valueconst MaximumScalarIndexEngineVersion = int32(2)
Currently Maximum equals Current for scalar indexes (unlike knowhere vector side, scalar has no reserved beta versions). In the future, Maximum will be bumped first, then Current once stable.
GetMaximumIndexEngineVersion() int32
GetMaximumScalarIndexEngineVersion() int32
ResolveVecIndexVersion() int32
ResolveScalarIndexVersion() int32
type IndexEngineVersion struct {
MinimalIndexVersion int32 `json:"MinimalIndexVersion,omitempty"`
CurrentIndexVersion int32 `json:"CurrentIndexVersion,omitempty"`
MaximumIndexVersion int32 `json:"MaximumIndexVersion,omitempty"` // new
}
Data flow after changes:
┌──────────────────────────────────────────────────────┐
│ QueryNode Startup │
│ │
│ Vector versions: knowhere C++ → [Min, Current, Max] │
│ Scalar versions: Go constants → [Min, Current, Max] │
│ │
│ Written to etcd Session │
└────────────────────────┬──────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────┐
│ DataCoord IndexEngineVersionManager │
│ │
│ Vector aggregation: │
│ GetCurrentIndexEngineVersion() = MIN(all QN.Cur) │
│ GetMinimalIndexEngineVersion() = MAX(all QN.Min) │
│ GetMaximumIndexEngineVersion() = MIN(all QN.Max) │
│ │
│ Scalar aggregation: │
│ GetCurrentScalarIndexEngineVersion() = MIN(...) │
│ GetMinimalScalarIndexEngineVersion() = MAX(...) │
│ GetMaximumScalarIndexEngineVersion() = MIN(...) │
│ │
│ Resolve methods (target override + max clamp): │
│ ResolveVecIndexVersion() │
│ ResolveScalarIndexVersion() │
└────────────┬─────────────────────┬────────────────────┘
│ │
▼ ▼
┌────────────────────────┐ ┌───────────────────────────┐
│ Build New Index │ │ Compaction Trigger │
│ (prepareJobRequest) │ │ (ShouldRebuildSegIndex) │
│ │ │ │
│ Vec: ResolveVec() │ │ Path 1: autoUpgrade │
│ Scalar: ResolveScalar()│ │ vec+scalar, version < │
│ │ │ │
│ Stats task: │ │ Path 2: forceRebuild vec │
│ ResolveScalar() │ │ resolved target != │
│ │ │ │
│ Mix compaction: │ │ Path 3: forceRebuild │
│ ResolveScalar() │ │ scalar (NEW) │
└────────────────────────┘ │ resolved target != │
└───────────────────────────┘
Takes MIN across all QNs — Maximum represents "highest version a node can handle", so the cluster-safe upper bound is the lowest across all nodes.
func (m *versionManagerImpl) GetMaximumIndexEngineVersion() int32 {
maximum := int32(math.MaxInt32)
for _, version := range m.versions {
if version.MaximumIndexVersion == 0 {
continue // skip old QNs that don't report Maximum
}
if version.MaximumIndexVersion < maximum {
maximum = version.MaximumIndexVersion
}
}
if maximum == math.MaxInt32 {
return math.MaxInt32 // all QNs are old, no upper bound check
}
return maximum
}
Centralize the version resolution logic (target override + MaxVersion clamp) to avoid duplication across task_index, task_stats, and compaction_task_mix:
func (m *versionManagerImpl) ResolveScalarIndexVersion() int32 {
version := m.GetCurrentScalarIndexEngineVersion()
if Params.DataCoordCfg.TargetScalarIndexVersion.GetAsInt64() != -1 {
if Params.DataCoordCfg.ForceRebuildScalarSegmentIndex.GetAsBool() {
version = Params.DataCoordCfg.TargetScalarIndexVersion.GetAsInt32()
} else {
version = max(version, Params.DataCoordCfg.TargetScalarIndexVersion.GetAsInt32())
}
}
if maxVersion := m.GetMaximumScalarIndexEngineVersion(); version > maxVersion {
version = maxVersion
}
return version
}
The compaction trigger's force-rebuild paths also use the resolved (clamped) target to compare against, ensuring that when target > cluster maximum, the trigger converges after a single rebuild (since the built index version matches the resolved target).
The DataNode's getCurrentScalarIndexVersion is fixed to clamp against MaximumScalarIndexEngineVersion instead of CurrentScalarIndexEngineVersion, aligning with how the vector side clamps against C.GetMaximumIndexVersion().
DataNode text index paths (stats task and sort compaction) are fixed to use the request-carried scalar version (from DataCoord's Resolve method) instead of hardcoding common.CurrentScalarIndexEngineVersion.
The only cross-node persistent data format change is the etcd Session JSON structure — adding MaximumIndexVersion field.
| Scenario | Behavior |
|---|---|
| Old QN + New DataCoord | Old QN's Session JSON lacks MaximumIndexVersion, Go deserializes it as zero. GetMaximumIndexEngineVersion() skips zero-valued entries. If all QNs are old, returns math.MaxInt32 (no upper bound check). |
| New QN + Old DataCoord | New QN's MaximumIndexVersion field is silently ignored by old DataCoord (Go encoding/json ignores unknown fields). No impact. |
| New QN + New DataCoord | All QNs report MaximumIndexVersion, aggregation works normally, upper bound check fully effective. |
New config parameters (TargetScalarIndexVersion default -1, ForceRebuildScalarSegmentIndex default false) are only read by DataCoord. Default behavior is identical to the old version.
GetMaximumIndexEngineVersion() and GetMaximumScalarIndexEngineVersion() aggregation, including mixed old/new QN scenariosResolveVecIndexVersion() and ResolveScalarIndexVersion() with various target/force/max combinationstargetScalarIndexVersion and verify scalar indexes are built with correct versionforceRebuildScalarSegmentIndex=true and verify all scalar indexes are rebuilt