docs/design-docs/design_docs/20260627-segment-reopen-atomic-read-update-cow.md
Date: 2026-06-27
Component: Segcore / QueryNode
Scope: ChunkedSegmentSealedImpl.{h,cpp}
ChunkedSegmentSealedImpl is responsible for a large amount of “serve queries while updating runtime state” behavior during the lifetime of a sealed segment, including but not limited to:
If these updates mutate live members directly, while readers use a separate published metadata / readiness view to determine visibility, the implementation becomes prone to mixed-state reads:
Therefore, the long-term goal for ChunkedSegmentSealedImpl should not be “patch a few individual reopen windows,” but rather to establish one unified copy-on-write runtime publication model:
All reader-visible runtime state that is mutable during reopen/load/update should, whenever possible, be prepared on cloned runtime state and published atomically together with metadata.
This design document describes the target end-state architecture for ChunkedSegmentSealedImpl:
(metadata + runtime)This is a long-term evolution direction and is not limited to the field or index types already migrated today.
This design does not require the following:
The goal is one unified direction with phased delivery, not a one-shot rewrite of the entire sealed segment implementation.
The index-update migration assumes a coordinator baseline whose IndexChecker
schedules segment index changes through LoadScope_Reopen. The historical
LoadScope_Index wire value (2) is retired and remains reserved so it cannot
be reused for another meaning.
This design does not support a mixed-version deployment in which an older
QueryCoord still emits LoadScope_Index requests to a newer QueryNode after the
legacy index-update handler has been removed. A QueryNode that receives wire
value 2 rejects it as an internal protocol mismatch with an explicit upgrade
message. The deployment must upgrade QueryCoord to a version that emits
LoadScope_Reopen before routing index-update requests to these QueryNodes.
The retired value must not fall through to the full segment load path: an index
update is not a full load and must not trigger full-load distribution, bloom
filter, or delete-stream side effects. It must also not be blindly remapped to
LoadScope_Reopen, because a legacy request may omit newer
SegmentLoadInfo fields; publishing that request as the new load-info snapshot
could discard metadata already held by the newer QueryNode. Supporting that
version skew would require a dedicated compatibility adapter that merges and
normalizes the legacy request against the current published load info before
reopen, which is outside the scope of this design.
Without a unified runtime snapshot, a typical read path is often split into two steps:
This means a reader does not see one single, coherent object graph. Instead, it observes a composition of two different ownership systems:
As soon as the update path is not perfectly synchronized, inconsistency becomes possible.
If reopen / load / replace / drop mutates live runtime first and publishes metadata later, it introduces prepare pollution:
This makes reopen semantics fragile, and failure paths become much harder to reason about than success paths.
Old readers and new readers must coexist safely across a publication boundary. If runtime resources are not owned by published state, then:
This class of lifetime-split problems can repeat across field data, array offsets, and many kinds of index runtime objects.
The final goal for ChunkedSegmentSealedImpl is to converge reader-visible runtime state onto a single source of truth:
PublishedSegmentState
├─ schema
├─ load_info
├─ commit_ts
├─ readiness bitsets
└─ runtime (snapshot-owned resource graph)
A reader that calls CapturePublishedState() should obtain, in one step:
In other words:
A reader should no longer assemble its view by combining published state with live members.
Any runtime update that affects reader-visible behavior should ultimately follow the same flow:
capture current published state
│
▼
clone current runtime graph / container topology
│
▼
prepare new runtime state on cloned graph
│
▼
publish new (metadata + runtime) atomically
│
▼
finalize post-publication cleanup / side work
Key requirements:
For any reader-visible mutable runtime resource, the following invariants should hold:
Prepare invariant
Publication invariant
Lifetime invariant
Readiness invariant
PublishedSegmentState should not only contain metadata/readiness. It should also own a snapshot-scoped runtime resource bundle.
Conceptually, the runtime bundle should be able to contain all reader-visible mutable runtime state, for example:
The principle is not “what has already been moved today,” but rather:
If a runtime resource is mutated during reopen/load/update and is directly consumed by readers, it should be treated as a candidate member of the runtime snapshot.
COW does not imply deep-copying all payloads.
The recommended strategy is:
map / set / indirection graph)shared_ptr, cache slots, or immutable payload objects whenever possibleThis allows the model to preserve:
Field-ready, index-ready, and raw-data-related readiness should not be treated as an independent fact source separate from runtime objects.
In the target architecture, readiness is a derived consistency expression of published state:
The design rule is:
The published runtime object graph and readiness bitsets must describe the same visible world.
Future ChunkedSegmentSealedImpl reader paths should follow this rule:
Whether the accessor is for field data, array views, json access, ngram access, or vector access, the long-term convergence direction should be the same:
PublishedSegmentState::runtimeWith a snapshot-owned runtime graph:
This is one of the main benefits of the COW publication model.
All important update paths should clearly separate three stages:
For any future mutable runtime state added to ChunkedSegmentSealedImpl, the default pattern should be:
preparepublishfinalizeIf a resource cannot follow this model, the PR should explicitly explain:
Drop / replace should also become snapshot-driven:
This avoids the fragile pattern of “delete from live member first, then hope readers do not step on it.”
This design allows phased rollout, but every phase must converge in the same direction rather than creating multiple long-lived state models.
In other words:
Future migration can be prioritized in this order:
From the long-term perspective, all of the following should be on an explicit migration path:
vector_indexings_text_indexes_json_indices / json_stats_reader_ is a special object:
Therefore, reader_ may migrate later than other resources, but it should still be treated as a dedicated concurrency / ownership redesign problem, not as a permanent exception to the COW model.
ChunkedSegmentSealedImpl ChangesAny future change to ChunkedSegmentSealedImpl reopen/load/update logic should, by default, pass the following checks:
PublishedSegmentState::runtime?The default answer should be “yes.”
If the answer is “no,” the change should be treated as an exception that must be justified explicitly, not as a new default implementation style.
This rule especially applies to:
Validation should not only prove that one helper is internally self-consistent. It must validate the full publication semantics.
Atomic visibility
Prepare isolation
Lifetime safety
Drop / replace safety
Readiness consistency
Validation should not only cover the resources already migrated today. It should cover the feature paths most likely to expose publication bugs, such as:
The core goal is to prove:
This model holds for all reader-visible mutable runtime state, not just for one field category or one index category.
The final target state of ChunkedSegmentSealedImpl should be:
In other words, the end state is not “a collection of local fixes,” but:
ChunkedSegmentSealedImplevolves under one stable consistency model in which all reader-visible mutable runtime state defaults to the COW runtime publication framework.
For any future change that touches reader-visible mutable runtime state in ChunkedSegmentSealedImpl, the default design principle should be:
prefer cloned runtime preparation + atomic snapshot publication + post-publication finalize over direct live-member mutation.
That is the final goal this design document establishes.