docs/design-docs/design_docs/20260720-segment-reopen-request-read-lease-drain-design.md
internal/core/src/segcore/ChunkedSegmentSealedImpl.{h,cpp}internal/core/src/segcore/segment_c.cppinternal/core/src/segcore/search_result_export_c.cppinternal/core/src/common/QueryResult.hinternal/querynodev2/tasks/search_task.goThis design retains the existing PublishedSegmentState and runtime
copy-on-write publication framework in ChunkedSegmentSealedImpl, but narrows
the serving consistency boundary for sealed segments to a
request-level read lease:
AsyncSearch on a sealed segment acquires a cross-thread-safe
SegmentReadLease.SearchResult and remains alive until
DeleteSearchResult.SearchResult. They do not acquire a new lease.SearchResult from the
rejected attempt, then retries the complete sealed segment batch with jitter
under the request context.As a result, one search request observes one sealed snapshot from Segcore search through reduce, export, and output-field fill.
This design explicitly stops supporting the following behavior:
Different CGO phases of the same search request may observe different sealed snapshots.
The design accepts a longer Reopen publication wait in exchange for:
The sealed segment implementation is already moving schema, load information,
readiness, and runtime resources into PublishedSegmentState. Writers clone
the current state, prepare the next state, and publish it with an atomic store.
With a call-scoped read guard, the following sequence is possible:
AsyncSearch reads S0
↓
Search returns and releases the guard
↓
Reopen publishes S1
↓
PrepareSearchResults / FillOutputFields reads S1
This can be made logically valid with strict row-mapping invariants, but
SearchResult may also carry iterators, offset mappings, or cache-backed
resources created from S0. The ownership model then becomes part of every
escaped resource.
Because Reopen is infrequent and publication is allowed to temporarily reject
and retry new sealed searches, this design extends read protection through the
entire SearchResult lifetime.
PublishedSegmentState.LazyCheckSchema remains available as a transitional mechanism, runs
before gate acquisition, and fails fast instead of waiting for either the
publication gate or another reopen.mutex_ with a writer-priority mutex.This design does not attempt to:
A complete reader-visible PublishedSegmentState:
PublishedSegmentState
├─ schema
├─ load_info
├─ commit_ts
├─ readiness
└─ runtime resource graph
A request-scoped claim on the currently published sealed snapshot.
The lease does not select an arbitrary historical snapshot. Its contract is:
While the lease is alive, the sealed segment cannot publish its next snapshot.
A writer has completed, or nearly completed, preparation and is ready to publish:
AcquireSegmentReadLease returns an empty lease for a growing segment.
Growing continues to rely on its existing synchronization:
mutex_;sch_mutex_;chunk_mutex_;This design does not change the lock relationship among Insert, AddTexts, Growing Reopen, and query execution.
Sealed search lifetime:
LazyCheckSchema
↓
Acquire SegmentReadLease for S0
↓
Search
↓
Prepare / Reduce / Export / Fill
↓
DeleteSearchResult
↓
Release SegmentReadLease
Reopen lifetime:
Prepare S1 while the gate is open
↓
writer_pending = true
↓
reject new sealed search leases
↓
wait for active_readers == 0
↓
publish S1 exactly once
↓
open the gate
For one sealed segment:
Even though the request lease prevents cross-snapshot reads, Reopen must still preserve:
Any operation that changes row layout or PK mapping must use full segment replacement instead of this Reopen protocol.
AsyncSearch may acquire a lease on a Segcore search executor thread, while
DeleteSearchResult may execute on a different CGO thread.
Therefore, SearchResult must not own:
std::shared_lock<folly::SharedMutexWritePriority>
and release it from another thread. Request-level protection should use a cross-thread-safe reference-counted token.
Conceptual structure:
struct GateState {
std::mutex mutex;
std::condition_variable cv;
uint64_t active_readers = 0;
bool writer_pending = false;
bool writer_active = false;
// Optional observability state.
uint64_t blocked_readers = 0;
std::chrono::steady_clock::time_point oldest_reader_start;
};
ChunkedSegmentSealedImpl and all outstanding leases share ownership of:
std::shared_ptr<GateState> operation_gate_;
This separates the gate lifetime from the segment object lifetime and prevents a lease destructor from dereferencing an already-destroyed segment.
class SegmentReadLease {
public:
SegmentReadLease() = default; // Growing/no-op.
SegmentReadLease(SegmentReadLease&&) = default;
SegmentReadLease& operator=(SegmentReadLease&&) = default;
~SegmentReadLease();
bool
valid() const;
private:
explicit SegmentReadLease(std::shared_ptr<GateState> state);
std::shared_ptr<GateState> state_;
};
Escaping Search read acquisition:
lock gate mutex
if writer_pending || writer_active:
increment rejection counter
return the narrow transient gate-busy error
active_readers++
unlock gate mutex
Escaping read admission must not wait inside one segment while the same QueryNode attempt retains leases from other segments. Otherwise two requests that visit segments in opposite orders can retain partial leases and deadlock two publishers. The Search retry boundary is therefore the complete sealed segment batch, not one segment or one goroutine.
Call-scoped Retrieve acquisition uses the same fail-fast gate operation:
lock gate mutex
if writer_pending || writer_active:
increment rejection counter
return the narrow transient gate-busy error
active_readers++
unlock gate mutex
Because Retrieve releases its lease before the CGO call returns, QueryNode can retry only the rejected segment call. No reader waits inside the Search CPU executor, so queued Retrieve work cannot prevent Search consume callbacks from transferring results to Go for eventual lease release.
Read release:
lock gate mutex
active_readers--
if active_readers == 0:
notify writer
unlock gate mutex
Writer acquisition:
lock gate mutex
check cancellation
writer_pending = true
wait, with cancellation polling and a bounded deadline, until
active_readers == 0 && !writer_active
check cancellation again
writer_active = true
unlock gate mutex
The second cancellation check is the cancellation linearization point. A cancellation observed by that check aborts publication and reopens the gate. Cancellation arriving after the check may not prevent publication and does not roll back a publication that has already been admitted.
The production drain deadline defaults to 30 seconds, including callers that
do not provide an OpContext. Cancellation or timeout clears
writer_pending, wakes waiters, discards the staged state, and leaves the old
snapshot active.
Writer release:
lock gate mutex
writer_active = false
writer_pending = false
notify all readers
unlock gate mutex
All online publishers must first serialize on reopen_mutex_ before entering
the writer protocol. The mutex may be renamed to publication_mutex_ after
publisher convergence, but the single-publisher property is required: a
boolean writer_pending is not a correct queue for multiple concurrent
publishers.
Recommended order:
milvus::OpContext op_ctx(cancel_token);
// Reopen is allowed before entering the gate.
segment->LazyCheckSchema(plan->schema_, &op_ctx);
// The request snapshot is stable from this point.
auto lease = AcquireSegmentReadLease(segment);
// Only pure reads and validation are allowed inside the gate.
auto snapshot = CapturePublishedStateForValidation(segment);
ValidatePlanAgainstSnapshot(plan, snapshot);
CheckExternalFieldsInLoadedManifest(...);
std::unique_ptr<SearchResult> result;
if (!filter_only && !segment->FieldAccessible(target_field)) {
result = BuildEmptySearchResult(...);
} else {
result = segment->Search(...);
}
result->read_lease_ = std::move(lease);
return result;
Every successfully returned result must receive the lease:
If an exception occurs before result ownership is established, the local lease releases through RAII.
LazyCheckSchema must execute before lease acquisition.
Executing it inside the gate would produce deterministic self-deadlock:
request increments active_readers
↓
LazyCheckSchema calls Reopen
↓
Reopen waits for active_readers == 0
↓
the request waits for itself
The lazy path must also avoid blocking on older requests. Search result consume callbacks run on the same Search CPU executor as AsyncSearch and Retrieve. If a lazy reopen waits for an old lease while enough newer-schema requests occupy the executor, those callbacks cannot run and the old lease cannot be released.
Sealed LazyCheckSchema therefore uses a fail-fast publication mode:
compare schema version
↓
try_lock reopen_mutex; fail gate-busy if another reopen owns it
↓
recheck schema version
↓
if readers or a publisher are already present: fail gate-busy
↓
prepare the staged schema/runtime snapshot
↓
atomically try to acquire publication with active_readers == 0
↓
publish, or discard the staged snapshot and fail gate-busy
The early gate check avoids known-unnecessary preparation but is not the
linearization point: a reader may enter during preparation, so final
publication must repeat the check while holding the gate mutex. A failed lazy
publication does not set writer_pending and leaves the old snapshot readable.
QueryNode handles the narrow gate-busy signal using the existing whole-batch
Search retry or per-segment Retrieve retry.
Explicit Reopen continues to serialize with a blocking reopen_mutex_ and use
bounded drain on the Load executor. Only the read-triggered lazy path is
fail-fast.
The following sequence is allowed:
LazyCheckSchema publishes S1
↓
another writer publishes compatible S2
↓
the request acquires a lease on S2
Therefore, LazyCheckSchema only guarantees:
acquired snapshot schema version >= plan schema version
It does not guarantee that the acquired snapshot is exactly the one published by the request's own lazy reopen.
The race is valid only if:
After lease acquisition, validate the stable snapshot without mutation:
snapshot.schema_version >= plan.schema_version;If the snapshot is unexpectedly older than the plan:
release lease
↓
retry LazyCheckSchema
↓
reacquire lease
Reopen must never execute while a request lease is held.
If the snapshot is newer but incompatible:
The following entry points must not acquire another lease:
PrepareSearchResultsForExport;ExportSearchResultAsArrowRecordBatch;FillOutputFieldsOrdered;They rely on the original SearchResult remaining alive and validate that a
sealed result already owns a valid lease.
Temporary SearchResult objects borrow protection from the original result;
they do not copy or release lease ownership.
Reacquisition would deadlock:
old SearchResult holds a lease
writer_pending waits for the old lease
Fill tries to acquire a new lease and waits for the writer
Fill cannot finish
the old lease cannot be released
AsyncRetrieve, AsyncRetrieveByOffsets, and direct read APIs without a
long-lived C++ result use a fail-fast call-scoped lease with per-call retry:
LazyCheckSchema before the gate, when applicable
attempt fail-fast lease acquisition
if gate busy: return without entering and retry this segment call in QueryNode
validate
read and serialize
release before returning across CGO
The serialized protobuf no longer refers to the sealed snapshot, so Query reduce and QueryStream send do not retain the lease. Separate Retrieve calls may observe different compatible snapshots; the Reopen row/offset invariants remain mandatory.
GetRowCount, GetRealCount, HasRawData, HasFieldData, and similar APIs
must be audited according to whether they consume published/runtime state.
QueryResult.h can forward-declare the lease token and let SearchResult own
either:
std::shared_ptr<segcore::SegmentReadLease> read_lease_;
or a move-only lease directly, depending on the final audit of
SearchResult move and copy behavior.
Requirements:
DeleteSearchResult remains the ownership boundary:
void
DeleteSearchResult(CSearchResult search_result) {
delete static_cast<SearchResult*>(search_result);
}
Destroying SearchResult releases the lease.
No separate “release lease” C API should be added because it would create double-release and missed-release risks across Go and C++.
lock reopen_mutex_
capture current state
compute load/schema diff
prepare cloned runtime
load/build/drop resources in staged state
normalize final state
compact manifest load info in staged state
freeze next state and retired resource list
release committer/internal/cache/reader locks
set writer_pending
wait for active_readers == 0
atomic_store final state exactly once
clear writer_pending and wake readers
retire old resources / cancel warmup
unlock reopen_mutex_
While waiting for active requests, a publisher must not hold:
mutex_;reader_mutex_;Otherwise:
writer holds an internal mutex
↓
writer waits for active request lease
↓
active request waits for the internal mutex
and the drain cannot complete.
Current Reopen paths may call CompactRuntimeLoadInfoForManifest() after
committer.Publish(...), producing a second publication.
This design requires:
The gate must cover every serving-time publication, not only Reopen:
SetCommitTimestamp;SetLoadInfo;ClearData;PublishState helpers.Initial load may use an explicit initialization publication only when a lifecycle assertion proves that the segment is not yet visible to readers.
Recommended API split:
PublishStateUnsafe(...); // Constructor/initialization only.
PublishState(PublishLease&, ...); // Online serving publication.
Ordinary helpers should not acquire the writer gate implicitly. Hidden gate acquisition could begin a drain while the caller still owns an old mutex.
Writer wait must be cancellation-aware:
OpContext is null and follows the
same rollback path;The first implementation keeps the existing shared mutex_ acquisition in
SegmentInternalInterface::Search/Retrieve.
Sealed search therefore uses:
Acquire request lease
↓
SegmentInternalInterface::Search
↓
mutex_ shared lock
These are different mechanisms:
mutex_ continues to protect live state that has not completed snapshot
migration.Growing receives a no-op lease and continues to use only its existing locks.
Historical sealed mutexes may be removed incrementally after all reader-visible live state and accessors are audited. That cleanup is not a prerequisite for the first drain implementation.
The request lease prevents Reopen publication, so the active
PublishedSegmentState and runtime graph are not replaced during the request.
Therefore, SearchResult does not need a separate whole-snapshot pin solely
for Reopen lifetime.
The cache manager may evict resources independently from Segment Reopen.
The current sealed index search uses a local cache accessor while a Knowhere iterator may survive beyond the search CGO call.
The implementation must determine:
OffsetMapping* points into an independently evictable object.If the iterator is not self-owning:
SearchResult must retain the corresponding cache cell pin.This is a cache-lifetime pin, not a snapshot-selection pin.
The current successful QueryNode search path effectively registers:
defer Segment.Unpin(searchedSegments)
...
defer DeleteSearchResults(results)
Go defers execute in LIFO order, so successful requests release
SearchResult and its lease before unpinning the segment.
For a multi-segment sealed attempt, fail-fast admission may leave successful partial results from other goroutines. The error path must therefore release all non-null results only after all attempt goroutines have completed, then retry the complete segment batch:
for {
results, err := searchSegmentsAttempt(...)
if err == nil {
return results, nil
}
DeleteSearchResults(nonNil(results))
if !isSealedReadGateBusy(err) {
return nil, err
}
waitJitteredBackoff(ctx)
}
The gate-busy classifier must require both the Segcore
FollyOtherException code and the stable segment read gate busy marker so
unrelated 2037 errors are not retried. Growing searches and other errors return
immediately.
The final contract must guarantee lease release for:
Suggested files:
internal/core/src/segcore/SegmentReadLease.h;internal/core/src/segcore/SegmentReadLease.cpp.Responsibilities:
shared_ptr<GateState> operation_gate_;AcquireReadLease;LazyCheckSchema before gate acquisition;AsyncSearch result branch;DeleteSearchResult destruction.DropIndex publication cancellation/timeout instead of returning
success.PublishState call sites;mutex_;At minimum, expose:
segcore_segment_active_read_leases;segcore_segment_oldest_read_lease_seconds;segcore_segment_publish_drain_wait_seconds;segcore_segment_writer_pending_seconds;segcore_segment_blocked_read_requests_total;segcore_segment_published_generation.Recommended structured log fields:
Normal read-lease acquire/release must not emit INFO logs per query.
DeleteSearchResult.reopen_mutex_.mutex_ or reader_mutex_ while waiting.Reopen waits for the slowest outstanding SearchResult.
This is accepted because:
Mitigations:
Once a writer is pending, new sealed searches fail admission. QueryNode cleans up the full attempt and retries the complete segment batch with jitter. This avoids retaining partial cross-segment leases, but can increase retry traffic while a long-lived old request is draining.
The publisher has a bounded wait and returns cancellation/timeout instead of forcing publication by ignoring a live lease, which would reintroduce mixed-snapshot and resource-lifetime risks.
Retrieve uses the same fail-fast admission signal as Search, but retries only the rejected segment call because no lease escapes the CGO call. Retry uses bounded jitter under the request context and never waits inside a Segcore executor task.
A long drain can increase retry traffic and latency, but it cannot occupy every Search CPU executor thread with gate waits or prevent Search consume callbacks from progressing. Rejection counts and retry duration should be observable.
LazyCheckSchema may still perform I/O and loading on one Search CPU executor
thread, but it never waits for active readers or queues behind
reopen_mutex_. Competing requests fail fast so consume callbacks retain
executor capacity and can release old SearchResult leases.
Preparation can still increase latency, and a reader racing with the final publication check can cause the staged work to be discarded and retried. Long-term alternatives include:
The request lease protects Segment publication but does not automatically protect independent cache eviction.
If a Knowhere iterator is not self-owning, a resource-level cache pin remains necessary.
DeleteSearchResult.mutex_ use remains in the first implementation.The following items must be resolved before the final design is approved:
Gate type location
SegmentReadLease.{h,cpp};SearchResult field type
SegmentReadLease;shared_ptr<SegmentReadLease>.Schema incompatibility error
Maximum lease duration
Writer-wait cancellation implementation
Cache iterator ownership
Initial versus online publication
Direct read C API scope
The final serving principle is:
A sealed segment fixes one published snapshot when a request starts. Reopen stops admitting new requests after preparation, waits for all old requests to finish, and publishes one complete new snapshot.
In addition:
LazyCheckSchema may prepare or reopen schema before gate acquisition. After gate acquisition, only pure validation is allowed; every repair or retry must release the lease first.