Design: New Vector Index Metrics

Overview

Add two new metrics to the INFO section for vector indexes:

HNSW Direct Insertions - Count of vectors inserted directly into HNSW by the main thread (bypassing the flat buffer)
Total Flat Buffer Size - Sum of all flat buffer sizes across tiered vector indexes

Background

In tiered HNSW indexes, vectors are normally inserted into a flat buffer first, then asynchronously moved to the HNSW index by background workers. However, in two scenarios, vectors are inserted directly into HNSW by the main thread:

WriteInPlace mode - When VecSim_WriteInPlace mode is active, all insertions go directly to HNSW
Full flat buffer - When the flat buffer reaches its limit (flatBufferLimit), new vectors are inserted directly into HNSW

Tracking these direct insertions helps monitor indexing behavior and identify potential performance bottlenecks.

The flat buffer size metric provides visibility into the pending work for background indexing threads.

Architecture

Data Flow

VecSimIndex::statisticInfo()     →  VecSimIndexStatsInfo (C++ struct in VectorSimilarity)
        ↓
VecSimIndex_StatsInfo()          →  C API wrapper
        ↓
IndexSpec_GetVectorIndexStats()  →  Populates VectorIndexStats (RediSearch C struct)
        ↓
IndexSpec_GetVectorIndexesStats()→  Aggregates across all vector fields in an index
        ↓
IndexesInfo_TotalInfo()          →  Aggregates across all indexes into TotalIndexesInfo
        ↓
AddToInfo_*()                    →  Renders to INFO command output

Key Files

File	Purpose
`deps/VectorSimilarity/src/VecSim/vec_sim_common.h`	`VecSimIndexStatsInfo` struct definition
`deps/VectorSimilarity/src/VecSim/vec_sim_tiered_index.h`	`statisticInfo()` implementation for tiered indexes
`deps/VectorSimilarity/src/VecSim/algorithms/hnsw/hnsw_tiered.h`	Tiered HNSW implementation (where direct insertions occur)
`src/info/vector_index_stats.h`	`VectorIndexStats` struct and getter/setter mappings
`src/info/vector_index_stats.c`	Getter/setter implementations and aggregation
`src/info/field_spec_info.c`	`IndexSpec_GetVectorIndexStats()` - bridges VecSim to RediSearch
`src/info/indexes_info.h`	`TotalIndexesFieldsInfo` struct for aggregated stats
`src/info/indexes_info.c`	`IndexesInfo_TotalInfo()` - aggregates across all indexes
`src/info/info_redis/info_redis.c`	`AddToInfo_*()` - renders to INFO output

Implementation Plan

Phase 1: VectorSimilarity Changes

1.1 Add counter to TieredHNSWIndex

In hnsw_tiered.h, add a counter field:

cpp

size_t directHNSWInsertions{0};

Increment in addVector() when inserting directly to HNSW (both WriteInPlace and full-buffer cases).

1.2 Extend VecSimIndexStatsInfo

In vec_sim_common.h:

typedef struct {
    size_t memory;
    size_t numberOfMarkedDeleted;
    size_t directHNSWInsertions;  // NEW: Direct insertions to HNSW by main thread
    size_t flatBufferSize;        // NEW: Current flat buffer size
} VecSimIndexStatsInfo;

1.3 Update statisticInfo() implementations

In vec_sim_tiered_index.h:

cpp

VecSimIndexStatsInfo statisticInfo() const override {
    return VecSimIndexStatsInfo{
        .memory = this->getAllocationSize(),
        .numberOfMarkedDeleted = this->getNumMarkedDeleted(),
        .directHNSWInsertions = 0,  // Base class returns 0
        .flatBufferSize = this->frontendIndex->indexSize(),
    };
}

In hnsw_tiered.h, override to include the counter:

cpp

VecSimIndexStatsInfo statisticInfo() const override {
    auto stats = VecSimTieredIndex<DataType, DistType>::statisticInfo();
    stats.directHNSWInsertions = this->directHNSWInsertions.load();
    return stats;
}

Phase 2: RediSearch Changes

2.1 Extend VectorIndexStats

In vector_index_stats.h:

typedef struct VectorIndexStats {
    size_t memory;
    size_t marked_deleted;
    size_t direct_hnsw_insertions;  // NEW
    size_t flat_buffer_size;        // NEW
} VectorIndexStats;

2.2 Add getter/setter functions

In vector_index_stats.c, add:

VectorIndexStats_GetDirectHNSWInsertions() / VectorIndexStats_SetDirectHNSWInsertions()
VectorIndexStats_GetFlatBufferSize() / VectorIndexStats_SetFlatBufferSize()

Update mapping arrays and VectorIndexStats_Agg().

2.3 Update VectorIndexStats_Metrics array

static char* const VectorIndexStats_Metrics[] = {
    "memory",
    "marked_deleted",
    "direct_hnsw_insertions",
    "flat_buffer_size",
    NULL
};

2.4 Update IndexSpec_GetVectorIndexStats()

In field_spec_info.c:

VectorIndexStats IndexSpec_GetVectorIndexStats(FieldSpec *fs) {
    // ... existing code ...
    stats.direct_hnsw_insertions = info.directHNSWInsertions;
    stats.flat_buffer_size = info.flatBufferSize;
    return stats;
}

2.5 Extend TotalIndexesFieldsInfo

In indexes_info.h:

typedef struct {
    size_t total_vector_idx_mem;
    size_t total_mark_deleted_vectors;
    size_t total_direct_hnsw_insertions;  // NEW
    size_t total_flat_buffer_size;        // NEW
} TotalIndexesFieldsInfo;

2.6 Update aggregation in IndexesInfo_TotalInfo()

In indexes_info.c:

info.fields_stats.total_direct_hnsw_insertions += vec_info.direct_hnsw_insertions;
info.fields_stats.total_flat_buffer_size += vec_info.flat_buffer_size;

2.7 Add to INFO output

In info_redis.c:

RedisModule_InfoAddFieldULongLong(ctx, "vector_direct_hnsw_insertions", 
    total_info->fields_stats.total_direct_hnsw_insertions);
RedisModule_InfoAddFieldULongLong(ctx, "vector_flat_buffer_size",
    total_info->fields_stats.total_flat_buffer_size);

INFO Output

The new metrics will appear in a new search_vector_indexe section:

# search_vector_index
...
used_memory_vector_index:12345678
hnsw_direct_main_thread_insertions:42
tiered_index_frontend_buffer_size:1000

And in per-field statistics (FT.INFO):

field statistics:
  - identifier: vec_field
    attribute: vec_field
    memory: 12345678
    marked_deleted: 0
    direct_hnsw_insertions: 42
    flat_buffer_size: 500

Testing

Unit tests: Verify counter increments in WriteInPlace mode and full-buffer scenarios (in VectorSimilarity)
Integration tests: Verify INFO output contains new metrics with correct values

Notes

The directHNSWInsertions counter is cumulative (never reset)
The flatBufferSize is a point-in-time snapshot
For non-tiered indexes, both new metrics return 0
For SVS tiered indexes, directHNSWInsertions returns 0 (SVS doesn't have this concept)