docs/design-docs/design_docs/20251221-entity_level_ttl.md
Currently, Milvus supports collection-level TTL for data expiration, but does not support defining an independent expiration time for individual entities (rows). As application scenarios become more diverse, for example:
Relying solely on collection-level TTL can no longer satisfy these requirements. If users want to retain only part of the data, they must periodically perform upsert operations to refresh the timestamps of those entities. This approach is unintuitive and increases operational and maintenance costs.
Therefore, Entity-level TTL becomes a necessary feature.
Related issues:
Milvus already supports the TIMESTAMPTZ data type. Entity TTL information will therefore be stored in a field of this type.
Entity-level TTL is implemented by allowing users to explicitly add a TIMESTAMPTZ column in the schema and mark it in collection properties:
"ttl_field": "ttl"
Here, ttl is the name of the column that stores TTL information. This mechanism is mutually exclusive with collection-level TTL.
Example:
t1t2DataType == TIMESTAMPTZ can be configured as a TTL field.TIMESTAMPTZ fields, but only one can be designated as the TTL field.ExpireAt).NULL value semantics:
NULL TTL value means the entity never expires.For an existing collection to enable entity-level TTL:
AlterCollection.TIMESTAMPTZ field using AddField.AlterCollection to mark the new field as the TTL field.If historical data should also have expiration times, users must perform an upsert operation to backfill the TTL field.
A new field expirQuantiles is added to the segment metadata:
message SegmentInfo {
int64 ID = 1;
int64 collectionID = 2;
int64 partitionID = 3;
string insert_channel = 4;
int64 num_of_rows = 5;
common.SegmentState state = 6;
int64 max_row_num = 7 [deprecated = true]; // deprecated, we use the binary size to control the segment size but not a estimate rows.
uint64 last_expire_time = 8;
msg.MsgPosition start_position = 9;
msg.MsgPosition dml_position = 10;
// binlogs consist of insert binlogs
repeated FieldBinlog binlogs = 11;
repeated FieldBinlog statslogs = 12;
// deltalogs consists of delete binlogs. FieldID is not used yet since delete is always applied on primary key
repeated FieldBinlog deltalogs = 13;
bool createdByCompaction = 14;
repeated int64 compactionFrom = 15;
uint64 dropped_at = 16; // timestamp when segment marked drop
// A flag indicating if:
// (1) this segment is created by bulk insert, and
// (2) the bulk insert task that creates this segment has not yet reached `ImportCompleted` state.
bool is_importing = 17;
bool is_fake = 18;
// denote if this segment is compacted to other segment.
// For compatibility reasons, this flag of an old compacted segment may still be False.
// As for new fields added in the message, they will be populated with their respective field types' default values.
bool compacted = 19;
// Segment level, indicating compaction segment level
// Available value: Legacy, L0, L1, L2
// For legacy level, it represent old segment before segment level introduced
// so segments with Legacy level shall be treated as L1 segment
SegmentLevel level = 20;
int64 storage_version = 21;
int64 partition_stats_version = 22;
// use in major compaction, if compaction fail, should revert segment level to last value
SegmentLevel last_level = 23;
// use in major compaction, if compaction fail, should revert partition stats version to last value
int64 last_partition_stats_version = 24;
// used to indicate whether the segment is sorted by primary key.
bool is_sorted = 25;
// textStatsLogs is used to record tokenization index for fields.
map<int64, TextIndexStats> textStatsLogs = 26;
repeated FieldBinlog bm25statslogs = 27;
// This field is used to indicate that some intermediate state segments should not be loaded.
// For example, segments that have been clustered but haven't undergone stats yet.
bool is_invisible = 28;
// jsonKeyStats is used to record json key index for fields.
map<int64, JsonKeyStats> jsonKeyStats = 29;
// This field is used to indicate that the segment is created by streaming service.
// This field is meaningful only when the segment state is growing.
// If the segment is created by streaming service, it will be a true.
// A segment generated by datacoord of old arch, will be false.
// After the growing segment is full managed by streamingnode, the true value can never be seen at coordinator.
bool is_created_by_streaming = 30;
bool is_partition_key_sorted = 31;
// manifest_path stores the fullpath of LOON manifest file of segemnt data files.
// we could keep the fullpath since one segment shall only have one active manifest
// and we could keep the possiblity that manifest stores out side of collection/partition/segment path
string manifest_path = 32;
// expirQuantiles records the expiration timestamps of the segment
// at the 20%, 40%, 60%, 80%, and 100% data distribution levels
repeated int64 expirQuantiles = 33;
}
Meaning:
expirQuantiles: The expiration timestamps corresponding to the 20%, 40%, 60%, 80%, and 100% percentiles of data within the segment.expirQuantiles is computed during sort or mix compaction tasks.expirQuantiles (rounded down).Special cases:
expirQuantiles is NULL, the segment is treated as non-expiring.During query execution, expired data is filtered by comparing the TTL field value with the MVCC timestamp inside mask_with_timestamps.
Specify the TTL field in the schema:
schema = client.create_schema(auto_id=False, description="test entity ttl")
schema.add_field("id", DataType.INT64, is_primary=True)
schema.add_field("ttl", DataType.TIMESTAMPTZ, nullable=True)
schema.add_field("vector", DataType.FLOAT_VECTOR, dim=dim)
prop = {"ttl_field": "ttl"}
client.create_collection(
collection_name,
schema=schema,
enable_dynamic_field=True,
properties=prop,
)
Insert data the same way as a normal TIMESTAMPTZ field:
rows = [
{"id": 0, "vector": [random.random() for _ in range(dim)], "ttl": None},
{"id": 1, "vector": [random.random() for _ in range(dim)], "ttl": None},
{"id": 2, "vector": [random.random() for _ in range(dim)], "ttl": None},
{"id": 3, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T00:00:00Z"},
{"id": 4, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T01:00:00Z"},
{"id": 5, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T02:00:00Z"},
{"id": 6, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T03:00:00Z"},
{"id": 7, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T04:00:00Z"},
{"id": 8, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T23:59:59Z"},
]
insert_result = client.insert(collection_name, rows)
client.flush(collection_name)
Index creation and loading are unaffected. Indexes can still be built on the TTL field if needed.
index_params = client.prepare_index_params()
index_params.add_index(
field_name="vector",
index_type="IVF_FLAT",
index_name="vector",
metric_type="L2",
params={"nlist": 128},
)
client.create_index(collection_name, index_params=index_params)
client.load_collection(collection_name)
Use queries with different timestamps to validate expiration behavior:
query_expr = "id > 0"
res = client.query(
collection_name,
query_expr,
output_fields=["id", "ttl"],
limit=100,
)
print(res)