docs/user_guides/snapshot_user_guide.md
⚠️ Python SDK Support Pending (temporary — remove this block when pymilvus catches up)
The Python examples in this document describe the intended API surface. As of this release, pymilvus does not yet expose
collection_nameon the snapshot APIs, so the examples below cannot be executed from Python. The corresponding python_client E2E tests undertests/python_client/milvus_client/test_milvus_client_snapshot.pyare currently skipped until pymilvus lands SDK support.Use the Go SDK or HTTP API for snapshot operations today. The Go examples in this guide are fully functional and covered by the
tests/go_client/testcases/snapshot_test.goE2E suite.TODO (when pymilvus adds
collection_nameto snapshot APIs):
- Delete this warning block
- Re-enable the skipped tests in
test_milvus_client_snapshot.py- Verify the Python snippets below actually run against the new SDK
Milvus snapshot feature allows users to create point-in-time copies of collections. This powerful capability enables data backup, versioning, and restoration scenarios. Snapshots capture the complete state of data including vector data, metadata, indexes, and schema information at a specific timestamp.
A Milvus snapshot consists of:
Snapshots are stored in object storage with the following structure:
snapshots/{collection_id}/
├── metadata/
│ └── {snapshot_id}.json # Snapshot metadata (JSON format)
│
└── manifests/
└── {snapshot_id}/ # Directory for each snapshot
├── {segment_id_1}.avro # Individual segment manifest (Avro format)
├── {segment_id_2}.avro
└── ...
Note: The metadata JSON file directly contains an array of manifest file paths, eliminating the need for a separate manifest list file.
Create a snapshot for a collection.
Best Practice (Strongly Recommended): Call flush() before creating a snapshot to ensure all data is persisted. The create_snapshot operation only captures existing sealed segments and does not trigger data flushing automatically. Data in growing segments will not be included in the snapshot.
Note: Calling flush is not mandatory, but highly recommended to avoid data loss. If you skip flush, only data that has already been flushed to sealed segments will be included in the snapshot.
Python SDK Example:
from pymilvus import MilvusClient
client = MilvusClient(uri="http://localhost:19530")
# Recommended: Flush data before creating snapshot to ensure all data is included
client.flush(collection_name="my_collection")
# Create snapshot for entire collection
client.create_snapshot(
collection_name="my_collection",
snapshot_name="backup_20240101",
description="Daily backup for January 1st, 2024"
)
Go SDK Example:
import (
"context"
"github.com/milvus-io/milvus/client/v3/milvusclient"
)
client, err := milvusclient.New(context.Background(), &milvusclient.ClientConfig{
Address: "localhost:19530",
})
// Recommended: Flush data before creating snapshot to ensure all data is included
err = client.Flush(context.Background(), milvusclient.NewFlushOption("my_collection"))
if err != nil {
log.Fatal(err)
}
// Create snapshot
createOpt := milvusclient.NewCreateSnapshotOption("backup_20240101", "my_collection").
WithDescription("Daily backup for January 1st, 2024")
err = client.CreateSnapshot(context.Background(), createOpt)
Parameters:
List existing snapshots for collections.
Python SDK Example:
# List all snapshots for a collection
snapshots = client.list_snapshots(collection_name="my_collection")
Go SDK Example:
// List snapshots for collection
listOpt := milvusclient.NewListSnapshotsOption().
WithCollectionName("my_collection")
snapshots, err := client.ListSnapshots(context.Background(), listOpt)
Parameters:
Returns:
Get detailed information about a specific snapshot.
Python SDK Example:
snapshot_info = client.describe_snapshot(
snapshot_name="backup_20240101",
collection_name="my_collection",
include_collection_info=True
)
print(f"Snapshot ID: {snapshot_info.id}")
print(f"Collection: {snapshot_info.collection_name}")
print(f"Created: {snapshot_info.create_ts}")
print(f"Description: {snapshot_info.description}")
Go SDK Example:
describeOpt := milvusclient.NewDescribeSnapshotOption("backup_20240101", "my_collection")
resp, err := client.DescribeSnapshot(context.Background(), describeOpt)
fmt.Printf("Snapshot ID: %d\n", resp.GetSnapshotInfo().GetId())
fmt.Printf("Collection: %s\n", resp.GetSnapshotInfo().GetCollectionName())
Parameters:
Returns:
Restore a snapshot to a new collection. This operation is asynchronous and returns a job ID for tracking the restore progress.
Restore Mechanism: Snapshot restore uses a Copy Segment mechanism instead of traditional bulk insert. This approach:
Python SDK Example:
# Restore snapshot to new collection
job_id = client.restore_snapshot(
snapshot_name="backup_20240101",
collection_name="my_collection", # source collection (where the snapshot lives)
target_collection_name="restored_collection", # target collection to create
)
# Wait for restore to complete
import time
while True:
state = client.get_restore_snapshot_state(job_id=job_id)
if state.state == "RestoreSnapshotCompleted":
print(f"Restore completed in {state.time_cost}ms")
break
elif state.state == "RestoreSnapshotFailed":
print(f"Restore failed: {state.reason}")
break
print(f"Restore progress: {state.progress}%")
time.sleep(1)
Go SDK Example:
restoreOpt := milvusclient.NewRestoreSnapshotOption("backup_20240101", "my_collection", "restored_collection")
jobID, err := client.RestoreSnapshot(context.Background(), restoreOpt)
if err != nil {
log.Fatal(err)
}
// Poll for restore completion
for {
state, err := client.GetRestoreSnapshotState(context.Background(),
milvusclient.NewGetRestoreSnapshotStateOption(jobID))
if err != nil {
log.Fatal(err)
}
if state.GetState() == milvuspb.RestoreSnapshotState_RestoreSnapshotCompleted {
log.Printf("Restore completed in %dms", state.GetTimeCost())
break
}
if state.GetState() == milvuspb.RestoreSnapshotState_RestoreSnapshotFailed {
log.Fatalf("Restore failed: %s", state.GetReason())
}
log.Printf("Restore progress: %d%%", state.Progress)
time.Sleep(time.Second)
}
Parameters:
Returns:
Export a snapshot as a self-contained object-storage bundle. The returned
snapshotMetadataURI points to the exported metadata file and can be used for
external restore in another cluster that can access the same object storage
location. The export target may be in another bucket when the object-storage
provider supports server-side copy and one resolved credential can read the
source bucket and write the target bucket.
Go SDK Example:
metadataURI, err := client.ExportSnapshot(context.Background(),
milvusclient.NewExportSnapshotOption(
"backup_20240101",
"my_collection",
"s3://bucket/snapshot-exports/backup_20240101",
).WithExternalSpec(`{"extfs":{"cloud_provider":"aws","region":"us-west-2","use_iam":"true"}}`))
if err != nil {
log.Fatal(err)
}
log.Printf("Exported snapshot metadata: %s", metadataURI)
REST Example:
POST /v2/vectordb/jobs/snapshot/export
{
"collectionName": "my_collection",
"snapshotName": "backup_20240101",
"targetS3Path": "s3://bucket/snapshot-exports/backup_20240101",
"externalSpec": "{\"extfs\":{\"cloud_provider\":\"aws\",\"region\":\"us-west-2\",\"use_iam\":\"true\"}}"
}
Restore a snapshot from a metadata URI instead of from the target cluster's local snapshot registry. This is intended for cross-cluster restore after a snapshot has been exported.
A self-contained snapshot generated from an existing snapshot and an explicit file mapping can be restored with the same external restore API. The metadata URI must point to the generated snapshot metadata file, and every file reference inside that metadata must be readable from the target cluster object storage configuration before the restore job is created.
Go SDK Example:
jobID, err := client.RestoreExternalSnapshot(context.Background(),
milvusclient.NewRestoreExternalSnapshotOption(
"restored_collection",
metadataURI,
).WithExternalSpec(`{"extfs":{"cloud_provider":"aws","region":"us-west-2","use_iam":"true"}}`))
if err != nil {
log.Fatal(err)
}
log.Printf("Restore job ID: %d", jobID)
REST Example:
POST /v2/vectordb/jobs/snapshot/restore_external
{
"targetCollectionName": "restored_collection",
"snapshotMetadataURI": "s3://bucket/snapshot-exports/backup_20240101/snapshots/100/metadata/1.json",
"externalSpec": "{\"extfs\":{\"cloud_provider\":\"aws\",\"region\":\"us-west-2\",\"use_iam\":\"true\"}}"
}
targetS3Path and snapshotMetadataURI can be object keys or s3://bucket/key
URIs. externalSpec is optional. When it is empty, Milvus uses the instance
object-storage credential and relies on bucket policy to authorize the other
bucket. When it is set, only storage-config-compatible extfs fields are
accepted. Avoid raw access keys in restore externalSpec unless operationally
required, because restore job state must propagate the spec through persistent
metadata before DataNode can execute the copy.
Delete a snapshot permanently.
Python SDK Example:
client.drop_snapshot(
snapshot_name="backup_20240101",
collection_name="my_collection"
)
Go SDK Example:
dropOpt := milvusclient.NewDropSnapshotOption("backup_20240101", "my_collection")
err := client.DropSnapshot(context.Background(), dropOpt)
Parameters:
Query the status and progress of a restore snapshot job.
Python SDK Example:
state = client.get_restore_snapshot_state(job_id=12345)
print(f"Job ID: {state.job_id}")
print(f"Snapshot Name: {state.snapshot_name}")
print(f"Collection ID: {state.collection_id}")
print(f"State: {state.state}")
print(f"Progress: {state.progress}%")
if state.state == "RestoreSnapshotFailed":
print(f"Failure Reason: {state.reason}")
print(f"Time Cost: {state.time_cost}ms")
Go SDK Example:
stateOpt := milvusclient.NewGetRestoreSnapshotStateOption(12345)
state, err := client.GetRestoreSnapshotState(context.Background(), stateOpt)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Job ID: %d\n", state.GetJobId())
fmt.Printf("Snapshot Name: %s\n", state.GetSnapshotName())
fmt.Printf("Collection ID: %d\n", state.GetCollectionId())
fmt.Printf("State: %s\n", state.GetState())
fmt.Printf("Progress: %d%%\n", state.GetProgress())
if state.GetState() == milvuspb.RestoreSnapshotState_RestoreSnapshotFailed {
fmt.Printf("Failure Reason: %s\n", state.GetReason())
}
fmt.Printf("Time Cost: %dms\n", state.GetTimeCost())
Parameters:
Returns:
List all restore snapshot jobs, optionally filtered by collection name.
Python SDK Example:
# List all restore jobs
jobs = client.list_restore_snapshot_jobs()
for job in jobs:
print(f"Job {job.job_id}: {job.snapshot_name} -> Collection {job.collection_id}")
print(f" State: {job.state}, Progress: {job.progress}%")
# List restore jobs for a specific collection
jobs = client.list_restore_snapshot_jobs(collection_name="my_collection")
Go SDK Example:
// List all restore jobs
listOpt := milvusclient.NewListRestoreSnapshotJobsOption()
jobs, err := client.ListRestoreSnapshotJobs(context.Background(), listOpt)
if err != nil {
log.Fatal(err)
}
for _, job := range jobs {
fmt.Printf("Job %d: %s -> Collection %d\n",
job.GetJobId(), job.GetSnapshotName(), job.GetCollectionId())
fmt.Printf(" State: %s, Progress: %d%%\n",
job.GetState(), job.GetProgress())
}
// List restore jobs for a specific collection
listOpt = milvusclient.NewListRestoreSnapshotJobsOption().
WithCollectionName("my_collection")
jobs, err = client.ListRestoreSnapshotJobs(context.Background(), listOpt)
Parameters:
Returns:
Snapshots provide a lightweight and efficient backup solution compared to traditional tools like milvus-backup.
import datetime
# Create daily backup
today = datetime.date.today().strftime("%Y%m%d")
snapshot_name = f"daily_backup_{today}"
# Recommended: Flush data to ensure all changes are persisted
client.flush(collection_name="production_vectors")
# Create snapshot
client.create_snapshot(
collection_name="production_vectors",
snapshot_name=snapshot_name,
description=f"Daily backup for {today}"
)
Comparison: Snapshot vs. milvus-backup
| Operation | milvus-backup | Snapshot |
|---|---|---|
| Backup Creation | Copies all data files | Creates metadata only (milliseconds) |
| Restore Process | Imports data and rebuilds indexes | Copies existing data and index files directly |
| Performance | Slower, resource-intensive | Fast and lightweight |
| System Impact | High I/O and CPU usage | Minimal impact |
Why Snapshots are More Efficient:
The snapshot capability provides a foundation for significantly improving the milvus-backup tool
Snapshots enable efficient offline data processing by providing stable, consistent data sources for analytical workloads. Users can directly access snapshot data stored in object storage (S3) with Spark or other big data processing frameworks without impacting the live Milvus cluster.
Key Benefits:
Use Case Example: Vector Similarity Analysis
from pyspark.sql import SparkSession
import datetime
import json
# Step 1: Create snapshot for offline processing
snapshot_name = f"analytics_snapshot_{datetime.date.today().strftime('%Y%m%d')}"
# Recommended: Flush data to ensure all changes are persisted
client.flush(collection_name="user_embeddings")
client.create_snapshot(
collection_name="user_embeddings",
snapshot_name=snapshot_name,
description="Snapshot for daily analytics job"
)
# Step 2: Get snapshot metadata to locate data files in S3
snapshot_info = client.describe_snapshot(
snapshot_name=snapshot_name,
collection_name="user_embeddings",
include_collection_info=True
)
# Step 3: Process snapshot data with Spark
spark = SparkSession.builder \
.appName("VectorAnalytics") \
.config("spark.hadoop.fs.s3a.access.key", "YOUR_ACCESS_KEY") \
.config("spark.hadoop.fs.s3a.secret.key", "YOUR_SECRET_KEY") \
.getOrCreate()
# Read and parse snapshot metadata to get actual file paths
# Note: This is a simplified example. In practice, you need to:
# 1. Parse the metadata JSON file to get manifest file paths
# 2. Parse manifest Avro files to get binlog/deltalog paths
# 3. Read the actual data files
s3_path = snapshot_info.s3_location
# Example: Read binlog files directly if you know the structure
# In reality, you would parse the manifest files first
df = spark.read.format("your_format").load(f"s3a://{s3_path}/binlogs/")
# Perform analytics operations
# Example: Compute vector statistics, clustering, or quality metrics
result = df.groupBy("partition_id").agg({
"vector_dim": "count",
"timestamp": "max"
})
result.write.mode("overwrite").parquet("s3a://analytics-results/daily_stats/")
# Step 4: Clean up snapshot after processing completes
client.drop_snapshot(
snapshot_name=snapshot_name,
collection_name="user_embeddings"
)
Common Offline Processing Scenarios:
Maintain multiple versions of data for experimentation:
# Create version snapshots before major updates
# Recommended: Flush to ensure all data is captured
client.flush(collection_name="ml_embeddings")
client.create_snapshot(
collection_name="ml_embeddings",
snapshot_name="v1.0_baseline",
description="Baseline model embeddings"
)
# After model update, flush and create new snapshot
client.flush(collection_name="ml_embeddings")
client.create_snapshot(
collection_name="ml_embeddings",
snapshot_name="v1.1_improved",
description="Improved model embeddings"
)
Create snapshots for testing environments:
# Create test data snapshot
# Recommended: Flush to ensure all test data is captured
client.flush(collection_name="test_collection")
client.create_snapshot(
collection_name="test_collection",
snapshot_name="test_dataset_v1",
description="Test dataset for regression testing"
)
# Restore for testing with progress tracking
job_id = client.restore_snapshot(
snapshot_name="test_dataset_v1",
collection_name="test_collection", # source collection
target_collection_name="test_environment", # target collection
)
# Monitor restore progress
import time
while True:
state = client.get_restore_snapshot_state(job_id=job_id)
if state.state == "RestoreSnapshotCompleted":
print(f"Test environment ready! Restored in {state.time_cost}ms")
break
elif state.state == "RestoreSnapshotFailed":
print(f"Restore failed: {state.reason}")
break
print(f"Setting up test environment: {state.progress}%")
time.sleep(1)
Track multiple restore jobs simultaneously:
# Start multiple restore operations
job_ids = []
for i in range(3):
job_id = client.restore_snapshot(
snapshot_name=f"snapshot_v{i}",
collection_name="my_collection",
target_collection_name=f"test_env_{i}"
)
job_ids.append(job_id)
# Monitor all jobs
while job_ids:
completed = []
for job_id in job_ids:
state = client.get_restore_snapshot_state(job_id=job_id)
if state.state in ["RestoreSnapshotCompleted", "RestoreSnapshotFailed"]:
completed.append(job_id)
print(f"Job {job_id} finished: {state.state}")
for job_id in completed:
job_ids.remove(job_id)
if job_ids:
time.sleep(1)
# Alternatively, list all restore jobs
jobs = client.list_restore_snapshot_jobs()
for job in jobs:
print(f"Job {job.job_id}: {job.snapshot_name} - {job.progress}% ({job.state})")
Use consistent and descriptive naming:
# Good naming examples
"daily_backup_20240101"
"v2.1_production_release"
"test_dataset_regression_suite"
# Avoid generic names
"backup1", "test", "snapshot"
Snapshot Creation:
Snapshot Restore:
GetRestoreSnapshotState API to monitor restore progress for large snapshotsImportant Storage Behavior:
GC Protection Mechanism:
IsSegmentGCBlocked() before deleting any segment
IsBuildIDGCBlocked() before deleting index files
rebuildAllSegmentProtection and
stored in in-memory sets, so per-file/per-buildID checks are constant-time set lookups.internal/datacoord/garbage_collector.goStorage Cost Considerations:
Storage Management Best Practices:
PreserveFieldId=true and PreserveIndexId=trueSnapshot creation fails:
Performance issues:
Common error patterns and solutions:
Error: "snapshot not found" Solution: Verify snapshot name and check if it was deleted
Error: "collection already exists" Solution: Use a different target collection name for restoration
Error: "insufficient storage" Solution: Free up storage space or increase limits
Error: "permission denied" Solution: Check object storage credentials and permissions
Error: "restore job not found" Solution: Verify job ID is correct; job may have expired or been cleaned up
Error: "restore snapshot failed" (check state.reason for details) Solution: Query GetRestoreSnapshotState for specific failure reason and address the underlying issue
Milvus snapshots provide a robust solution for data backup, versioning, and recovery scenarios. By following the best practices and understanding the limitations, users can effectively leverage snapshots to ensure data durability and enable sophisticated data management workflows.
For additional support and advanced configuration options, refer to the Milvus documentation or contact the Milvus community.