docs/design-docs/design_docs/20260131-client_side_telemetry.md
This MEP introduces a client-side telemetry system for the Go SDK that collects operational metrics, sends periodic heartbeats to the server, and supports bidirectional communication through server-pushed commands. The system provides visibility into client behavior, enables real-time monitoring through a WebUI dashboard, and allows server-initiated configuration changes.
Currently, Milvus lacks visibility into client-side operations and behavior. Operators cannot:
This feature addresses these gaps by implementing a comprehensive client telemetry system that:
// TelemetryConfig holds configurable settings for client telemetry
type TelemetryConfig struct {
Enabled bool // Enable/disable telemetry collection
HeartbeatInterval time.Duration // Heartbeat frequency (default: 30s)
SamplingRate float64 // Sampling rate 0.0-1.0 (default: 1.0)
ErrorMaxCount int // Max errors to track (default: 100)
}
// ClientConfig gains a new field
type ClientConfig struct {
// ... existing fields ...
TelemetryConfig *TelemetryConfig
}
GET /api/v1/telemetry/clients - List connected clients with optional filtering
POST /api/v1/telemetry/commands - Push commands to clients
DELETE /api/v1/telemetry/commands/{id} - Delete a command
service RootCoord {
// Client heartbeat with metrics
rpc ClientHeartbeat(ClientHeartbeatRequest) returns (ClientHeartbeatResponse);
// Query connected clients
rpc GetClientTelemetry(GetClientTelemetryRequest) returns (GetClientTelemetryResponse);
// Push commands to clients
rpc PushClientCommand(PushClientCommandRequest) returns (PushClientCommandResponse);
// Delete commands
rpc DeleteClientCommand(DeleteClientCommandRequest) returns (DeleteClientCommandResponse);
}
┌─────────────────────────────────────────────────────────────────────────────┐
│ Client (Go SDK) │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ ClientTelemetryManager │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ Operation │ │ Error │ │ Command │ │ │
│ │ │ Metrics │ │ Collector │ │ Handler │ │ │
│ │ │ Collector │ │ (Ring Buffer) │ │ Router │ │ │
│ │ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │ │
│ │ │ │ │ │ │
│ │ └────────────────────┼────────────────────┘ │ │
│ │ ▼ │ │
│ │ ┌───────────────────────┐ │ │
│ │ │ Heartbeat Loop │───────── 30s interval │ │
│ │ │ (Background) │ │ │
│ │ └───────────┬───────────┘ │ │
│ └────────────────────────────────┼──────────────────────────────────────┘ │
└───────────────────────────────────┼─────────────────────────────────────────┘
│
ClientHeartbeat RPC
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ Server (Milvus) │
│ ┌────────────────┐ ┌───────────────────────────────────────────┐ │
│ │ Proxy │◄────────►│ RootCoord │ │
│ │ │ │ ┌─────────────────────────────────────┐ │ │
│ │ HTTP API │ │ │ Telemetry Manager │ │ │
│ │ /telemetry/* │ │ │ ┌───────────┐ ┌───────────────┐ │ │ │
│ │ │ │ │ │ Client │ │ Command │ │ │ │
│ │ WebUI │ │ │ │ Store │ │ Store │ │ │ │
│ │ telemetry.html│ │ │ └───────────┘ └───────────────┘ │ │ │
│ └────────────────┘ │ └─────────────────────────────────────┘ │ │
│ └───────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
The central component managing telemetry collection and heartbeat communication.
type ClientTelemetryManager struct {
config *TelemetryConfig
client *Client
clientID string // Stable UUID
collectors map[string]*OperationMetricsCollector
errorCollector *ErrorCollectorImpl
commandHandlers map[string]CommandHandler
// Heartbeat management
stopCh chan struct{}
wg sync.WaitGroup
}
Key behaviors:
Start()HeartbeatIntervalPer-operation metrics collection with global and per-collection breakdown.
type OperationMetricsCollector struct {
// Global metrics
requestCount int64
successCount int64
errorCount int64
totalLatency int64 // microseconds
maxLatency int64
// P99 calculation (ring buffer of 1000 samples)
latencySamples []int64
totalSamples int64
// Per-collection metrics
collectionMetrics map[string]*CollectionMetrics
}
Metrics tracked:
Ring buffer implementation for tracking recent errors.
type ErrorCollectorImpl struct {
errors []*ErrorInfo
maxCount int
index int // Ring buffer index
}
type ErrorInfo struct {
Timestamp int64
Operation string
ErrorMsg string
Collection string
RequestID string
}
Supports server-pushed commands with extensible handler registration.
type CommandHandler func(cmd *ClientCommand) string // Returns error message or ""
// Built-in commands:
const (
CmdSetSamplingRate = "set_sampling_rate" // Adjust sampling rate
CmdEnableCollections = "enable_collections" // Enable specific collections
CmdUpdateConfig = "update_config" // Update telemetry config
)
Central server-side storage for client telemetry data.
type TelemetryManager struct {
clients map[string]*ClientTelemetry // clientID -> telemetry
commandStore *CommandStore
}
type ClientTelemetry struct {
ClientInfo *commonpb.ClientInfo
LastHeartbeatTime int64
Status string // "active" or "inactive"
Databases []string
Metrics []*commonpb.OperationMetrics
}
Manages pending commands for clients with support for persistent and one-time commands.
type CommandStore struct {
commands []*commonpb.ClientCommand // One-time commands
persistent []*commonpb.ClientCommand // Persistent commands
}
Command targeting:
TargetScope = "*" - All clientsTargetScope = "client_id:xxx" - Specific clientTargetScope = "db:database_name" - Clients using specific databaseREST API endpoints for WebUI and external integrations.
| Endpoint | Method | Description |
|---|---|---|
/api/v1/telemetry/clients | GET | List connected clients |
/api/v1/telemetry/clients?database=X | GET | Filter by database |
/api/v1/telemetry/clients?include_metrics=true | GET | Include operation metrics |
/api/v1/telemetry/commands | POST | Push command to clients |
/api/v1/telemetry/commands/{id} | DELETE | Remove a command |
Authentication: Uses existing Milvus Basic Auth when authorization is enabled.
Client Server
│ │
│──── ClientHeartbeatRequest ──────────────►│
│ - ClientInfo (ID, SDK version, host) │
│ - Metrics (per-operation, per-coll) │
│ - CommandReplies │
│ - ConfigHash (for change detection) │
│ │
│◄─── ClientHeartbeatResponse ─────────────│
│ - Commands (pending for this client) │
│ - NewConfigHash (if config changed) │
│ │
Heartbeat interval: 30 seconds (configurable, server can override)
Config hash: SHA-256 of client configuration, used to detect when server pushes new config.
Metrics Collection:
Heartbeat Cycle:
Command Processing:
A new telemetry dashboard (/webui/telemetry.html) provides:
telemetry_test.go: Metrics collection, P99 calculation, ring buffertelemetry_http_handler_test.go: HTTP API handlerscommand_router_test.go: Command routing and handlingtelemetry_integration_test.go: End-to-end heartbeat flowUsing streaming RPC instead of periodic heartbeats.
Push metrics to external systems (Prometheus, etc.) from client.
Server polls clients for metrics.
internal/proxy/impl.go