ContextQMD
Back to Kubo

Metrics

docs/metrics.md

0.41.06.3 KB
Original Source

Kubo metrics

By default, a Prometheus endpoint is exposed by Kubo at http://127.0.0.1:5001/debug/metrics/prometheus.

It includes default Prometheus Go client metrics + Kubo-specific metrics listed below.

Table of Contents

[!WARNING] This documentation is incomplete. For an up-to-date list of metrics available at daemon startup, see test/sharness/t0119-prometheus-data/prometheus_metrics_added_by_measure_profile.

Additional metrics may appear during runtime as some components (like boxo/gateway) register metrics only after their first event occurs (e.g., HTTP request/response).

DHT RPC

Metrics from go-libp2p-kad-dht for DHT RPC operations:

Inbound RPC metrics

  • rpc_inbound_messages_total - Counter: total messages received per RPC
  • rpc_inbound_message_errors_total - Counter: total errors for received messages
  • rpc_inbound_bytes_[bucket|sum|count] - Histogram: distribution of received bytes per RPC
  • rpc_inbound_request_latency_[bucket|sum|count] - Histogram: latency distribution for inbound RPCs

Outbound RPC metrics

  • rpc_outbound_messages_total - Counter: total messages sent per RPC
  • rpc_outbound_message_errors_total - Counter: total errors for sent messages
  • rpc_outbound_requests_total - Counter: total requests sent
  • rpc_outbound_request_errors_total - Counter: total errors for sent requests
  • rpc_outbound_bytes_[bucket|sum|count] - Histogram: distribution of sent bytes per RPC
  • rpc_outbound_request_latency_[bucket|sum|count] - Histogram: latency distribution for outbound RPCs

Provide

Legacy Provider

Metrics for the legacy provider system when Provide.DHT.SweepEnabled=false:

  • provider_reprovider_provide_count - Counter: total successful provide operations since node startup
  • provider_reprovider_reprovide_count - Counter: total reprovide sweep operations since node startup

DHT Provider

Metrics for the DHT provider system when Provide.DHT.SweepEnabled=true:

  • provider_provides_total - Counter: total successful provide operations since node startup (includes both one-time provides and periodic provides done on Provide.DHT.Interval)

[!NOTE] These metrics are exposed by go-libp2p-kad-dht. You can enable debug logging for DHT provider activity with GOLOG_LOG_LEVEL=dht/provider=debug.

Gateway (boxo/gateway)

[!TIP] These metrics are limited to IPFS Gateway endpoints. For general HTTP metrics across all endpoints, consider using a reverse proxy.

Gateway metrics appear after the first HTTP request is processed:

HTTP metrics

  • ipfs_http_gw_responses_total{code} - Counter: total HTTP responses by status code
  • ipfs_http_gw_retrieval_timeouts_total{code,truncated} - Counter: requests that timed out during content retrieval
  • ipfs_http_gw_concurrent_requests - Gauge: number of requests currently being processed

Blockstore cache metrics

  • ipfs_http_blockstore_cache_hit - Counter: global block cache hits
  • ipfs_http_blockstore_cache_requests - Counter: global block cache requests

Backend metrics

  • ipfs_gw_backend_api_call_duration_seconds_[bucket|sum|count]{backend_method} - Histogram: time spent in IPFSBackend API calls

Generic HTTP Servers

[!TIP] The metrics below are not very useful and exist mostly for historical reasons. If you need non-gateway HTTP metrics, it's better to put a reverse proxy in front of Kubo and use its metrics.

Core HTTP metrics (ipfs_http_*)

Prometheus metrics for the HTTP API exposed at port 5001:

  • ipfs_http_requests_total{method,code,handler} - Counter: total HTTP requests (Legacy - new metrics are provided by boxo/gateway for gateway traffic)
  • ipfs_http_request_duration_seconds[_sum|_count]{handler} - Summary: request processing duration
  • ipfs_http_request_size_bytes[_sum|_count]{handler} - Summary: request body sizes
  • ipfs_http_response_size_bytes[_sum|_count]{handler} - Summary: response body sizes

HTTP Server metrics (http_server_*)

Additional HTTP instrumentation for all handlers (Gateway, API commands, etc.):

  • http_server_request_body_size_bytes_[bucket|count|sum] - Histogram: distribution of request body sizes
  • http_server_request_duration_seconds_[bucket|count|sum] - Histogram: distribution of request processing times
  • http_server_response_body_size_bytes_[bucket|count|sum] - Histogram: distribution of response body sizes

These metrics are automatically added to Gateway handlers, Hostname Gateway, Libp2p Gateway, and API command handlers.

[!NOTE] The server_address label from otelhttp is dropped via an OTel SDK View to prevent cardinality explosion on subdomain gateways (where each unique Host header creates a new time series). All handlers include a server_domain label instead:

  • Gateway and Hostname Gateway handlers group requests by their matching Gateway.PublicGateways domain suffix (e.g., dweb.link, ipfs.io). Unmatched hosts are labeled localhost, loopback, or other.
  • The RPC API handler uses api.
  • The Libp2p Gateway handler uses libp2p.

OpenTelemetry Metadata

Kubo uses Prometheus for metrics collection for historical reasons, but OpenTelemetry metrics are automatically exposed through the same Prometheus endpoint. These metadata metrics provide context about the instrumentation:

  • otel_scope_info - Information about instrumentation libraries producing metrics
  • target_info - Service metadata including version and instance information