v1.x Guarantees

For the v1.0 release, we want to provide the following guarantees:

Flags, Config and minor version upgrades

Upgrading cortex from one minor version to the next should "just work"; that being said, we don't want to bump the major version every time we remove a flag, so we will will keep deprecated flags around for 2 minor release. There is a metric (deprecated_flags_inuse_total) you can alert on to find out if you're using a deprecated flag.

Similarly to flags, minor version upgrades using config files should "just work". If we do need to change config, we will keep the old way working for two minor version. There will be a metric you can alert on for this too.

These guarantees don't apply for experimental features.

Reading old data

The Cortex maintainers commit to ensuring future version of Cortex can read data written by versions up to two years old. In practice we expect to be able to read more, but this is our guarantee.

API Compatibility

Cortex strives to be 100% API compatible with Prometheus (under /prometheus/* and /api/prom/*); any deviation from this is considered a bug, except:

• Additional API endpoints for creating, removing and modifying alerts and recording rules.
• Additional API around pushing metrics (under /api/push).
• Additional API endpoints for management of Cortex itself, such as the ring. These APIs are not part of the any compatibility guarantees.

Experimental features

Cortex is an actively developed project and we want to encourage the introduction of new features and capability. As such, not everything in each release of Cortex is considered "production-ready". We don't provide any backwards compatibility guarantees on these and the config and flags might break.

Currently experimental features are:

• Overrides API
  • Runtime configuration API for managing tenant limits
• Ruler
  • Evaluate rules to query frontend instead of ingesters (enabled via -ruler.frontend-address).
  • When -ruler.frontend-address is specified, the response format can be specified (via -ruler.query-response-format).
• S3 Server Side Encryption (SSE) using KMS (including per-tenant KMS config overrides).
• Alertmanager:
  • Receiver integrations firewall (configured via -alertmanager.receivers-firewall.*)
• gRPC Store.
• TLS configuration in gRPC and HTTP clients.
• TLS configuration in Etcd client.
• OpenStack Swift storage support.
• Metric relabeling in the distributor.
• Ingester: do not unregister from ring on shutdown (-ingester.unregister-on-shutdown=false)
• Distributor:
  • Do not extend writes on unhealthy ingesters (-distributor.extend-writes=false)
  • Accept multiple HA pairs in the same request (enabled via -experimental.distributor.ha-tracker.mixed-ha-samples=true)
  • Accept Prometheus remote write 2.0 request (-distributor.remote-writev2-enabled=true)
• Tenant Deletion in Purger, for blocks storage.
• Blocks storage user index
• Querier: tenant federation
  • -tenant-federation.regex-matcher-enabled
  • -tenant-federation.user-sync-interval
  • -tenant-federation.allow-partial-data
• The thanosconvert tool for converting Thanos block metadata to Cortex
• Alertmanager limits
  • notification rate (-alertmanager.notification-rate-limit and -alertmanager.notification-rate-limit-per-integration)
  • dispatcher groups (-alertmanager.max-dispatcher-aggregation-groups)
  • user config size (-alertmanager.max-config-size-bytes)
  • templates count in user config (-alertmanager.max-templates-count)
  • max template size (-alertmanager.max-template-size-bytes)
• Disabling ring heartbeat timeouts
  • -distributor.ring.heartbeat-timeout=0
  • -ring.heartbeat-timeout=0
  • -ruler.ring.heartbeat-timeout=0
  • -alertmanager.sharding-ring.heartbeat-timeout=0
  • -compactor.ring.heartbeat-timeout=0
  • -store-gateway.sharding-ring.heartbeat-timeout=0
• Disabling ring heartbeats
  • -distributor.ring.heartbeat-period=0
  • -ingester.heartbeat-period=0
  • -ruler.ring.heartbeat-period=0
  • -alertmanager.sharding-ring.heartbeat-period=0
  • -compactor.ring.heartbeat-period=0
  • -store-gateway.sharding-ring.heartbeat-period=0
• Vertical sharding at query frontend for range/instant queries
  • -frontend.query-vertical-shard-size (int) CLI flag
  • query_vertical_shard_size (int) field in runtime config file
• Snapshotting of in-memory TSDB on disk during shutdown
  • -blocks-storage.tsdb.memory-snapshot-on-shutdown (boolean) CLI flag
• Out of order samples support
  • -blocks-storage.tsdb.out-of-order-cap-max (int) CLI flag
  • -ingester.out-of-order-time-window (duration) CLI flag
  • out_of_order_time_window (duration) field in runtime config file
• Store Gateway Zone Stable Shuffle Sharding
  • -store-gateway.sharding-ring.zone-stable-shuffle-sharding CLI flag
  • zone_stable_shuffle_sharding (boolean) field in config file
• Basic Lifecycler (Storegateway, Alertmanager, Ruler) Final Sleep on shutdown, which tells the pod wait before shutdown, allowing a delay to propagate ring changes.
  • -ruler.ring.final-sleep (duration) CLI flag
  • store-gateway.sharding-ring.final-sleep (duration) CLI flag
  • alertmanager-sharding-ring.final-sleep (duration) CLI flag
• OTLP Receiver
  • Ingest delta temporality OTLP metrics (-distributor.otlp.allow-delta-temporality=true)
• Persistent tokens in the Ruler Ring:
  • -ruler.ring.tokens-file-path (path) CLI flag
• Native Histograms
  • Ingestion can be enabled by setting -blocks-storage.tsdb.enable-native-histograms=true on Ingester.
• String interning for metrics labels
  • Enable string interning for metrics labels by setting -ingester.labels-string-interning-enabled on Ingester.
• Query-frontend: query rejection (-frontend.query-rejection.enabled)
• Querier: protobuf codec (-api.querier-default-codec)
• Querier: Series batch size (-querier.store-gateway-series-batch-size)
• Query-frontend: dynamic query splits
  • querier.max-shards-per-query (int) CLI flag
  • querier.max-fetched-data-duration-per-query (duration) CLI flag
• Ingester/Store-Gateway: Query rejection
  • -ingester.query-protection.rejection
  • -store-gateway.query-protection.rejection
• Distributor/Ingester: Stream push connection
  • Enable stream push connection between distributor and ingester by setting -distributor.use-stream-push=true on Distributor.
  • Add __type__ and __unit__ labels to OTLP and remote write v2 requests (-distributor.enable-type-and-unit-labels)
• Ingester: Series Queried Metric
  • Enable on Ingester via -ingester.active-queried-series-metrics-enabled=true
  • Set the time window to expose via metrics using -ingester.active-queried-series-metrics-windows=2h. At least 1 time window is required to expose the metric.
  • -ingester.active-queried-series-metrics-update-period metric update interval
  • -ingester.active-queried-series-metrics-window-duration each HyperLogLog time window size
  • -ingester.active-queried-series-metrics-sample-rate query sampling rate
• Ingester: Regex Matcher Limits
  • Enable regex matcher limits and metrics collection via -ingester.enable-regex-matcher-limits=true
  • Per-tenant limits for unoptimized regex matchers:
    • -validation.max-regex-pattern-length (int) - maximum pattern length in bytes
    • -validation.max-label-cardinality-for-unoptimized-regex (int) - maximum label cardinality
    • -validation.max-total-label-value-length-for-unoptimized-regex (int) - maximum total length of all label values in bytes
• HATracker: -distributor.ha-tracker.enable-startup-sync (bool) - If enabled, fetches all tracked keys on startup to populate the local cache.