docs/guides/ingesters-scaling-up-and-down.md
This guide explains how to scale up and down ingesters.
If you're looking for how to run ingesters rolling updates, please refer to the dedicated guide._
Adding more ingesters to a Cortex cluster is considered a safe operation. When a new ingester starts, it will register to the hash ring and the distributors will reshard received series accordingly. Ingesters that were previously receiving those series will see data stop arriving and will consider those series "idle".
If you run with -distributor.shard-by-all-labels=false (the default), before adding a second ingester, you have to wait until data has migrated from idle series to the back-end store; otherwise, you will see gaps in queries. This will happen after the next "head compaction" (typically every 2 hours).
If you have set -querier.query-store-after, then that is also a minimum time you have to wait before adding a second ingester.
If you run with -distributor.shard-by-all-labels=true,
no special care is required to take when scaling up ingesters.
A running ingester holds several hours of time series data in memory before they’re flushed to the long-term storage. When an ingester shuts down because of a scale down operation, the in-memory data must not be discarded in order to avoid any data loss.
Ingesters don’t flush series to blocks at shutdown by default. However, Cortex ingesters expose an API endpoint /shutdown that can be called to flush series to blocks and upload blocks to the long-term storage before the ingester terminates.
Even if ingester blocks are compacted and shipped to the storage at shutdown, it takes some time for queriers and store-gateways to discover the newly uploaded blocks. This is due to the fact that the blocks storage runs a periodic scanning of the storage bucket to discover blocks. If two or more ingesters are scaled down in a short period of time, queriers may miss some data at query time due to series that were stored in the terminated ingesters but their blocks haven’t been discovered yet.
Starting with Cortex 1.19.0, a new READONLY state for ingesters was introduced that enables gradual, safe scaling down without data loss or performance impact. This approach eliminates the need for complex configuration changes and allows for more flexible scaling operations.
The READONLY state allows ingesters to:
Set ingesters to READONLY mode
# Transition ingester to READONLY state
curl -X POST http://ingester-1:8080/ingester/mode -d '{"mode": "READONLY"}'
curl -X POST http://ingester-2:8080/ingester/mode -d '{"mode": "READONLY"}'
curl -X POST http://ingester-3:8080/ingester/mode -d '{"mode": "READONLY"}'
Monitor data aging (Optional but recommended)
# Check user statistics and loaded blocks on the ingester
curl http://ingester-1:8080/ingester/all_user_stats
Wait for safe removal window
querier.query-ingesters-within duration (e.g., 5 hours)Remove ingesters
# Terminate the ingester processes
kubectl delete pod ingester-1 ingester-2 ingester-3
For a cluster with querier.query-ingesters-within=5h:
Any time after T2 is safe for removal without service impact.
If you’re running an older version of Cortex that doesn’t support the READONLY state, you’ll need to follow the legacy approach.
The ingesters scale down is deemed an infrequent operation and no automation is currently provided. However, if you need to scale down ingesters, please be aware of the following:
-querier.query-store-after=0s-blocks-storage.bucket-store.sync-interval=5m-compactor.cleanup-interval=5m-blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl=1m-blocks-storage.bucket-store.metadata-cache.tenant-blocks-list-ttl=1m-blocks-storage.bucket-store.metadata-cache.metafile-doesnt-exist-ttl=1m/shutdown endpoint on the ingester to shutdown/shutdown will not do it)-blocks-storage.bucket-store.sync-interval and -compactor.cleanup-interval