docs/getting-started/microservices.md
This guide will help you run Cortex in microservices mode using Kubernetes (Kind). In this mode, each Cortex component runs as an independent service, mirroring how Cortex runs in production.
Time to complete: ~30 minutes
This setup creates a production-like Cortex deployment with independent microservices:
┌─────────────────────────────────────┐
│ Kubernetes Cluster (Kind) │
│ │
┌─────────────┐ │ ┌──────────────┐ ┌──────────────┐ │
│ Prometheus │────remote───┼─>│ Distributor │ │ Ingester │ │
│ │ write │ └──────────────┘ └──────────────┘ │
└─────────────┘ │ │ │ │
│ │ │ │
│ └────────────────┘ │
│ │ │
│ ▼ │
┌─────────────┐ │ ┌──────────────┐ ┌──────────────┐ │
│ Grafana │◄──────┼──┤ Querier │ │ SeaweedFS │ │
└─────────────┘ │ └──────────────┘ │ (S3) │ │
│ ▲ └──────────────┘ │
│ │ │ │
│ ┌──────────────┐ │ │
│ │Store Gateway │◄────────┘ │
│ └──────────────┘ │
│ │
│ ┌──────────────┐ │
│ │ Compactor │ │
│ └──────────────┘ │
│ │
│ ┌──────────────┐ │
│ │ Ruler │ │
│ └──────────────┘ │
└─────────────────────────────────────┘
Components:
Create a local Kubernetes cluster using Kind:
kind create cluster --name cortex-demo
What's happening?
cortex-demoVerify the cluster:
kubectl cluster-info
kubectl get nodes
You should see one node in the Ready state.
Add the Helm repositories for Cortex, Grafana, and Prometheus:
helm repo add cortex-helm https://cortexproject.github.io/cortex-helm-chart
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
git clone https://github.com/cortexproject/cortex.git
cd cortex/docs/getting-started
The getting-started directory contains Helm values files and Kubernetes manifests.
kubectl create namespace cortex
All Cortex components will be deployed in this namespace.
Cortex requires object storage for blocks. We'll use SeaweedFS as an S3-compatible alternative.
kubectl --namespace cortex apply -f seaweedfs.yaml --wait --timeout=5m
kubectl --namespace cortex wait --for=condition=ready pod -l app=seaweedfs --timeout=5m
What's happening?
SeaweedFS needs buckets for Cortex data. First, port-forward to SeaweedFS:
kubectl --namespace cortex port-forward svc/seaweedfs 8333 &
Note: This runs in the background. If the command fails with "port already in use", kill the existing process:
lsof -ti:8333 | xargs kill
Now create the buckets:
for bucket in cortex-blocks cortex-ruler cortex-alertmanager; do
curl --aws-sigv4 "aws:amz:local:seaweedfs" \
--user "any:any" \
-X PUT http://localhost:8333/$bucket
done
What are these buckets?
cortex-blocks: Stores TSDB blocks (metric data)cortex-ruler: Stores ruler configurationcortex-alertmanager: Stores alertmanager configurationVerify buckets were created:
curl --aws-sigv4 "aws:amz:local:seaweedfs" --user "any:any" http://localhost:8333
Deploy Cortex using the Helm chart with the provided values file:
helm upgrade --install \
--version=2.4.0 \
--namespace cortex \
cortex cortex-helm/cortex \
-f cortex-values.yaml \
--wait
What's in cortex-values.yaml?
This takes ~5 minutes. The --wait flag waits for all pods to be ready.
kubectl --namespace cortex get pods
You should see pods for each Cortex component:
cortex-distributor-*cortex-ingester-* (multiple replicas)cortex-querier-*cortex-query-frontend-*cortex-store-gateway-*cortex-compactor-*cortex-ruler-*cortex-nginx-* (reverse proxy)Check logs if pods aren't starting:
kubectl --namespace cortex logs -l app.kubernetes.io/name=cortex -f --max-log-requests 20
Deploy Prometheus to scrape metrics from Kubernetes and send them to Cortex:
helm upgrade --install \
--version=25.20.1 \
--namespace cortex \
prometheus prometheus-community/prometheus \
-f prometheus-values.yaml \
--wait
What's in prometheus-values.yaml?
cortex-distributorX-Scope-OrgID: cortex header for multi-tenancyVerify Prometheus is running:
kubectl --namespace cortex get pods -l app.kubernetes.io/name=prometheus
Deploy Grafana to visualize metrics from Cortex:
helm upgrade --install \
--version=7.3.9 \
--namespace cortex \
grafana grafana/grafana \
-f grafana-values.yaml \
--wait
What's in grafana-values.yaml?
cortex-nginx)Create ConfigMaps for Cortex operational dashboards:
for dashboard in $(ls dashboards); do
basename=$(basename -s .json $dashboard)
cmname=grafana-dashboard-$basename
kubectl create --namespace cortex configmap $cmname \
--from-file=$dashboard=dashboards/$dashboard \
--save-config=true -o yaml --dry-run=client | kubectl apply -f -
kubectl patch --namespace cortex configmap $cmname \
-p '{"metadata":{"labels":{"grafana_dashboard":"1"}}}'
done
What's happening?
grafana_dashboard label tells Grafana's sidecar to load itPort-forward to access Grafana:
kubectl --namespace cortex port-forward deploy/grafana 3000 &
Open Grafana.
For other services, port-forward as needed:
# Prometheus
kubectl --namespace cortex port-forward deploy/prometheus-server 9090 &
# Cortex Nginx (API gateway)
kubectl --namespace cortex port-forward svc/cortex-nginx 8080:80 &
# Cortex Distributor (admin UI)
kubectl --namespace cortex port-forward deploy/cortex-distributor 9009:8080 &
Tip: Open a new terminal for each port-forward, or use & to run in the background.
Port-forward to Prometheus:
kubectl --namespace cortex port-forward deploy/prometheus-server 9090 &
Open Prometheus and:
prometheus_remote_storage_samples_total is increasingPort-forward to the Cortex API:
kubectl --namespace cortex port-forward svc/cortex-nginx 8080:80 &
Query metrics:
curl -H "X-Scope-OrgID: cortex" \
"http://localhost:8080/prometheus/api/v1/query?query=up" | jq
Note: The X-Scope-OrgID header specifies the tenant. Cortex is multi-tenant.
upNavigate to Dashboards to see:
If not already running:
kubectl --namespace cortex port-forward svc/cortex-nginx 8080:80 &
macOS:
wget https://github.com/cortexproject/cortex-tools/releases/download/v0.17.0/cortextool_0.17.0_mac-os_x86_64 -O cortextool
chmod +x cortextool
sudo mv cortextool /usr/local/bin/
Linux:
wget https://github.com/cortexproject/cortex-tools/releases/download/v0.17.0/cortextool_0.17.0_linux_x86_64 -O cortextool
chmod +x cortextool
sudo mv cortextool /usr/local/bin/
Or use Docker:
alias cortextool="docker run --rm --network host -v $(pwd):/workspace -w /workspace quay.io/cortexproject/cortex-tools:v0.17.0"
cortextool rules sync rules.yaml alerts.yaml \
--id cortex \
--address http://localhost:8080
What's happening?
rules.yaml contains recording rules (pre-computed PromQL queries)alerts.yaml contains alerting rules (conditions that trigger alerts)View rules in Grafana: Alerting → Alert rules
Or check via API:
curl -H "X-Scope-OrgID: cortex" \
"http://localhost:8080/prometheus/api/v1/rules" | jq
Load Alertmanager configuration:
cortextool alertmanager load alertmanager-config.yaml \
--id cortex \
--address http://localhost:8080
Verify in Grafana: Alerting → Notification policies
Cortex uses a hash ring to distribute time series across ingesters. Let's add more ingesters:
kubectl --namespace cortex scale deployment cortex-ingester --replicas=5
Observe:
kubectl --namespace cortex port-forward deploy/cortex-distributor 9009:8080
Scale back down:
kubectl --namespace cortex scale deployment cortex-ingester --replicas=3
Simulate a failure by deleting a single ingester pod. This demonstrates Cortex's resilience.
Step 1: List the ingester pods
kubectl --namespace cortex get pods -l app.kubernetes.io/component=ingester
You should see multiple ingester pods (typically 3 replicas).
Step 2: Delete one specific ingester pod
# Replace <pod-name> with an actual pod name from the list above
kubectl --namespace cortex delete pod <pod-name> --force --grace-period=0
Example:
kubectl --namespace cortex delete pod cortex-ingester-76d95464d8 --force --grace-period=0
Observe:
Why it works: Cortex replicates data across multiple ingesters (default: 3 replicas), so losing one ingester doesn't cause data loss.
See what each component is doing:
# Distributor logs (receives metrics from Prometheus)
kubectl --namespace cortex logs -l app.kubernetes.io/component=distributor -f
# Ingester logs (stores metrics in memory)
kubectl --namespace cortex logs -l app.kubernetes.io/component=ingester -f
# Querier logs (handles PromQL queries)
kubectl --namespace cortex logs -l app.kubernetes.io/component=querier -f
# Compactor logs (compacts blocks in S3)
kubectl --namespace cortex logs -l app.kubernetes.io/component=compactor -f
Cortex uses a hash ring for consistent hashing. View the ingester ring:
kubectl --namespace cortex port-forward deploy/cortex-distributor 9009:8080 &
Open http://localhost:9009/ingester/ring to see:
View blocks in SeaweedFS using the S3 API:
kubectl --namespace cortex port-forward svc/seaweedfs 8333 &
List buckets:
curl --aws-sigv4 "aws:amz:local:seaweedfs" --user "any:any" http://localhost:8333
List objects in the cortex-blocks bucket:
curl --aws-sigv4 "aws:amz:local:seaweedfs" --user "any:any" http://localhost:8333/cortex-blocks?list-type=2
You'll see:
cortex/ directory (tenant ID)index, chunks/, and meta.jsonTip: You can also use the AWS CLI:
export AWS_ACCESS_KEY_ID=any
export AWS_SECRET_ACCESS_KEY=any
aws --endpoint-url=http://localhost:8333 s3 ls s3://cortex-blocks/cortex/
| File | Purpose |
|---|---|
seaweedfs.yaml | Kubernetes manifest for SeaweedFS (S3) |
cortex-values.yaml | Helm values for Cortex (component config, storage) |
prometheus-values.yaml | Helm values for Prometheus (scrape configs, remote_write) |
grafana-values.yaml | Helm values for Grafana (datasources, dashboards) |
rules.yaml | Recording rules for the ruler |
alerts.yaml | Alerting rules for the ruler |
alertmanager-config.yaml | Alertmanager notification configuration |
Want to customize? Edit the Helm values files and upgrade:
helm upgrade --namespace cortex cortex cortex-helm/cortex -f cortex-values.yaml
# Check pod status
kubectl --namespace cortex get pods
# Describe a pod to see events
kubectl --namespace cortex describe pod <pod-name>
# Check logs
kubectl --namespace cortex logs <pod-name>
cortex-values.yaml has correct S3 configkubectl --namespace cortex logs -l app.kubernetes.io/component=ingesterkubectl --namespace cortex get configmap prometheus-server -o yamlkubectl --namespace cortex logs -l app.kubernetes.io/component=distributorcurl -H "X-Scope-OrgID: cortex" "http://localhost:8080/prometheus/api/v1/query?query=up"lsof -i :<port>lsof -ti:<port> | xargs killKind requires Docker to have enough resources:
kubectl --namespace cortex port-forward svc/cortex-nginx 8080:80 &curl http://localhost:8080/ready# Kill all kubectl port-forwards
killall kubectl
# Or kill specific port-forwards
lsof -ti:3000,8080,9009,9090 | xargs kill
kind delete cluster --name cortex-demo
This removes all Kubernetes resources and the Kind cluster.
Congratulations! You've successfully deployed Cortex in microservices mode on Kubernetes. Here's what to explore next:
| Aspect | Single Binary | Microservices |
|---|---|---|
| Components | All in one process | Separate pods per component |
| Scaling | Vertical (bigger instance) | Horizontal (more pods) |
| Resource Usage | Lower (1 process) | Higher (multiple processes) |
| Complexity | Simple | Complex (orchestration needed) |
| Failure Isolation | None (single point of failure) | Yes (component failures isolated) |
| Use Case | Dev, testing, learning | Production deployments |