docs/sources/setup/upgrade/_index.md
Every attempt is made to keep Grafana Loki backwards compatible, such that upgrades should be low risk and low friction.
Unfortunately Loki is software and software is hard and sometimes we are forced to make decisions between ease of use and ease of maintenance.
If we have any expectation of difficulty upgrading, we will document it here.
As more versions are released it becomes more likely unexpected problems arise moving between multiple versions at once. If possible try to stay current and do sequential updates. If you want to skip versions, try it in a development environment before attempting to upgrade production.
Using docker you can check changes between 2 versions of Loki with a command like this:
export OLD_LOKI=2.9.4
export NEW_LOKI=3.0.0
export CONFIG_FILE=local-config.yaml
diff --color=always --side-by-side <(docker run --rm -t -v "${PWD}":/config grafana/loki:${OLD_LOKI} -config.file=/etc/loki/${CONFIG_FILE} -print-config-stderr 2>&1 | sed '/Starting Loki/q' | tr -d '\r') <(docker run --rm -t -v "${PWD}":/config grafana/loki:${NEW_LOKI} -config.file=/etc/loki/${CONFIG_FILE} -print-config-stderr 2>&1 | sed '/Starting Loki/q' | tr -d '\r') | less -R
The tr -d '\r' is likely not necessary for most people, seems like WSL2 was sneaking in some windows newline characters...
The output is incredibly verbose as it shows the entire internal config struct used to run Loki, you can play around with the diff command if you prefer to only show changes or a different style output.
The next Loki release introduces a new configuration option (i.e. -distibutor.max-recv-msg-size) for the distributors to control the max receive size of uncompressed stream data. The new options's default value is set to 100MB.
Supported clients should check the configuration options for max send message size if applicable.
{{< admonition type="note" >}} With the move to the Grafana-community/helm-charts repository, the chart numbering has changed. Major version updates signal breaking changes in the chart. For more information, refer to the README. {{< /admonition >}}
If you are upgrading from the Helm chart previously hosted in the Loki repository (chart version 6.x) to the Grafana Community Helm chart (chart version 18.x), refer to the dedicated migration guide: Upgrade to the Community Helm chart.
If you prefixed the sidecar container with a private registry (sidecar.image.repository), this is no longer necessary and is deprecated as the global registry is used starting with Helm chart 6.46.1. Therefore please use global.imageRegistry or alternatively, sidecar.image.registry for more fine-grained control.
For most images used in the helm chart, a .digest is available to pin an image to a specific hash. The sidecar images diverges from this convention by introducing a .tag.
Starting with Helm chart 6.46.1, the .tag is deprecated and .digest should be used.
{{< admonition type="warning" >}} Helm chart version 6.46.0 introduces a breaking change that affects users who rely on the default service account name to bind external identity, such as AWS EKS Pod Identity, IAM Roles for Service Accounts (IRSA), GCP Workload Identity, or Azure Workload Identity. {{< /admonition >}}
Starting with Helm chart 6.46.0 (#19590), when serviceAccount.create is true and serviceAccount.name is not set, the default service account name is now derived from the chart's fullname template instead of the chart name. For example:
| Deployment | Default service account name before 6.46.0 | Default service account name in 6.46.0 and later |
|---|---|---|
| Grafana Enterprise Logs (GEL) | enterprise-logs | <release-name>-enterprise-logs |
| Open source Loki | loki | <release-name>-loki |
If an external identity, such as an AWS IAM role through EKS Pod Identity or IRSA, is bound to the previous service account name, your pods lose access to that identity after the upgrade. For object-storage backends, this typically results in an outage because Loki components can no longer read or write to the bucket.
Recommended action:
To preserve the previous behavior and avoid the rename, set the service account name explicitly in your values.yaml:
serviceAccount:
name: enterprise-logs # use "loki" for open source deployments
Setting serviceAccount.name explicitly is forward-compatible and works on both pre- and post-6.46.0 chart versions, so it is also the recommended setting going forward if you want the service account name to be independent of the Helm release name.
If you have already upgraded and your pods have lost cloud-provider IAM access, you have two options:
serviceAccount.name to the previous default (for example, enterprise-logs) and run helm upgrade again. The previously bound external identity will resume working.{{< admonition type="warning" >}} Helm chart version 6.34.0 introduces a breaking change that affects users with zone-aware ingester replication enabled. {{< /admonition >}}
If you are using zone-aware ingesters (ingester.zoneAwareReplication.enabled: true), upgrading to Helm chart 6.34.0 requires manual StatefulSet deletion before the upgrade. This is due to a fix for the serviceName field in zone-aware ingester StatefulSets, which is an immutable field in Kubernetes.
For detailed upgrade instructions, see: Helm Chart 6.x Upgrade Guide - Zone-aware Ingester Breaking Change
Key points:
ingester.zoneAwareReplication.enabled: true--cascade=orphanLoki uses the official AWS SDK for configuring and communication with S3 object storage. Version 1 of the SDK reached its end of life on 31st, 2025, and therefore had to be replaced with Version 2. While the user-facing configuration in Loki did not change, internal functionality of the object store client did change, without affecting functionality of Loki.
Please refer to the full release notes of v2 https://github.com/aws/aws-sdk-go-v2/releases/tag/release-2025-01-15 for further information and whether you may be impacted by any of the changes.
Starting in Loki version 3.5.8, the busybox utility was removed from the official Loki Docker images. This means that shell utilities like /bin/sh are no longer available inside the container by default.
exec into the Loki container to use a shell as before. Commands like kubectl exec -it podname -- sh or docker exec -it containername sh will fail because /bin/sh does not exist in the image.ls, cat, ps) are also not available inside the container.Removing BusyBox addresses the following CVEs:
If you need to debug or inspect a Loki container:
kubectl debug -it <pod-name> --image=busybox --target=<container-name>
kubectl cp or docker cp to move logs or config files for inspection.FROM grafana/loki:3.5.8
USER root
RUN apk add --no-cache busybox
USER 10001
Loki release 3.4.0 introduces new object storage clients based on the Thanos Object Storage Client Go module, this is an opt-in feature. In a future release, this will become the default way of configuring storage and the existing storage clients will be deprecated.
The new storage configuration deviates from the existing format. Refer to the Thanos storage configuration reference to view the complete list of supported storage providers and their configuration options.
The documentation now also includes a migration guide and configuration examples for using Thanos-based storage clients.
With Loki 3.3.0, the bloom block format changed and any previously created block is incompatible with the new format.
Before upgrading, we recommend deleting all the existing bloom blocks in the object store. We store bloom blocks and
metas inside the bloom path in the configured object store. To get rid of all the bloom blocks, delete all the objects
inside the bloom path in the object store.
The API endpoint for instant queries /api/v1/query now returns a HTTP status 400 (Bad Request) when the provided query
parameter contains a log selector query instead of returning inconsistent results. Please use the range query endpoint
/api/v1/query_range (Range type in Grafana Explore) instead.
Loki changes the default value of -ruler.alertmanager-use-v2 from false to true. Alertmanager APIv1 was deprecated in Alertmanager 0.16.0 and is removed as of 0.27.0.
{{< admonition type="note" >}} Experimental features are subject to rapid change and/or removal, which can introduce breaking changes even between minor version. They also don't follow the deprecation lifecycle of regular features. {{< /admonition >}}
The bloom compactor component, which builds bloom filter blocks for query acceleration, has been removed in favor of two new components: bloom planner and bloom builder. Please consult the Query Acceleration with Blooms docs for more information.
CLI arguments (and their YAML counterparts) of per-tenant settings that have been removed as part of this change:
-bloom-compactor.enable-compaction-bloom-compactor.shard-size-bloom-compactor.shard-size-bloom-compactor.shard-sizeCLI arguments of per-tenant settings that have been moved to a different prefix as part of this change:
-bloom-compactor.max-page-size changed to -bloom-builder.max-page-size-bloom-compactor.max-block-size changed to -bloom-builder.max-block-size-bloom-compactor.ngram-length changed to -bloom-builder.ngram-length-bloom-compactor.ngram-skip changed to -bloom-builder.ngram-skip-bloom-compactor.false-positive-rate changed to -bloom-builder.false-positive-rate-bloom-compactor.block-encoding changed to -bloom-builder.block-encodingTheir YAML counterparts in the limits_config block are kept identical.
All other CLI arguments (and their YAML counterparts) prefixed with -bloom-compactor. have been removed.
{{< admonition type="note" >}}
If you have questions about upgrading to Loki 3.0, please join us on the community Slack in the #loki-3 channel.
Or leave a comment on this Github Issue. {{< /admonition >}}
{{< admonition type="tip" >}} If you have not yet migrated to TSDB, do so before you upgrade to Loki 3.0. {{< /admonition >}}
Loki 3.0 is a major version increase and comes with several breaking changes.
Here is the shortlist of things we think most people may encounter:
tsdb and v13 schema or Loki won't start. Refer to Structured Metadata, Open Telemetry, Schemas and Indexes.shared_store config is removed. Refer to Removed shared_store and shared_store_key_prefix from shipper configuration.service_name label on ingestion. Refer to service_name label.If you would like to see if your existing configuration will work with Loki 3.0:
loki-config.yaml.docker run --rm -t -v "${PWD}":/config grafana/loki:3.0.0 -config.file=/config/loki-config.yaml -verify-config=true
{{< admonition type="note" >}} If you introduce a new schema_config entry it may cause additional validation errors. {{< /admonition >}}
{{< admonition type="tip" >}}
If you configure path_prefix in the common config section this can help save a lot of configuration. Refer to the Common Config Docs.
{{< /admonition >}}
The Helm chart has gone through some significant changes and has a separate upgrade guide: Upgrading to Helm 6.x.
{{< admonition type="caution" >}} Before upgrading your software from Loki 2.x to 3.0, you should follow the instructions to Migrate to TSDB. {{< /admonition >}}
A flagship feature of Loki 3.0 is native support for the Open Telemetry Protocol (OTLP). This is made possible by a new feature in Loki called Structured Metadata, a place for metadata which doesn't belong in labels or log lines. OTel resources and attributes are often a great example of data which doesn't belong in the index nor in the log line.
Structured Metadata is enabled by default in Loki 3.0, however, it requires your active schema be using both the tsdb index type AND the v13 storage schema. If you are not using both of these you have two options:
allow_structured_metadata: false in the limits_config section or set the command line argument -validation.allow-structured-metadata=false.service_name labelLoki 3.0 will automatically assign a service_name label to all ingested logs by default. A service name is something required by Open Telemetry semantic conventions and is something Grafana Labs is building into our future user interface and query experiences.
Loki will attempt to create the service_name label by looking for the following labels in this order:
If no label is found matching the list, a value of unknown_service is applied.
You can change this list by providing a list of labels to discover_service_name in the limits_config block.
{{< admonition type="note" >}}
If you are already using a service_label, Loki will not make a new assignment.
{{< /admonition >}}
You can disable this by providing an empty value for discover_service_name.
shared_store and shared_store_key_prefix from shipper configurationThe following CLI flags and the corresponding YAML settings to configure shared store for TSDB and BoltDB shippers are now removed:
-boltdb.shipper.shared-store-tsdb.shipper.shared-storeGoing forward the object_store setting in the period_config will be used to configure the store for the index.
This enforces chunks and index files to reside together in the same storage bucket for a given period.
We are removing the shared store setting in an effort to simplify storage configuration and reduce the possibility for misconfiguration.
{{< admonition type="warning" >}}
With this change Loki no longer allows storing chunks and indexes for a given period in different storage buckets.
This is a breaking change for setups that store chunks and indexes in different storage buckets by setting -boltdb.shipper.shared-store or -tsdb.shipper.shared-store to a value different from object_store in period_config.
{{< /admonition >}}
-boltdb.shipper.shared-store,-tsdb.shipper.shared-store or their corresponding YAML setting before, no changes are required as part of the upgrade.-boltdb.shipper.shared-store or its YAML setting:
object_store for all the periods that use boltdb-shipper as index type, no additional changes are required besides removing the usage of the deleted configuration option.-boltdb.shipper.shared-store does not match object_store.
object_store.-tsdb.shipper.shared-store or its YAML setting:
object_store for all the periods that use tsdb as index type, no additional changes are required besides removing the usage of the deleted configuration option.-tsdb.shipper.shared-store does not match object_store.
object_store.The following CLI flags and the corresponding YAML settings to configure a path prefix for TSDB and BoltDB shippers are now removed:
-boltdb.shipper.shared-store.key-prefix-tsdb.shipper.shared-store.key-prefixPath prefix for storing the index can now be configured by setting path_prefix under index key in period_config.
This enables users to change the path prefix by adding a new period config.
period_config:
index:
path_prefix: "index/"
period: 24h
{{< admonition type="note" >}}
path_prefix only applies to TSDB and BoltDB indexes. This setting has no effect on legacy indexes.
{{< /admonition >}}
path_prefix defaults to index/ which is same as the default value of the removed configurations.
-boltdb.shipper.shared-store.key-prefix, -tsdb.shipper.shared-store.key-prefix or the corresponding YAML setting previously.-boltdb.shipper.shared-store.key-prefix or its YAML setting to a value other than index/, ensure that all the existing period configs that use boltdb-shipper as the index have path_prefix set to the value previously configured.-tsdb.shipper.shared-store.key-prefix or its YAML setting to a value other than index/, ensure that all the existing period configs that use tsdb as the index have the path_prefix set to the value previously configured.shared_store and shared_store_key_prefix from compactor configurationThe following CLI flags and the corresponding YAML settings to configure the shared store and path prefix for compactor are now removed:
-boltdb.shipper.compactor.shared-store-boltdb.shipper.compactor.shared-store.key-prefixGoing forward compactor will run compaction and retention on all the object stores configured in period configs where the index type is either tsdb or boltdb-shipper.
delete_request_store should be explicitly configured-compactor.delete-request-store or its YAML setting should be explicitly configured when retention is enabled, this is required for storing delete requests.
The path prefix under which the delete requests are stored is decided by -compactor.delete-request-store.key-prefix, it defaults to index/.
async_cache_write_back_concurrency and async_cache_write_back_buffer_size have been removedThese configurations were redundant with the Background configuration in the cache-config.
async_cache_write_back_concurrency can be set with writeback_goroutines
async_cache_write_back_buffer_size can be set with writeback_buffer
additionally the Background configuration also lest you set writeback_size_limit which can be used to set a maximum amount of memory to use for writeback objects vs a count of objects.
The already deprecated handler /ingester/flush_shutdown is removed in favor of /ingester/shutdown?flush=true.
max_transfer_retries is removedThe setting max_transfer_retries (-ingester.max-transfer-retries) is removed in favor of the Write Ahead log (WAL).
It was used to allow transferring chunks to new ingesters when the old ingester was shutting down during a rolling restart.
Alternatives to this setting are:
flush_on_shutdown (-ingester.flush-on-shutdown) to flush to long-term storage on shutdowns./shutdown?flush=true endpoint.default section of the runtime overrides config fileThis was introduced in 2.9 and likely not widely used. This only affects you if you run Loki with a runtime config file AND you had populated the new default block added in 2.9.
The default block was removed and instead a top level config now exists in the standard Loki config called operational_config, you can set default values here for runtime configs.
use_boltdb_shipper_as_backup is removedThe setting use_boltdb_shipper_as_backup (-tsdb.shipper.use-boltdb-shipper-as-backup) was a remnant from the development of the TSDB storage.
It was used to allow writing to both TSDB and BoltDB when TSDB was still highly experimental.
Since TSDB is now stable and the recommended index type, the setting has become irrelevant and therefore was removed.
The previous default value false is applied.
store.max-look-back-period CLI flag and the corresponding YAML settings. Use querier.max-query-lookback config instead.-querier.engine.timeout CLI flag and the corresponding YAML setting.query_timeout from the querier YAML section. Instead of configuring query_timeout under querier, you now configure it in Limits Config.s3.sse-encryption is removed. AWS now defaults encryption of all buckets to SSE-S3. Use sse.type to set SSE type.ruler.wal-cleaer.period is removed. Use ruler.wal-cleaner.period instead.experimental.ruler.enable-api is removed. Use ruler.enable-api instead.split_queries_by_interval is removed from query_range YAML section. You can instead configure it in Limits Config.frontend.forward-headers-list CLI flag and its corresponding YAML setting are removed.frontend.cache-split-interval CLI flag is removed. Results caching interval is now determined by querier.split-queries-by-interval.querier.worker-parallelism CLI flag and its corresponding yaml setting are now removed as it does not offer additional value to already existing querier.max-concurrent.
We recommend configuring querier.max-concurrent to limit the max concurrent requests processed by the queriers.ruler.evaluation-delay-duration CLI flag and the corresponding YAML setting are removed.validation.enforce-metric-name CLI flag and the corresponding YAML setting are removed.boltdb.shipper.compactor.deletion-mode CLI flag and the corresponding YAML setting are removed. You can instead configure the compactor.deletion-mode CLI flag or deletion_mode YAML setting in Limits Config.boltdb.shipper.compactor. are removed. You can instead use CLI flags with the compactor. prefix.The loki_distributor_ingester_append_failures_total metric has been removed in favour of loki_distributor_ingester_append_timeouts_total.
This new metric will provide a more clear signal that there is an issue with ingesters, and this metric can be used for high-signal alerting.
{{< responsive-table >}}
| configuration | new default | old default | notes |
|---|---|---|---|
compactor.delete-max-interval | 24h | 0 | splits the delete requests into intervals no longer than delete_max_interval |
distributor.max-line-size | 256KB | 0 | - |
ingester.sync-period | 1h | 0 | ensures that the chunk cuts for a given stream are synchronized across the ingesters in the replication set. Helps with deduplicating chunks. |
ingester.sync-min-utilization | 0.1 | 0 | - |
frontend.max-querier-bytes-read | 150GB | 0 | - |
frontend.max-cache-freshness | 10m | 1m | - |
frontend.max-stats-cache-freshness | 10m | 0 | - |
frontend.embedded-cache.max-size-mb | 100MB | 1GB | embedded results cache size now defaults to 100MB |
memcached.batchsize | 4 | 1024 | - |
memcached.parallelism | 5 | 100 | - |
querier.compress-http-responses | true | false | compress response if the request accepts gzip encoding |
querier.max-concurrent | 4 | 10 | Consider increasing this if queriers have access to more CPU resources. Note that you risk running into out of memory errors if you set this to a very high value. |
querier.split-queries-by-interval | 1h | 30m | - |
querier.tsdb-max-query-parallelism | 128 | 512 | - |
query-scheduler.max-outstanding-requests-per-tenant | 32000 | 100 | - |
validation.max-label-names-per-series | 15 | 30 | - |
legacy-read-mode | false | true | Deprecated. It will be removed in the next minor release. |
{{< /responsive-table >}}
Automatic stream sharding helps keep the write load of high volume streams balanced across ingesters and helps to avoid hot-spotting. Check out the operations page for more information
The TSDB index type has support for caching results for 'stats' and 'volume' queries which are now enabled by default.
'label' and 'series' requests can be cached now too and this is enabled by default.
All of these are cached to the results_cache which is configured in the query_range config section. By default, an in memory cache is used.
Write dedupe cache is deprecated because it not required by the newer single store indexes (TSDB and boltdb-shipper). If you using a legacy index type, consider migrating to TSDB (recommended).
The following embedded cache metrics are removed. Instead use loki_cache_fetched_keys, loki_cache_hits, loki_cache_request_duration_seconds which instruments requests made to the configured cache (embeddedcache, memcached or redis).
querier_cache_added_totalquerier_cache_gets_totalquerier_cache_misses_totalThe following embedded cache metrics are renamed:
querier_cache_added_new_total is renamed to loki_embeddedcache_added_new_totalquerier_cache_evicted_total is renamed to loki_embeddedcache_evicted_totalquerier_cache_entries is renamed to loki_embeddedcache_entriesquerier_cache_memory_bytes is renamed to loki_embeddedcache_memory_bytesAlready deprecated metric querier_cache_stale_gets_total is now removed.
Some Loki metrics started with the prefix cortex_. In this release they will be changed so they start with loki_. To keep them at cortex_ change the metrics_namespace from the default loki to cortex. These metrics will be changed:
cortex_distributor_ingester_clientscortex_dns_failures_totalcortex_dns_lookups_totalcortex_dns_provider_resultscortex_frontend_query_range_duration_seconds_bucketcortex_frontend_query_range_duration_seconds_countcortex_frontend_query_range_duration_seconds_sumcortex_ingester_flush_queue_lengthcortex_kv_request_duration_seconds_bucketcortex_kv_request_duration_seconds_countcortex_kv_request_duration_seconds_sumcortex_member_consul_heartbeats_totalcortex_prometheus_last_evaluation_samplescortex_prometheus_notifications_alertmanagers_discoveredcortex_prometheus_notifications_dropped_totalcortex_prometheus_notifications_errors_totalcortex_prometheus_notifications_latency_secondscortex_prometheus_notifications_latency_seconds_countcortex_prometheus_notifications_latency_seconds_sumcortex_prometheus_notifications_queue_capacitycortex_prometheus_notifications_queue_lengthcortex_prometheus_notifications_sent_totalcortex_prometheus_rule_evaluation_duration_secondscortex_prometheus_rule_evaluation_duration_seconds_countcortex_prometheus_rule_evaluation_duration_seconds_sumcortex_prometheus_rule_evaluation_failures_totalcortex_prometheus_rule_evaluations_totalcortex_prometheus_rule_group_duration_secondscortex_prometheus_rule_group_duration_seconds_countcortex_prometheus_rule_group_duration_seconds_sumcortex_prometheus_rule_group_interval_secondscortex_prometheus_rule_group_iterations_missed_totalcortex_prometheus_rule_group_iterations_totalcortex_prometheus_rule_group_last_duration_secondscortex_prometheus_rule_group_last_evaluation_timestamp_secondscortex_prometheus_rule_group_rulescortex_query_frontend_connected_schedulerscortex_query_frontend_queries_in_progresscortex_query_frontend_retries_bucketcortex_query_frontend_retries_countcortex_query_frontend_retries_sumcortex_query_scheduler_connected_frontend_clientscortex_query_scheduler_connected_querier_clientscortex_query_scheduler_inflight_requestscortex_query_scheduler_inflight_requests_countcortex_query_scheduler_inflight_requests_sumcortex_query_scheduler_queue_duration_seconds_bucketcortex_query_scheduler_queue_duration_seconds_countcortex_query_scheduler_queue_duration_seconds_sumcortex_query_scheduler_queue_lengthcortex_query_scheduler_runningcortex_ring_member_heartbeats_totalcortex_ring_member_tokens_ownedcortex_ring_member_tokens_to_owncortex_ring_memberscortex_ring_oldest_member_timestampcortex_ring_tokens_totalcortex_ruler_client_request_duration_seconds_bucketcortex_ruler_client_request_duration_seconds_countcortex_ruler_client_request_duration_seconds_sumcortex_ruler_clientscortex_ruler_config_last_reload_successfulcortex_ruler_config_last_reload_successful_secondscortex_ruler_config_updates_totalcortex_ruler_managers_totalcortex_ruler_ring_check_errors_totalcortex_ruler_sync_rules_totalThe metrics_namespace setting is deprecated already. It will be removed in the next minor release. The default prefix will be loki then.
Previously LogCLI used to fetch remote schema from the store configured in -boltdb.shipper.shared-store when -remote-schema is set to true.
A new CLI flag -schema-store is introduced as a replacement to configure the store for retrieving remote schema.
To upgrade from one of the versions of Loki 2.x, refer to Upgrade Loki 2.x versions.
To upgrade from one of the versions of Loki 1.x, refer to Upgrade Loki 1.x versions.