V2.3

The Loki team is excited to announce the release of Loki 2.3!

It's been nearly 6 months since 2.2 was released and we have made good use of that time to bring forward several significant improvements and requested features.

2.3 is also the first version of Loki released under the AGPLv3 license. You can read more about our licensing here.

Some parts of the Loki repo will remain Apache-2.0 licensed (mainly clients and some tooling), for more details read LICENSING.md.

Features and enhancements

• Loki now has the ability to apply custom retention based on stream selectors! This will allow much finer control over log retention all of which is now handled by Loki, no longer requiring the use of object store configs for retention.
• Coming along hand in hand with storing logs for longer durations is the ability to delete log streams. The initial implementation lets you submit delete request jobs which will be processed after 24 hours.
• A very exciting new LogQL parser has been introduced: the pattern parser. Much simpler and faster than regexp for log lines that have a little bit of structure to them such as the Common Log Format. This is now Loki's fastest parser so try it out on any of your log lines!
• Extending on the work of Alerting Rules, Loki now accepts recording rules. This lets you turn your logs into metrics and push them to Prometheus or any Prometheus compatible remote_write endpoint.
• LogQL can understand IP addresses! This enables filtering on IP addresses and subnet ranges.

For those of you running Loki as microservices, the following features will improve performance operations significantly for many operations.

• We created an index gateway which takes on the task of downloading the boltdb-shipper index files allowing you to run your queriers without any local disk requirements, this is really helpful in Kubernetes environments where you can return your queriers from Statefulsets back to Deployments and save a lot of PVC costs and operational headaches.
• Ingester queriers are now shardable, this is a significant performance boost for high volume log streams when querying recent data.
• Instant queries can now be split and sharded making them just as fast as range queries.

A very common feature requested has also been included in 2.3:

• 3919 github-vincent-miszczak: Add ingester.autoforget-unhealthy-timeout opt-in feature

Without revisiting the decisions and discussions around the somewhat controversial behavior of unhealthy ingesters, you can now decided how you would like them to be handled: manually or automatically.

Lastly several useful additions to the LogQL query language have been included:

• More text/template functions are included for label_format and line_format with PR 3515, for more information,see the documentation for template functions.
• Also support for math functions withing label_format and line_format was included with 3434.
• Two additional metric functions with some interesting use cases first_over_time and last_over_time were added in PR 3050. These can be useful for some down sampling approaches where instead of taking an average, max, or min of samples over a range in a metrics query, you can select the first or last log line to use from that range.

Upgrade considerations

The path from 2.2.1 to 2.3.0 should be smooth, as always, read the Upgrade Guide for important upgrade guidance.

One change we consider noteworthy however is:

• 3216 sandeepsukhani: check for stream selectors to have at least one equality matcher.

This change now rejects any query which does not contain at least one equality matcher, an example may better illustrate:

{namespace=~".*"}

This query will now be rejected, however there are several ways to modify it for it to succeed:

Add at least one equals label matcher:

{cluster="us-east-1",namespace=~".*"}

Use .+ instead of .*

{namespace=~".+"}

This difference may seem subtle but if we break it down . matches any character, * matches zero or more of the preceding character and + matches one or more of the preceding character. The .* case will match empty values where .+ will not, this is the important difference. {namespace=""} is an invalid request (unless you add another equals label matcher like the example above).

The reasoning for this change has to do with how index lookups work in Loki, if you don't have at least one equality matcher Loki has to perform a complete index table scan which is an expensive and slow operation.

Security fixes

List of security fixes for 2.3.x.

2.3.0 security fixes

2.3.0 contains an important security fix:

• 4020 simonswine: Restrict path segments in TenantIDs (CVE-2021-36156 CVE-2021-36157).

{{< admonition type="note" >}} Exploitation of this vulnerability requires the ability for an attacker to craft and send directly to Loki an X-Scope-OrgID header, end users should not have the ability to create and send this header directly to Loki as it controls access to tenants and is important to control setting of this header for proper tenant isolation and security. We always recommend having a proxy or gateway be responsible for setting the X-Scope-OrgID.{{< /admonition >}}

Bug fixes

Lists of bug fixes for 2.3.x.

2.3.0 bug fixes

• An important fix for leaking resources was patched with 3733, when queries were canceled a goroutine was left running which would hold memory resources creating a memory leak.
• 3686 fixes a panic with the frontend when use with downstream URL. Note we recommend using the GRPC Pull Model, better performance and fair scheduling between tenants can be obtained with the GPRC Pull Model.