Manage storage

You can read a high level overview of Loki storage here

Grafana Loki needs to store two different types of data: chunks and indexes.

When using Accelerated Search (experimental), then a third data type is used: bloom blocks.

Loki receives logs in separate streams, where each stream is uniquely identified by its tenant ID and its set of labels. As log entries from a stream arrive, they are compressed as chunks and saved in the chunks store. See chunk format for how chunks are stored internally.

The index stores each stream's label set and links them to the individual chunks. Refer to the Loki configuration for details on how to configure the storage and the index.

For more information:

• Table Manager
• Retention
• Logs Deletion

Store Types

✅ Supported index stores

• Single Store TSDB index store which stores TSDB index files in the object store. This is the recommended index store for Loki 2.8 and newer.
• Single Store BoltDB (boltdb-shipper) index store which stores boltdb index files in the object store. Recommended store for Loki 2.0 through 2.7.x.

❌ Deprecated index stores

• Amazon DynamoDB. Support for this is deprecated and will be removed in a future release.
• Google Bigtable. Support for this is deprecated and will be removed in a future release.
• Apache Cassandra. Support for this is deprecated and will be removed in a future release.
• BoltDB (doesn't work when clustering Loki)

✅ Supported and recommended chunks stores

• Amazon Simple Storage Service (S3)
• Google Cloud Storage (GCS)
• Microsoft Azure Blob Storage
• IBM Cloud Object Storage (COS)
• Baidu Object Storage (BOS)
• Alibaba Object Storage Service (OSS)

⚠️ Supported chunks stores, not typically recommended for production use

• Filesystem (please read more about the filesystem to understand the pros/cons before using with production data)
• S3 API compatible storage, such as MinIO

❌ Deprecated chunks stores

• Amazon DynamoDB. Support for this is deprecated and will be removed in a future release.
• Google Bigtable. Support for this is deprecated and will be removed in a future release.
• Apache Cassandra. Support for this is deprecated and will be removed in a future release.

Cloud Storage Permissions

S3

When using S3 as object storage, the following permissions are needed:

• s3:ListBucket
• s3:PutObject
• s3:GetObject
• s3:DeleteObject (if running the Single Store (boltdb-shipper) compactor)

Resources: arn:aws:s3:::<bucket_name>, arn:aws:s3:::<bucket_name>/*

See the AWS deployment section on the storage page for a detailed setup guide.

DynamoDB

{{< admonition type="note" >}} DynamoDB support is deprecated and will be removed in a future release. {{< /admonition >}}

When using DynamoDB for the index, the following permissions are needed:

• dynamodb:BatchGetItem
• dynamodb:BatchWriteItem
• dynamodb:DeleteItem
• dynamodb:DescribeTable
• dynamodb:GetItem
• dynamodb:ListTagsOfResource
• dynamodb:PutItem
• dynamodb:Query
• dynamodb:TagResource
• dynamodb:UntagResource
• dynamodb:UpdateItem
• dynamodb:UpdateTable
• dynamodb:CreateTable
• dynamodb:DeleteTable (if table_manager.retention_period is more than 0s)

Resources: arn:aws:dynamodb:<aws_region>:<aws_account_id>:table/<prefix>*

• dynamodb:ListTables

Resources: *

AutoScaling

If you enable autoscaling from table manager, the following permissions are needed:

Application Autoscaling

• application-autoscaling:DescribeScalableTargets
• application-autoscaling:DescribeScalingPolicies
• application-autoscaling:RegisterScalableTarget
• application-autoscaling:DeregisterScalableTarget
• application-autoscaling:PutScalingPolicy
• application-autoscaling:DeleteScalingPolicy

Resources: *

IAM

• iam:GetRole
• iam:PassRole

Resources: arn:aws:iam::<aws_account_id>:role/<role_name>

IBM Cloud Object Storage

When using IBM Cloud Object Storage (COS) as object storage, IAM Writer role is needed.

See the IBM Cloud Object Storage section on the storage page for a detailed setup guide.

Chunk Format

// Header
+-----------------------------------+
| Magic Number (uint32, 4 bytes)    |
+-----------------------------------+
| Version (1 byte)                  |
+-----------------------------------+
| Encoding (1 byte)                 |
+-----------------------------------+

// Blocks
+--------------------+----------------------------+
| block 1 (n bytes)  | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+
| block 1 (n bytes)  | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+
| ...                                             |
+--------------------+----------------------------+
| block N (n bytes)  | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+

// Metas
+------------------------------------------------------------------------------------------------------------------------+
| #blocks (uvarint)                                                                                                      |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| ...                                                                                                                    |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| checksum (uint32, 4 bytes)                                                                                             | 
+------------------------------------------------------------------------------------------------------------------------+

// Structured Metadata
+---------------------------------+
| #labels (uvarint)               |
+---------------+-----------------+
| len (uvarint) | value (n bytes) |
+---------------+-----------------+
| ...                             |
+---------------+-----------------+
| checksum (uint32, 4 bytes)      |
+---------------------------------+

// Footer
+-----------------------+--------------------------+
| len (uint64, 8 bytes) | offset (uint64, 8 bytes) |   // offset to Structured Metadata
+-----------------------+--------------------------+
| len (uint64, 8 bytes) | offset (uint64, 8 bytes) |   // offset to Metas
+-----------------------+--------------------------+