Parquet-based Storage

• Author: Alan Protasio, Ben Ye
• Date: April 2025
• Status: Proposed

Background

Since the introduction of Block Storage in Cortex, TSDB format and Store Gateway is the de-facto way to query long term data on object storage. However, it presents several significant challenges:

TSDB Format Limitations

TSDB format, while efficient for write-heavy workloads on local SSDs, is not designed for object storage:

• Index relies heavily on random reads to serve queries, where each random read becomes a request to object store
• In order to reduce requests to object store, requests needs to be merged, leading to higher overfetch
• Index relies on postings, which can be a huge bottleneck for high cardinality data

Store Gateway Operational Challenges

Store Gateway is originally introduced in Thanos. Both Cortex and Thanos community have been collaborating to add a lot of optimizations to Store Gateway. However, it has its own problems related to the design.

1. Resource Intensive

   • Requires significant local disk space to store index headers
   • High memory utilization due to index header mmap
   • Often needs over-provisioning to handle query spikes

2. State Management and Scaling Difficulties

   • Requires complex data sharding when scaling. Often causing issues such as consistency check failure. Hard to configure for users
   • Initial sync causes long startup time. This affects service availability on both scaling and failure recovery scenario

3. Query Inefficiencies

   • Attempts to minimize storage requests often lead to overfetching, causing high bandwidth usage
   • Complex caching logic with varying effectiveness. Latency varies a lot when cache miss
   • Processes single block with one goroutine, leading to high latency for large blocks and cannot scale without complex data partitioning

Why Parquet?

Apache Parquet is a columnar storage format designed specifically for efficient data storage and retrieval from object storage systems. It offers several key advantages that directly address the problems we face with TSDB and Store Gateway:

• Data organized by columns rather than rows, reduces number of requests to object storage as only limited IO is required to fetch the whole column
• Rich file metadata and index, no local state like index header required to query the data, making it stateless
• Advanced compression techniques reduce storage costs and improve query performance
• Parallel processing friendly using Parquet Row Group

There are other benefits of Parquet formats, but they are not directly related to the proposal:

• Wide ecosystem and tooling support
• Column pruning opportunity using projection pushdown

Out of Scope

• Allow Ingester and Compactor to create Parquet files instead of TSDB blocks directly. This could be in the future roadmap but this proposal only focuses on converting and querying Parquet files.

Proposed Design

Components

There are 2 new Cortex components/modules introduced in this design.

1. Parquet Converter

Parquet Converter is a new component that converts TSDB blocks on object store to Parquet file format.

It is similar to compactor, however, it only converts single block. The converted Parquet files will be stored in the same TSDB block folder so that the lifecycle of Parquet file will be managed together with the block.

Only certain blocks can be configured to convert to Parquet file and it can be block duration based, for example we only convert if block duration is >= 12h.

2. Parquet Queryable

Similar to the existing distributorQueryable and blockStorageQueryable, Parquet queryable is a queryable implementation which allows Cortex to query parquet files and can be used in both Cortex Querier and Ruler.

If Parquet queryable is enabled, block storage queryable will be disabled and Cortex querier will not query Store Gateway anymore. distributorQueryable remains unchanged so it still queries Ingesters.

Parquet queryable uses bucket index to discovers parquet files in object storage. The bucket index is the same as the existing TSDB bucket index file, but using a different name bucket-index-parquet.json.gz. It is updated periodically by Cortex Compactor/Parquet Converter if parquet storage is enabled.

Cortex querier remains a stateless component when Parquet queryable is enabled.

Architecture

┌──────────┐    ┌─────────────┐    ┌──────────────┐
│ Ingester │───>│   TSDB      │───>│   Parquet    │
└──────────┘    │   Blocks    │    │  Converter   │
                └─────────────┘    └──────────────┘
                                          │
                                          v
┌──────────┐    ┌─────────────┐    ┌──────────────┐
│  Query   │───>│   Parquet   │───>│    Parquet   │
│ Frontend │    │   Querier   │    │    Files     │
└──────────┘    └─────────────┘    └──────────────┘

Data Format

Parquet file is converted from TSDB block so it follows the same time range constraint.

If the largest block is 1 day then parquet file can go up to 1 day. Max block range is configurable in Cortex but default value is 24h. So following schema will use 24h as example.

Schema Overview

The Parquet format consists of two types of files:

1. Labels Parquet File

   • Each row represents a unique time series
   • Each column corresponds to a label name (e.g., __name__, label1, ..., labelN)
   • Row groups are sorted by __name__ alphabetically in ascending order

2. Chunks Parquet File

   • Maintains row and row group order matching the Labels file
   • Contains multiple chunk columns for time-series data ordered by time. With 3 chunk columns for example, each column covers 8h of chunks: 0-8h, 8h-16h, 16-24h. It is possible that a single TSDB chunk spans over time ranges of 2 columns and Parquet file writer needs to split and re-encode chunks for each chunk column.

Column Specifications

data_cols_count will be a parquet file metadata and its value is default to 3 but it can be configurable to adjust for different usecases.

Open Questions

1. Should we use Parquet Gateway to replace Store Gateway
   • Separate query engine and storage
   • We can make Parquet Gateway semi-stateful like data locality for better performance

Acknowledgement

We'd like to give huge credits for people from the Thanos community who started this initiative.

• Filip Petkovski and his initial talk about Parquet
• Michael Hoffmann and his great work of parquet poc