docs/internals/adr/001-parquet-data-model.md
Quickwit's metrics pipeline stores data in Parquet files. A fundamental design question is how metrics data is represented at the row level: what does one row in a Parquet file correspond to?
Two models are in consideration:
This decision is foundational because it determines the shape of every downstream system: how compaction merges data, how sort schemas are defined, how DataFusion queries are structured, and what encodings are effective.
Each row in a Parquet split represents a single data point: one metric value at one timestamp for one timeseries.
Point-per-row is chosen because:
We do not support LWW. If two data points arrive for the same timeseries at the same timestamp in separate ingest requests, both are stored. There is no per-point deduplication at the storage layer. This eliminates:
Existing deduplication guarantees. Quickwit provides deduplication at coarser granularities than individual points:
Per-point deduplication is not implemented. If the same metric data point (identical metric name, tags, timestamp, and value) arrives in two separate ingest requests — due to client retries, overlapping sources, or upstream replay — both copies are stored. Per-point deduplication would require either sticky routing (binding a timeseries to a specific shard) or a dedup index (tracking recently-seen points), both of which add significant complexity and reliability risk. If per-point deduplication becomes a product requirement, it should be designed as a separate capability rather than baked into the storage data model. See GAP-005.
Interpolation (filling gaps in a timeseries, aligning timestamps across series) is not performed during ingestion or compaction. The storage layer stores raw points. Interpolation is a query-time operation.
This decouples storage format from query semantics. Different query patterns may require different interpolation strategies (linear, last-value, none), and embedding one strategy in the storage format would constrain future flexibility.
When data is sorted by a sort schema (see ADR-002), the explicit sort columns may not be granular enough to distinguish individual point sources. For example, if the sort schema is metric_name|env, hundreds of hosts within the same environment produce interleaved points within each (metric_name, env) group.
To improve locality, the data model includes an optional timeseries_id column: a hash of all tag names and values. When present, it is placed after explicit sort columns and before timestamp in the sort schema, acting as a tiebreaker that clusters points from the same tag combination.
Properties:
host or container), timeseries_id adds little value and can be omitted. It can be added or removed from the sort schema at any time — this is a schema change handled by the normal transition mechanism (new splits use the new schema, old splits age out via retention).Limitations:
NA and an actual value), the hash changes and points from what a user would consider "the same source" get different timeseries_id values. This degrades locality but never correctness. If tag flapping is prevalent, omitting timeseries_id and relying on explicit sort columns alone may be preferable.timeseries_id in per-column min/max/regex metadata.Map columns are fundamentally non-columnar: all key-value pairs for a row are packed into a single column value. This has two consequences for Parquet storage:
host column with sorted data produces long runs of repeated values; a map column containing host alongside env, region, and dozens of other keys produces no useful runs.attributes['host'] = 'web-01', the query engine must deserialize the entire map for each row and search for the key. There is no way to seek to host values specifically, and Parquet page-level statistics are meaningless for a map column (the min/max of a serialized map has no relationship to the values of individual keys within it).Schema-on-read: attributes as columns. A more effective storage representation is to extract each attribute into its own Parquet column at write time. Rather than storing Map{"host": "web-01", "env": "prod", "region": "us-east-1"} as one map value, we store three separate columns: attr.host = "web-01", attr.env = "prod", attr.region = "us-east-1". Each column is independently typed, independently compressed, and independently accessible for predicate evaluation and page-level pruning.
This is a schema-on-read approach: the storage layer stores data in whatever shape arrives, creating columns as needed, and any schema interpretation happens at query time. There is no requirement for the schema to be specified up front — new attribute keys that appear in incoming data produce new columns automatically. This is a key characteristic of cloud-native observability storage: "store all the data that the user sends, in whatever types they send it, resolving any ambiguity at query time."
Dense vs sparse columns. Not every attribute needs its own column. Attributes that appear in <1% of rows produce extremely sparse columns that waste storage on null markers and add schema complexity. A practical threshold is to extract attributes as dedicated columns when they are dense (present in >1% of rows) and keep rare attributes in a residual map column. The density threshold is a tunable parameter. Over time, compaction could consolidate: an attribute that starts sparse (few sources report it) but becomes dense (adopted widely) can be promoted to its own column.
Implications for the sort schema. Extracting attributes into columns is a prerequisite for effective sorting. The sort schema (ADR-002) references column names like host, env, metric_name. If these values are buried inside a map column, the sort is impossible — the writer cannot extract sort keys from a serialized map efficiently. Columnar attributes enable the sort schema to reference any attribute by name, and enable page-level statistics on sort columns that make intra-file pruning effective.
Transition. The current OTel map-based ingestion format is the starting point. The indexing pipeline can extract attributes into columns at write time, presenting the original OTel map interface at the API boundary while storing columnar data internally. This is transparent to ingest clients — they continue sending OTel-format data. Queries can access attributes either by the original map path (for compatibility) or by direct column access (for performance). The storage representation is an internal optimization, not a change to the external data model.
The point-per-row model's performance depends on columnar encodings being preserved through the query pipeline. Currently, RLE and dictionary encoding are decoded to plain arrays early in DataFusion's execution. As DataFusion grows operator-level support for these encodings, the performance benefits of sorted point-per-row data increase: longer runs in sorted columns translate directly to better RLE compression ratios that are maintained through query execution. This makes point-per-row a bet that improves over time rather than a static trade-off.
These invariants must hold across all code paths (ingestion, compaction, query).
| ID | Invariant | Rationale |
|---|---|---|
| DM-1 | Each row in a Parquet split is exactly one data point: one metric value at one timestamp for one timeseries | Foundational data model. Enables row-level sorted merge without series-level merge logic |
| DM-2 | No last-write-wins. If two data points with the same (metric name, tags, timestamp) arrive in separate ingest requests, both are stored | Eliminates sticky routing, series-level dedup, and ordering dependencies between nodes |
| DM-3 | The storage layer does not perform interpolation. Points are stored as received; interpolation is a query-time operation | Decouples storage format from query semantics |
| DM-4 | timeseries_id, if present, is deterministic: the same canonicalized tag set always produces the same hash value | Required for locality grouping to be consistent across ingestion and compaction |
| DM-5 | timeseries_id persists through compaction without recomputation. The column is written once at ingestion and carried through all subsequent merges | Avoids recomputing hashes during merge (tags may not all be available as separate columns at merge time) |
timeseries_id in the sort schema, points from the same series may be interleaved with points from other series that share the same sort-column values. This is a configuration choice, not an inherent limitation.k8s.cpu.usage, k8s.cpu.limit, k8s.mem.usage as columns sharing one tag set). This is the approach taken by TimescaleDB's hypertables. It would amortize tag storage further but requires significant compactor changes. Worth investigating as future research; it is compatible with point-per-row as an evolution, not a replacement.This ADR applies to metrics (Parquet pipeline). The data model decisions generalize as follows:
timeseries_id equivalent would be a hash of trace attributes for locality grouping.timeseries_id equivalent could group log entries by service/host.The no-LWW and no-storage-interpolation decisions are universal across signals. The timeseries_id concept generalizes to any signal where grouping related records improves compression.
| Date | Decision | Rationale |
|---|---|---|
| 2026-02-19 | Initial ADR created | Establish foundational data model for Parquet metrics pipeline |
| 2026-02-19 | Point-per-row chosen over timeseries-per-row | Simpler compaction, no LWW, standard DataFusion operators. Performance parity via columnar encoding and dictionary/RLE preservation through more operators |
| 2026-02-19 | No LWW semantics | Eliminates sticky routing and series-level dedup. Simplifies ingestion and compaction |
| 2026-02-19 | Dedup clarified: batch-level exists, per-point does not | WAL checkpoints provide exactly-once at the batch level. File-level dedup for queue sources. Per-point dedup not implemented; identified as GAP-005 if needed |
| 2026-02-19 | timeseries_id defined as optional synthetic column | Provides intra-group locality tiebreaker without adding complexity to the core data model |
| 2026-02-19 | Schema-on-read identified as target for attribute storage | OTel map-based attributes are non-columnar, defeating compression and sort/pruning. Extract dense attributes (>1% non-null) into individual columns at write time, keep rare attributes in residual map |
| Component | Location | Status |
|---|---|---|
| Point-per-row Parquet schema | quickwit-parquet-engine/src/schema/fields.rs | Done. Each row is one metric data point |
| Tag columns in Parquet | quickwit-parquet-engine/src/schema/fields.rs | Done. Tags stored as dictionary-encoded columns per row |
| Component | Notes |
|---|---|
| timeseries_id computation | Hash of canonicalized tag key/value pairs, added as column at ingestion |
| timeseries_id persistence through compaction | Column must survive merge without recomputation |
| Schema-on-read attribute extraction | Extract dense attributes from OTel map columns into individual Parquet columns at write time |
| Dense/sparse column threshold | Determine density threshold (e.g., >1% non-null) for column extraction vs residual map |
| Residual map for sparse attributes | Keep rare attributes in a fallback map column alongside extracted dense columns |