docs/internals/tantivy-parquet-architecture.md
Date: 2026-01-27 Updated: 2026-01-29 (with experimental validation) Purpose: Architectural design for replacing Tantivy fast fields with Parquet for logs and traces
Quickwit currently uses Tantivy (full-text search engine) for logs and traces. We have a proven DataFusion/Parquet analytics engine at scale for metrics. This document proposes replacing Tantivy's fast fields with Parquet to enable unified analytics across all Quickwit products (metrics, logs, and traces) while maintaining full-text search capabilities.
Key question: What is the storage cost of replacing Tantivy fast fields with Parquet, and what benefits do we gain from DataFusion ecosystem integration?
Proposal: Replace Tantivy's fast fields (columnar indexes) with Parquet to unify logs/traces analytics with our proven metrics engine, while keeping Tantivy for full-text search and document storage.
Architecture:
SELECT * queriesStorage impact: Experimentally validated at +10.3% total storage overhead for production-scale workloads. Parquet VARIANT (Map type) costs ~39% more than fast fields alone, but enables queryable semi-structured data and the entire DataFusion ecosystem.
Key benefits:
Design principle: Separate full-text search from columnar analytics:
| System | What It Stores | Purpose |
|---|---|---|
| Tantivy (no fast fields) | Inverted indices + doc store + body text | Full-text search + SELECT * retrieval |
| Parquet | Structured columns only (replaces fast fields) | Columnar analytics + filtering via DataFusion |
Storage breakdown (experimental, production-scale attributes):
Current: Tantivy with fast fields (691 MB)
Proposed: Tantivy without fast fields + Parquet (762 MB)
Net cost: +71 MB (+10.3%) to replace fast fields with Parquet VARIANT and gain DataFusion ecosystem.
Included (logs): 16 of 17 OpenTelemetry fields
Excluded (logs): Body field only
WHERE body LIKE '%error%')Traces: All 25 fields included in Parquet
Pure Parquet analytics (no Tantivy needed):
-- Logs: Error rates by service
SELECT service_name, COUNT(*) FROM logs
WHERE severity >= 17 AND timestamp > '2025-01-01'
GROUP BY service_name;
-- Traces: P95 latency
SELECT service_name, PERCENTILE_CONT(0.95, span_duration_millis)
FROM traces GROUP BY service_name;
Hybrid queries (Parquet + Tantivy):
-- Full-text search with structured filters
SELECT * FROM logs
WHERE service_name = 'api' -- Parquet filter (fast)
AND timestamp > '2025-01-01' -- Parquet filter (fast)
AND body LIKE '%connection timeout%' -- Tantivy FTS (slow)
The Parquet-first optimization: Apply structured filters in Parquet first to get candidate doc IDs (filters 90-99% of data), then run Tantivy FTS only on the filtered set. This provides 10-100x speedup compared to pure Tantivy search.
Example: 1M logs → Parquet filters to 10K docs (~50ms) → Tantivy FTS on 10K docs (~100ms) instead of 1M docs (~10 seconds).
Based on experimental validation with production-scale attributes (see Appendix A):
| Current (with fast fields) | Proposed (no fast fields + Parquet VARIANT) | Total Storage | Net Cost |
|---|---|---|---|
| 100 GB | 73.2 GB + 37.1 GB | 110.3 GB | +10.3 GB (+10.3%) |
| 1 TB | 732 GB + 371 GB | 1.103 TB | +103 GB (+10.3%) |
| 10 TB | 7.32 TB + 3.71 TB | 11.03 TB | +1.03 TB (+10.3%) |
Cost example (AWS S3 Standard at $0.023/GB/month):
Value proposition: For ~10% storage cost, gain unified DataFusion analytics engine with queryable VARIANT attributes across metrics, logs, and traces.
Approach: Remove Tantivy fast fields, dual-write structured columns to Parquet, leverage DataFusion for all analytics.
Benefits:
Trade-offs:
Implementation:
FAST flag from all fields except those needed for sorting search resultsMethodology: Benchmark tool comparing storage sizes with 7,500 synthesized OTEL logs to validate architectural assumptions. Tool available at scripts/storage-benchmark/.
Complete Storage Breakdown:
| Configuration | Total Size | Doc Store | Fast Fields | Inverted + Terms | Parquet VARIANT |
|---|---|---|---|---|---|
| Current (Tantivy with fast fields) | 691.16 MB | 321 MB (46%) | 185 MB (27%) | 185 MB (27%) | — |
| Proposed (Tantivy no fast + Parquet) | 762.39 MB | 321 MB (42%) | — | 185 MB (24%) | 256 MB (34%) |
| Difference | +71.23 MB | 0 MB | -185 MB | 0 MB | +256 MB |
Key Findings:
Parquet VARIANT costs 38.5% more than fast fields (256 MB vs 185 MB)
WHERE attributes.service_id = 'xyz'Net impact: +10.3% total storage (71 MB overhead on 691 MB baseline)
Doc store unchanged (321 MB)
SELECT *Inverted indices unchanged (185 MB)
Map type vs JSON strings: Map type is properly queryable VARIANT
WHERE attributes.key = value without JSON parsingWith 20 attributes per log (minimal production logs before host tag resolution):
Interpretation: Parquet's fixed overhead (bloom filters, metadata, Map structure) doesn't scale down well for small datasets. Production workloads with host tag resolution will match the 200+ attribute case (+10.3%), not the minimal case.
The benchmark tool (scripts/storage-benchmark/) accepts any OTEL JSON log file:
cargo run --release -- --input /path/to/logs.json --output ./results
cargo run --release -- --input /path/to/logs.json --max-attributes 20 # Simulate minimal logs
cargo run --release -- --input /path/to/logs.json --no-fast-fields # Test without fast fields
Run on actual production data to validate assumptions before deployment.
Last updated: 2026-01-29