packages/shared-skills/skills/data-scientist/references/performance-benchmarks.md
Routing heuristics distilled from public 2024-2025 results in the db-benchmark suite (rendered at h2oai.github.io/db-benchmark) and from the vendors' own documentation (duckdb.org, pola.rs).
Read these as routing guidance, not guarantees. Multipliers vary with dataset size, cardinality, data types, and hardware. When the choice materially matters, measure on the actual data.
| Operation Type | Winner | Indicative advantage | When to Use |
|---|---|---|---|
| Filtering | Polars | often the fastest by a wide margin (SIMD, predicate pushdown) | Single-table filters |
| Sorting | Polars | typically the fastest | ORDER BY operations, ranking |
| Joins | DuckDB | typically faster; richer join types | Multi-table joins, especially complex joins |
| Aggregations | DuckDB | typically faster on large datasets | GROUP BY, complex aggregations |
| Window Funcs | Polars | typically faster | RANK, LAG/LEAD, running totals |
| Transformations | Polars | typically faster | Pivot, melt, string operations |
| Direct Query | DuckDB | avoids loading to memory entirely | Ad-hoc exploration without loading to memory |
| Streaming | Polars | handles datasets larger than RAM | Datasets larger than RAM |
Both tools are decisively faster than pandas on 1M+ row workloads, which is why pandas is banned outright in this skill.
Automatically detect operation types from user requests: