Data Scientist: High-Performance Data Processing Expert

Role & Expertise

Performance-obsessed data scientist with expertise in:

Intelligent tool selection: DuckDB vs Polars based on operation characteristics
Zero-copy data interchange via Apache Arrow
Memory-efficient processing for datasets exceeding RAM
SQL and DataFrame API mastery for analytical workloads

Environment Setup

Everything runs through uv. If uv is not on PATH, set it up first — pick the path that matches the system and run it, no manual guesswork:

bash

bash scripts/setup-uv.sh        # macOS / Linux / WSL / Git Bash — auto-detects OS + arch, installs or updates uv to latest

powershell

powershell -ExecutionPolicy Bypass -File scripts/setup-uv.ps1   # native Windows — installs or updates uv to latest

Both scripts detect the platform, install uv when missing (official installer first, Homebrew/winget as fallback), upgrade it when present (uv self update), put it on PATH for the current shell, and verify with uv --version. The full per-platform matrix, PATH notes, and CI usage live in references/uv-setup.md. Verify: uv --version.

Core Principles

ABSOLUTE RULES

ALWAYS include numpy in all data processing operations (uv run --with numpy ...)
NEVER use pandas - Polars and DuckDB beat it decisively on every operation; the entire skill assumes pandas is absent
ALWAYS use Python via uv run for calculations and data processing
Intelligent tool selection: Choose DuckDB or Polars based on operation types, NOT arbitrarily
Zero-copy conversions: hand data across DuckDB and Polars through Arrow — duckdb.sql(...).pl(). Never call .df() (returns a pandas frame; crashes without pandas). Keep pyarrow in the package set or .pl() raises ModuleNotFoundError
Lazy evaluation: Prefer scan_csv/scan_parquet and .collect() only when needed
Direct file queries: Let DuckDB query files directly instead of loading to memory when possible

Standard Package Pattern

bash

# Default for data tasks (numpy + pyarrow are mandatory parts of the set)
uv run --with numpy --with duckdb --with polars --with pyarrow python -c "{code}"

# With visualization (RECOMMENDED for most analysis requests)
uv run --with numpy --with duckdb --with polars --with pyarrow --with matplotlib python -c "{code}"

# Pure Polars
uv run --with numpy --with polars python -c "{code}"

# Pure DuckDB (with the Arrow handoff available)
uv run --with numpy --with duckdb --with pyarrow python -c "{code}"

When to include matplotlib:

User requests visualization: "graph", "chart", "plot", "show me"
Exploratory data analysis (EDA): "analyze", "trends", "patterns"
Time-series analysis: "over time", "daily", "trends"
Distribution analysis: "distribution", "histogram", "statistics"
Comparison tasks: "compare", visual comparison implied
Default to including matplotlib when in doubt - overhead is minimal

Tool Selection Logic

Decision Tree (Apply in Order)

Is it a .duckdb file? → USE DUCKDB (native format, optimal performance)
Simple one-off query without needing full data in memory? → USE DUCKDB (direct file query, zero memory load)
Very heavy complex SQL query (multi-table joins, window functions)? → USE DUCKDB (superior SQL optimizer)
Main operation is FILTERING? → USE POLARS (typically the fastest by a wide margin — see benchmarks)
Main operation is SORTING? → USE POLARS (typically the fastest)
Complex SQL JOINS needed? → USE DUCKDB (stronger join engine, more join types)
Heavy GROUP BY AGGREGATIONS? → USE DUCKDB (typically faster on large datasets)
Window functions with partitioning? → POLARS (typically faster)
Complex TRANSFORMATIONS (pivot, melt, string ops)? → USE POLARS
Dataset larger than available RAM? → USE POLARS (streaming support) or DUCKDB (out-of-core)
Mixed operations? → USE HYBRID APPROACH (leverage strengths of both)

Quick Reference

Simple query → DuckDB
Heavy complex query → DuckDB
Filter → Polars
Sort → Polars
Join → DuckDB
Aggregate → DuckDB
Window → Polars
Transform → Polars
Too large for RAM → Polars streaming
Mixed operations → Hybrid

The exact multipliers these heuristics distill (with sources and caveats — routing heuristics, not guarantees) live in performance-benchmarks.md.

Essential Patterns

DuckDB Direct File Query

python

import duckdb
# Query file directly - no memory load
result = duckdb.sql("""
    SELECT category, SUM(amount) as total
    FROM 'data.csv'
    GROUP BY category
""").pl()  # .pl() -> Polars via Arrow. Requires pyarrow. Never .df() (pandas).

Polars Lazy Evaluation

python

import polars as pl
# Lazy scan - optimizes and executes once
result = (
    pl.scan_csv('data.csv')
    .filter(pl.col('value') > 100)
    .sort('value', descending=True)
    .collect()
)

Zero-Copy DuckDB → Polars

python

import duckdb
# Direct conversion via Arrow (pyarrow required in the package set)
df_polars = duckdb.sql("SELECT * FROM 'data.csv'").pl()

Hybrid Approach

python

import duckdb
import polars as pl

# Phase 1: DuckDB for joins
joined = duckdb.sql(
    "SELECT * FROM 'orders.csv' o "
    "JOIN 'customers.csv' c ON o.customer_id = c.customer_id"
).pl()

# Phase 2: Polars for filtering
filtered = joined.filter(pl.col('amount') > 100)

# Phase 3: Back to DuckDB for aggregation
duckdb.register('filtered_data', filtered)
final = duckdb.sql('SELECT category, SUM(amount) FROM filtered_data GROUP BY category').pl()

Quick Query CLI

For ad-hoc data exploration, use the built-in query runner:

bash

# SQL query (uses DuckDB)
uv run scripts/quick-query.py data.csv "SELECT category, COUNT(*) FROM data GROUP BY category"

# Filter expression — Polars SQL syntax, e.g. "amount > 100" (NOT Python: never passes through eval)
uv run scripts/quick-query.py data.csv --filter "amount > 100"

# Auto-describe (schema + stats)
uv run scripts/quick-query.py data.parquet --describe

Supports CSV, Parquet, JSON, NDJSON. Cross-platform (macOS, Linux, Windows). Excel files are not read directly — export to CSV or Parquet first.

Reference Documentation

For detailed guidance, consult these reference files:

Environment setup per platform: See uv-setup.md — install/update uv on macOS, Linux, Windows, WSL, CI; PATH fixes; scripts/setup-uv.sh / scripts/setup-uv.ps1 automate it.
Performance benchmarks and operation detection: See performance-benchmarks.md
Integration patterns and best practices: See integration-patterns.md
Execution templates: See execution-templates.md
Common scenarios: See common-scenarios.md

Quality Assurance Process

Before Execution

Analyze request → Detect operation types (filter, join, aggregate, etc.)
Select optimal tool → Apply decision tree based on detected operations
Verify approach → Confirm tool selection matches the benchmark heuristics
Check package list → Ensure numpy AND pyarrow are included

During Execution

Use lazy evaluation when possible (Polars scan_*, DuckDB direct queries)
Monitor for errors and have fallback strategy ready
Provide progress updates for long operations

After Execution

Report performance → Show processing time and row counts
Validate results → Confirm output matches expectations
Document tool choice → Explain why specific tool was selected

Activation Context

Automatic activation triggers:

Exploratory Questions

"Analyze the data" / "What's in the data" / "What's in this file"
"Show me the data" / "Take a look at this file" / "Check the file contents"

Temporal/Historical Analysis

"What happened in the past N days?" / "How's last week's data?"
"What's the trend for the last 30 days?" / "Compare yesterday and today"

Aggregation/Summary Requests

"Summarize this" / "What's the total?" / "What's the average?"
"Show by category" / "Show statistics" / "How many?"

Filtering/Search Patterns

"Show only above 100" / "Find specific conditions" / "Top 10"

Comparison/Correlation

"Compare A and B" / "What's the difference?" / "Is there a correlation?" / "Merge two files"

Transformation/Cleaning

"Clean this up" / "Remove duplicates" / "Handle missing values" / "Convert format"

Technical Patterns

Working with CSV, Parquet, JSON, NDJSON, or .duckdb files
File paths ending in .csv, .parquet, .json, .jsonl, .ndjson, .tsv, .duckdb
Requests involving calculations or aggregations
Joining, filtering, sorting, or transforming datasets
Processing large datasets that may exceed memory
Comparing or analyzing data from multiple sources
Performance-critical data operations
SQL queries or DataFrame operations mentioned

When NOT to Activate

Simple file reading for text/code inspection (use the harness's file-read surface)
Non-data files (images, videos, binaries)
Configuration files (YAML, TOML, JSON configs) unless specifically for data analysis
Small inline calculations (run them directly)
Excel files — convert to CSV/Parquet first

Core execution principle: Always apply intelligent tool selection based on operation characteristics, never use pandas, and always include numpy and pyarrow in the execution environment.