Benchmark Report — Rust SDK vs Python SDK

Results captured from .work/. Nothing was rerun to produce this report.

See README.md if you don't know what the pipeline or the four phases are — this file just reports numbers.

The one-line answer

At 10,000+ input files:

• Rust SDK cold run is 30–80× faster than the Python SDK, depending on profile.
• Rust SDK warm run is 8–10× faster.
• Incremental edits stay small in both — the incremental contract isn't the thing that separates them; raw per-call overhead is.

Speedup at a glance — codebase / cold phase

xychart-beta
    title "Cold speedup (py_ms / rust_ms) — codebase scenario"
    x-axis ["tiny", "medium", "large", "xlarge"]
    y-axis "speedup (x)" 0 --> 90
    bar [2.22, 10.54, 16.89, 32.36]
    bar [5.75, 30.04, 52.71, 81.61]
    bar [3.13, 11.70, 13.16, 17.58]

Bars are mixed, cpu, io in that order. The docs scenario has the same shape — bigger scale, bigger gap.

Peak wins (xlarge, cold phase):

Did the incremental contract hold?

Yes, at every scale:

• warm phases had cache_misses = 0 (every section hit the memo cache).
• shape phases invalidated only 9–15 sections and rewrote only 4–11 output files, at every scale from tiny to xlarge. Adding/renaming one file doesn't cascade into a near-full rebuild.

Where the numbers come from (the matrix)

Input dataset sizes

Raw file counts generated by common.py. mixed and cpu share the same counts; io doubles the fanout.

Full per-phase tables

Each cell is rust_ms / py_ms / ratio. Medians across trials. Bigger ratio is a bigger Rust win.

Tiny — noisy, treat as smoke test

At tiny, fixed overhead (process start, LMDB open) dominates. The one row where Python looks faster is shape / mixed — expected at this scale.

Medium — first scale where the gap is consistent

Large — full matrix separates cleanly

XLarge — the headline run (10k+ files)

Shape-phase trend

Shape is the mutation-heaviest phase. Tracking the ratio across scales shows the Rust advantage compounds as the dataset grows:

Codebase

Docs

How to read these numbers

• Cold is the raw build-from-scratch cost. Big ratios here = Rust handles the fanout and per-section work more efficiently.
• Warm is the cached-everything rerun. Python still pays more per cache-hit than Rust does, so the gap is still there — just narrower than cold.
• Edit / Shape prove the incremental contract: cache misses and rebuilt outputs stay small and bounded. A regression here would show up as a near-cold-sized wall time.

When to use which scale

Footnote: artifact caveat

runner.py overwrites rust_metrics.json and python_metrics.json on every phase. That means the on-disk JSON for any given trial only keeps the final (shape) phase. The per-phase tables above come from what was captured in-memory during the runs that produced this report; they are reproducible, just not all present as files.

If you need per-phase persistence, pipe the runner output with --format json into your own store instead.