mistral.rs v0.9.0 CPU Benchmark Report

CPU-only comparison of mistral.rs against llama.cpp on two architectures: GB10 (aarch64, 10x Cortex-X925) and a c7i.8xlarge Xeon (Sapphire Rapids, AVX512/VNNI/AMX). Values are tokens per second; speedups are mistral.rs divided by llama.cpp at the same prompt length or decode depth, with each engine at its best measured configuration.

Decode is at or ahead of llama.cpp at every measured depth on both architectures, and the lead grows with context: 1.81x (x86) and 1.79x (ARM) at 16K depth. The mechanism, in one line: decode attention streams the KV cache at memory bandwidth with the output accumulators held in registers, so per-token cost approaches the hardware's memory floor while llama.cpp's kernel carries per-position overhead that compounds with depth. A detailed writeup of the kernel design is forthcoming.

Headline results

GB10 (aarch64), mean speedup across context lengths 128-8192/16384:

x86 (Sapphire Rapids), qwen3-4b, per-point ratios:

Known gaps, stated plainly: x86 prefill trails llama.cpp's mature AMX path (a feature most of the x86 fleet lacks; a non-AMX comparison point is planned); MoE prefill at 8192 is 0.87x; q8_0 shallow x86 decode is 0.84x. Causes are understood and fixes are scoped for the next release.

Method

• Workloads: prompt lengths and decode depths of 128, 512, 2048, 4096, 8192 (and 16384 for qwen3-4b Q4_K); 256 generated tokens per decode depth; 1 warmup, 2-3 measured iterations.
• CPU-only builds: mistral.rs without GPU features, run with --cpu; llama.cpp with GGML_CUDA=OFF GGML_NATIVE=ON (Release), pinned commit below.
• Quantization: mistral.rs ISQ q4k/q6k/q8_0 (benchmarked from prequantized UQFF, numerically identical to --isq) versus llama.cpp GGUF Q4_K_M/Q6_K/Q8_0. Q8_0 is the same scheme on both engines; ISQ q4k is uniform while GGUF Q4_K_M mixes Q4_K/Q6_K per tensor, so the 4-bit tiers are close but not bit-identical.
• Affinity (GB10 big.LITTLE): both engines pinned to the 10 big cores for decode, each at its best measured configuration - mistral.rs CANDLE_CPU_MASK=5-9,15-19; llama.cpp decode taskset -c 5-9,15-19 -t 10, prefill at its faster stock -t 20. An affinity study (raw/results_affinity.jsonl) verified pinning mechanism does not matter (engine mask vs OS taskset within noise) and that unpinned llama.cpp decode loses >2x to little-core stragglers, while unpinned mistral.rs does not (its default thread sizing avoids the little cores).
• llama.cpp fairness: on x86 its flash-attention CPU kernel inverts at depth (fa=1 wins through ~2k, fa=0 wins beyond: 8.8 vs 6.3 t/s at d16384), so every reported llama.cpp number is its best configuration at that point, mixing fa settings as needed. Raw data includes both.
• The box was otherwise idle; contended runs were discarded and rerun.

Reproduction

# full sweep at best-per-engine affinity (GB10)
python3 releases/v0.9.0/scripts/bench_cpu_sweep.py --phase full \
  --mrs-mode mask --lcpp-mode default --lcpp-decode-mode taskset \
  --iters 2 --warmup 1 --gen-len 256 --lengths 128,512,2048,4096,8192

Engine command shapes:

# mistral.rs (ISQ from BF16 safetensors; or --from-uqff a file made by `mistralrs quantize`)
CANDLE_CPU_MASK=5-9,15-19 target/release/mistralrs bench --cpu \
  --prompt-len 128,512,2048,4096,8192 --depth 128,512,2048,4096,8192 \
  --gen-len 256 --iterations 2 --warmup 1 -m Qwen/Qwen3-4B --isq q4k

# llama.cpp prefill / decode
llama-bench -m Qwen3-4B-Q4_K_M.gguf -p 128,...,8192 -n 0 -r 2 -o json -t 20
taskset -c 5-9,15-19 llama-bench -m Qwen3-4B-Q4_K_M.gguf \
  -p 0 -n 256 -d 128,...,8192 -r 2 -o json -t 10

• scripts/bench_cpu_sweep.py - sweep orchestrator; one JSON row per measurement appended to raw/results_full.jsonl (later rows supersede earlier ones for the same point), raw engine stdout under raw/raw_full/.
• raw/results_x86.jsonl + raw/x86_sweep.log - the x86 sweep (includes both fa configs).
• scripts/capture_metadata.sh - host/commit/model metadata (raw/metadata.txt).

Model artifacts

Versions and hosts

Hosts: GB10 (Linux 6.17, 10x Cortex-X925 3.9 GHz + 10x Cortex-A725, 1 NUMA node; full details in raw/metadata.txt); AWS c7i.8xlarge (Xeon Platinum 8488C, 16 physical cores, avx512f/avx512_vnni/amx_int8).

Build notes

All benchmarks are source builds of both engines on the same machine (llama.cpp with GGML_NATIVE=ON, mistral.rs with target-cpu=native), which is what the reproducer scripts do. Prebuilt installer binaries are portable (runtime-dispatched kernels) and land within ~8% of source-built throughput on both architectures; build from source to reproduce the tables exactly. Two aarch64 assets ship: the default assumes ARMv8.2 (Graviton2+, Pi 5, 2018+ ARM) and a v8.0 compat build covers A72-class boards (Pi 4); the installer picks by cpuinfo probe.

Appendix: Full Tables

All values are tokens per second; speedup is mistral.rs divided by llama.cpp in the same row.

qwen3-4b (GB10)

Q4_K Prefill

Q4_K Decode

Q6_K Prefill

Q6_K Decode

Q8_0 Prefill

Q8_0 Decode

qwen3-4b (x86, Sapphire Rapids)

llama.cpp values are its best fa configuration per point (fa=1 shallow, fa=0 deep).

Q4_K Decode

Q8_0 Decode

Full x86 rows (including prefill and both fa configs): raw/results_x86.jsonl.

gemma4-e4b (GB10)

Q4_K Prefill

Q4_K Decode

Q6_K Prefill

Q6_K Decode

Q8_0 Prefill

Q8_0 Decode

lfm2.5-230m (GB10)

Q4_K Prefill

Q4_K Decode

Q6_K Prefill

Q6_K Decode

Q8_0 Prefill

Q8_0 Decode

lfm2.5-8b-a1b (MoE, GB10)

Q4_K Prefill

Q4_K Decode

Appendix: ik_llama.cpp

ik_llama.cpp is the CPU-performance-focused fork by the author of llama.cpp's K-quants, and the fastest CPU implementation in the llama.cpp family. Same box (c7i.8xlarge), same GGUF, commit bbc7de4, pinned taskset -c 0-15 -t 16, its best fa configuration per point (fa=1 won every point for it). qwen3-4b Q4_K_M, tokens per second:

Decode diverges in mistral.rs's favor with depth (+30% at 16K) against a flash-attention implementation substantially better than mainline's (12.2 vs mainline's best 8.8 t/s at d16384). ik's prefill GEMM kernels lead the family and nearly match mainline's AMX path without using AMX. Not benchmarked on aarch64: ik HEAD does not compile there (gcc 13, Ubuntu 24.04).