MatMulNBits Small-M GEMV Profiling Experiments

This file records CUDA MatMulNBits 4-bit (M = 2..16) and 8-bit (M = 2..5) profiling results for the small-M range (e.g. multi-row decode or short prefill) so future kernel and dispatch changes can be compared against a stable baseline.

Note: These are point-in-time measurements captured on the specific GPU, driver, CUDA toolkit, and ORT build noted in the section header. Treat the numbers as a historical baseline for regression comparison, not as current performance guidance — re-run the benchmark script on your own hardware before drawing conclusions.

2026-06-30 Baseline: A100 (SM80), Warmup 25, Repeat 200

Setup

• Machine/GPU: A100 (SM80).
• CUDA toolkit: 13.0. ONNX Runtime build: build/Release.
• Benchmark script: onnxruntime/test/python/transformers/profile_matmul_nbits.py.
• Warmup: 25 ORT runs before the measured benchmark NVTX range. Repeat: best of 10 trials x 200 runs.
• Weights: 4-bit, block_size 32, asymmetric (with zero points), per the standard [N, blocks, blob] layout.
• Matrices sized after a Qwen3-8B-class dense decoder + lm_head.

Command template:

# Host-timing table across all matrices:
python onnxruntime/test/python/transformers/profile_matmul_nbits.py --warmup 25 --repeat 200

# Single case:
python onnxruntime/test/python/transformers/profile_matmul_nbits.py --k 4096 --n 12288 --m 8

# Kernel-level via nsys + the repo parser:
nsys profile -t cuda,nvtx -o mnb --export=sqlite \
    python onnxruntime/test/python/transformers/profile_matmul_nbits.py --k 4096 --n 12288 --m 8
python onnxruntime/test/python/transformers/parse_nsys.py mnb.sqlite --nvtx-range benchmark --pattern '%'

Before / After (4-bit)

before is the prior dispatch (M=1 single-row GEMV; M>1 falls back to weight dequantization + cuBLAS GEMM, which is M-independent and dequantizes the full weight regardless of M). after is the batched small-M GEMV (MatMulFloat4BatchedKernel) covering M=2..16. Values are average op latency in microseconds (lower is better).

Before (dequant + cuBLAS for M>1):

After (batched small-M GEMV for M=2..16):

Speedup (before / after, >1 means the batched GEMV is faster):

Before / After (8-bit)

The 8-bit batched GEMV (MatMulFloat8bKernelBatched) covers M=2..5. 8-bit weights are twice the bytes of 4-bit and the GEMV runs on CUDA cores, so it crosses over to the dequantize + cuBLAS (tensor-core) fallback at a lower M than the 4-bit path; M>=6 keeps the fallback. Values are average op latency in microseconds.

Before (dequant + cuBLAS for M>1):

After (batched small-M GEMV for M=2..5):

Speedup (before / after, >1 means the batched GEMV is faster):

Observations

• The previous M>1 path dequantizes the entire weight matrix to a temporary buffer and calls cuBLAS, so its latency is flat across M (e.g. gate_up ~172 us, lm_head ~1.9 ms even at M=2). The batched small-M GEMV reads the quantized weight once and scales with M, giving 2.6-5.7x at M=2 (4-bit) / 1.8-3.1x at M=2 (8-bit).
• 4-bit stays at or above parity through M=16; 8-bit wins through M=5 and falls back to dequant + cuBLAS for M>=6, where the tensor-core GEMM beats the CUDA-core GEMV on the heavier 8-bit weights.
• M=1 decode is unchanged (same single-row GEMV in both builds).
• No prepacking is used, so there is no extra resident weight memory and no GEMM tactic profiling at session init.

Next Experiments

• Sweep block_size {16, 32, 64, 128} and bf16 activations.
• Add kernel-level (nsys) breakdowns to separate compute from launch/dispatch overhead.