llama.cpp/example/batched-bench

Benchmark the batched decoding performance of llama.cpp

Usage

There are 2 modes of operation:

• prompt not shared - each batch has a separate prompt of size PP (i.e. N_KV = B*(PP + TG))
• prompt is shared - there is a common prompt of size PP used by all batches (i.e. N_KV = PP + B*TG)

./batched-bench MODEL_PATH [N_KV_MAX] [IS_PP_SHARED] [NGL] [MMQ] <PP> <TG> <PL>

# LLaMA 7B, F16, N_KV_MAX = 16384 (8GB), prompt not shared
./batched-bench ./models/llama-7b/ggml-model-f16.gguf 16384 0 99

# LLaMA 7B, Q8_0, N_KV_MAX = 16384 (8GB), prompt is shared
./batched-bench ./models/llama-7b/ggml-model-q8_0.gguf 16384 1 99

# custom set of batches
./batched-bench ./models/llama-7b/ggml-model-q8_0.gguf 2048 0 999 0 128,256,512 128,256 1,2,4,8,16,32

Sample results

• PP - prompt tokens per batch
• TG - generated tokens per batch
• B - number of batches
• N_KV - required KV cache size
• T_PP - prompt processing time (i.e. time to first token)
• S_PP - prompt processing speed ((B*PP)/T_PP or PP/T_PP)
• T_TG - time to generate all batches
• S_TG - text generation speed ((B*TG)/T_TG)
• T - total time
• S - total speed (i.e. all tokens / total time)