GGUF Quantization Guide

Complete guide to GGUF quantization formats and model conversion.

Hub-first quant selection

Before using generic tables, open the model repo with:

https://huggingface.co/<repo>?local-app=llama.cpp

Prefer the exact quant labels and sizes shown in the Hardware compatibility section of the fetched ?local-app=llama.cpp page text or HTML. Then confirm the matching filenames in:

https://huggingface.co/api/models/<repo>/tree/main?recursive=true

Use the Hub page first, and only fall back to the generic heuristics below when the repo page does not expose a clear recommendation.

Quantization Overview

GGUF (GPT-Generated Unified Format) - Standard format for llama.cpp models.

Format Comparison

Recommendation: Use Q4_K_M for best balance of quality and speed.

Converting Models

Hugging Face to GGUF

# 1. Download Hugging Face model
hf download meta-llama/Llama-2-7b-chat-hf \
    --local-dir models/llama-2-7b-chat/

# 2. Convert to FP16 GGUF
python convert_hf_to_gguf.py \
    models/llama-2-7b-chat/ \
    --outtype f16 \
    --outfile models/llama-2-7b-chat-f16.gguf

# 3. Quantize to Q4_K_M
./llama-quantize \
    models/llama-2-7b-chat-f16.gguf \
    models/llama-2-7b-chat-Q4_K_M.gguf \
    Q4_K_M

Batch quantization

# Quantize to multiple formats
for quant in Q4_K_M Q5_K_M Q6_K Q8_0; do
    ./llama-quantize \
        model-f16.gguf \
        model-${quant}.gguf \
        $quant
done

K-Quantization Methods

K-quants use mixed precision for better quality:

• Attention weights: Higher precision
• Feed-forward weights: Lower precision

Variants:

• _S (Small): Faster, lower quality
• _M (Medium): Balanced (recommended)
• _L (Large): Better quality, larger size

Example: Q4_K_M

• Q4: 4-bit quantization
• K: Mixed precision method
• M: Medium quality

Quality Testing

# Calculate perplexity (quality metric)
./llama-perplexity \
    -m model.gguf \
    -f wikitext-2-raw/wiki.test.raw \
    -c 512

# Lower perplexity = better quality
# Baseline (FP16): ~5.96
# Q4_K_M: ~6.06 (+1.7%)
# Q2_K: ~6.87 (+15.3% - too much degradation)

Use Case Guide

General purpose (chatbots, assistants)

Q4_K_M - Best balance
Q5_K_M - If you have extra RAM

Code generation

Q5_K_M or Q6_K - Higher precision helps with code

Creative writing

Q4_K_M - Sufficient quality
Q3_K_M - Acceptable for draft generation

Technical/medical

Q6_K or Q8_0 - Maximum accuracy

Edge devices (Raspberry Pi)

Q2_K or Q3_K_S - Fit in limited RAM

Model Size Scaling

7B parameter models

13B parameter models

70B parameter models

Recommendation for 70B: Use Q3_K_M or Q4_K_S to fit in consumer hardware.

Finding Pre-Quantized Models

Use the Hub search with the llama.cpp app filter:

https://huggingface.co/models?apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending

For a specific repo, open:

https://huggingface.co/<repo>?local-app=llama.cpp
https://huggingface.co/api/models/<repo>/tree/main?recursive=true

Then launch directly from the Hub without extra Hub tooling:

llama-cli -hf <repo>:Q4_K_M
llama-server -hf <repo>:Q4_K_M

If you need the exact file name from the tree API:

llama-server --hf-repo <repo> --hf-file <filename.gguf>

Importance Matrices (imatrix)

What: Calibration data to improve quantization quality.

Benefits:

• 10-20% perplexity improvement with Q4
• Essential for Q3 and below

Usage:

# 1. Generate importance matrix
./llama-imatrix \
    -m model-f16.gguf \
    -f calibration-data.txt \
    -o model.imatrix

# 2. Quantize with imatrix
./llama-quantize \
    --imatrix model.imatrix \
    model-f16.gguf \
    model-Q4_K_M.gguf \
    Q4_K_M

Calibration data:

• Use domain-specific text (e.g., code for code models)
• ~100MB of representative text
• Higher quality data = better quantization

Troubleshooting

Model outputs gibberish:

• Quantization too aggressive (Q2_K)
• Try Q4_K_M or Q5_K_M
• Verify model converted correctly

Out of memory:

• Use lower quantization (Q4_K_S instead of Q5_K_M)
• Offload fewer layers to GPU (-ngl)
• Use smaller context (-c 2048)

Slow inference:

• Higher quantization uses more compute
• Q8_0 much slower than Q4_K_M
• Consider speed vs quality trade-off