Attention Backend Feature Support

This document is auto-generated by tools/pre_commit/generate_attention_backend_docs.py. It shows the feature support for each registered attention backend based on the checks in AttentionBackend.validate_configuration().

Do not edit this file manually. Run the following command to regenerate it:

python tools/pre_commit/generate_attention_backend_docs.py

Setting the Attention Backend

Command Line

There are two ways to specify the backend from the command line:

Option 1: Using --attention-backend (simple)

vllm serve <model> --attention-backend FLASH_ATTN

Option 2: Using --attention-config.backend / -ac.backend (structured config)

# Dot notation
vllm serve <model> --attention-config.backend FLASH_ATTN
vllm serve <model> -ac.backend FLASH_ATTN

# JSON format
vllm serve <model> --attention-config '{"backend": "FLASH_ATTN"}'
vllm serve <model> -ac '{"backend": "FLASH_ATTN"}'

Note: --attention-backend and --attention-config.backend are mutually exclusive. Use one or the other, not both.

Python API

Use AttentionConfig with the LLM class:

from vllm import LLM
from vllm.config import AttentionConfig
from vllm.v1.attention.backends.registry import AttentionBackendEnum

# Method 1: Using AttentionConfig with enum
llm = LLM(
    model="Qwen/Qwen3-0.6B",
    attention_config=AttentionConfig(backend=AttentionBackendEnum.FLASH_ATTN),
)

# Method 2: Using attention_backend parameter with string
llm = LLM(
    model="Qwen/Qwen3-0.6B",
    attention_backend="FLASH_ATTN",
)

Backend Selection Behavior

Manual Selection

When you explicitly set a backend via --attention-backend or AttentionConfig:

1. The backend is validated against your configuration (model dtype, head size, compute capability, etc.)
2. If the backend doesn't support your configuration, an error is raised with the specific reason
3. If valid, the backend is used

Example error when selecting an incompatible backend:

ValueError: Selected backend FLASHMLA is not valid for this configuration.
Reason: ['compute capability not supported']

Automatic Selection

When no backend is specified (the default):

1. vLLM iterates through backends in priority order (see tables below)
2. Each backend is validated against your configuration
3. The first compatible backend is selected
4. If no backend is compatible, an error is raised listing all backends and their incompatibility reasons

Backend Priority (CUDA)

When no backend is explicitly selected, vLLM chooses the first compatible backend from these priority-ordered lists.

Priority is 1 = highest (tried first).

Standard Attention (MHA, MQA, GQA)

Blackwell (SM 10.x):

Ampere/Hopper (SM 8.x-9.x):

MLA Attention (DeepSeek-style)

Blackwell (SM 10.x):

Ampere/Hopper (SM 8.x-9.x):

* For sparse MLA, FP8 KV cache always prefers FLASHINFER_MLA_SPARSE. With BF16 KV cache, FLASHINFER_MLA_SPARSE is preferred for low query-head counts (<= 16), while FLASHMLA_SPARSE is preferred otherwise.

Note: ROCm and CPU platforms have their own selection logic. See the platform-specific documentation for details.

Legend

Symbols: ✅ = Supported, ❌ = Not supported

Standard Attention (MHA, MQA, GQA) Backends

† FlashInfer uses TRTLLM attention on Blackwell (SM100), which supports sinks. Disable via --attention-config.use_trtllm_attention=0.

* Specify the FlashAttention version via --attention-config.flash_attn_version=2, 3, or 4. Default is FA4 on SM100+ (Blackwell), FA3 on SM90 (Hopper), FA2 otherwise.

MLA (Multi-head Latent Attention) Backends

MLA uses separate backends for prefill and decode phases.

Prefill Backends

The prefill backend is selected at runtime based on hardware and configuration.

‡ TRT-LLM Ragged is the default on Blackwell (SM100). On other GPUs, FlashAttention is used as the default.

Decode Backends