docs/content/features/vllm-cpp.md
+++ disableToc = false title = "vllm.cpp backend" weight = 16 url = "/features/vllm-cpp/" +++
vllm.cpp is the LocalAI team's C++20 port of
vLLM: the same continuous-batching scheduler, paged KV cache and automatic prefix
caching, with no Python at inference time. LocalAI serves it through the native
vllm-cpp backend, which loads either a HuggingFace safetensors directory or a
.gguf file and applies the chat template, tool-call parsing and reasoning split
inside the engine.
This page covers installing the backend and the models LocalAI ships ready to run
on it. For the full engine_args reference (KV sizing, scheduling policy,
speculative decoding, LMCache), see the
[vllm.cpp section of the text generation guide]({{% relref "features/text-generation" %}}#vllmcpp).
local-ai backends install vllm-cpp
Or install it from the Backends page in the web UI. Images are published for CPU, CUDA 13, Vulkan, Metal and Jetson L4T.
The CUDA images are currently built for Blackwell-family architectures only:
sm_120a (RTX 50 series, RTX PRO 6000 Blackwell) and sm_121a (GB10 / DGX
Spark) on x86-64, and sm_121a alone on arm64. CUDA 13 is required, so there is
no CUDA 12 variant.
That is narrower than vllm.cpp itself, which builds ten architectures. On an
Ampere, Ada, Hopper, Jetson Orin or Jetson Thor GPU the CUDA backend installs
successfully and then fails at the first request with no kernel image is available for execution on the device. Until the build widens, use the Vulkan
or CPU image on those cards: vulkan-vllm-cpp builds with CUDA off entirely and
gives them a GPU path, without the NVFP4 and Marlin kernels.
The model gallery carries a curated set of vllm.cpp configurations. Each one arrives with the engine settings already applied, so tool calling, the reasoning split and speculative decoding work without hand-editing YAML.
| Gallery entry | Model | Size | Needs |
|---|---|---|---|
qwen3.6-27b-nvfp4-vllm-cpp | Qwen3.6-27B, NVFP4 | 25 GB | Blackwell GPU |
qwen3.6-27b-nvfp4-mtp-vllm-cpp | the same, with MTP speculative decoding | 25 GB | Blackwell GPU |
qwen3.6-27b-nvfp4-dflash-vllm-cpp | the same, with DFlash speculative decoding | 28 GB | Blackwell GPU |
qwen3.6-35b-a3b-nvfp4-vllm-cpp | Qwen3.6-35B-A3B MoE, NVFP4 | 23 GB | Blackwell GPU |
qwen3.6-35b-a3b-nvfp4-mtp-vllm-cpp | the same, with MTP speculative decoding | 23 GB | Blackwell GPU |
qwen3-coder-30b-a3b-vllm-cpp | Qwen3-Coder-30B-A3B, bf16 | 57 GB | Blackwell GPU, or CPU |
qwen3-4b-vllm-cpp | Qwen3-4B, bf16 | 8 GB | CPU, Metal, Vulkan, Blackwell GPU |
qwen3-0.6b-vllm-cpp | Qwen3-0.6B, bf16 | 1.4 GB | CPU, Metal, Vulkan, Blackwell GPU |
local-ai models install qwen3-0.6b-vllm-cpp
The two small bf16 entries are the ones that run anywhere the backend does, including CPU. The NVFP4 entries need a Blackwell-class NVIDIA GPU on two counts: NVFP4 has no kernel on older architectures, and the CUDA images are built only for Blackwell in any case.
Sizing note: every entry sets num_blocks to give roughly one to four full
contexts of KV cache, which is a starting point rather than a tuned value. KV is
not free; the 4B entry, for instance, spends 144 KiB per token, so its 1024
blocks are about 4.5 GB on top of the weights. Raise num_blocks for more
concurrency, lower it on a small box.
The Qwen3.6-27B entries pin their weights to a specific HuggingFace commit rather than tracking the repository's default branch. This is deliberate and worth understanding before you copy one of these configs.
The upstream repository was later re-quantized in place, under the same name, from NVFP4 to FP8 W8A8. A config that names the repository without a revision therefore resolves to entirely different weights, with different numerics and different performance, and nothing about the load reports that anything changed. Pinning is what makes the entry reproducible:
artifacts:
- name: model
target: model
source:
type: huggingface
repo: unsloth/Qwen3.6-27B-NVFP4
revision: 890bdef7a42feba6d83b6e17a03315c694112f2a
The same reasoning applies to any quantized community repository you depend on.
Speculative decoding trades memory for decode throughput. All three Qwen3.6-27B entries serve the same weights and produce the same quality; they differ only in how tokens are proposed.
| Entry | Method | Extra weights | Extra memory |
|---|---|---|---|
qwen3.6-27b-nvfp4-vllm-cpp | none | none | none |
qwen3.6-27b-nvfp4-mtp-vllm-cpp | MTP, depth 1 | none, the draft head ships inside the checkpoint | about 3.6 GB |
qwen3.6-27b-nvfp4-dflash-vllm-cpp | DFlash, 16-token blocks | a separate 3.5 GB drafter | drafter plus draft cache |
MTP drafts one token per step from a head that already lives in the target
checkpoint's own mtp.* tensors, so it costs no extra download. DFlash drafts a
whole 16-token block in one non-autoregressive pass from a separate drafter, which
is the larger win at the cost of a second checkpoint on disk.
Start with the plain entry if you are short on memory, and with the DFlash entry if you are not.
Every entry above sets use_tokenizer_template: true and disables LocalAI's
Go-side grammar path, so tool calls are detected and parsed by the engine's own
streaming parsers and arrive as real tool_calls on the OpenAI response.
The parser is normally auto-detected from the chat template, but one case cannot
be: Qwen3-Coder's tool dialect is byte-identical on the wire to another family's,
so template sniffing would pick the wrong parser. The
qwen3-coder-30b-a3b-vllm-cpp entry therefore names it explicitly, and any
Qwen3-Coder config you write yourself should do the same:
engine_args:
tool_parser: qwen3_coder
The vllm-cpp backend also serves MiniMax-H3, which generates video and audio
jointly. See [Video generation]({{% relref "features/video-generation" %}}#minimax-h3-vllmcpp).