Supported Models

vLLM supports generative and pooling models across various tasks.

For each task, we list the model architectures that have been implemented in vLLM. Alongside each architecture, we include some popular models that use it.

Model Implementation

vLLM

If vLLM natively supports a model, its implementation can be found in vllm/model_executor/models.

These models are what we list in supported text models and supported multimodal models.

Transformers

vLLM also supports model implementations that are available in Transformers. You should expect the performance of a Transformers model implementation used in vLLM to be within <5% of the performance of a dedicated vLLM model implementation. We call this feature the "Transformers modeling backend".

Currently, the Transformers modeling backend works for the following:

• Modalities: embedding models, language models and vision-language models*
• Architectures: encoder-only, decoder-only, mixture-of-experts
• Attention types: full attention and/or sliding attention

*Vision-language models currently accept only image inputs. Support for video inputs will be added in a future release.

If the Transformers model implementation follows all the steps in writing a custom model then, when used with the Transformers modeling backend, it will be compatible with the following features of vLLM:

• All the features listed in the compatibility matrix
• Any combination of the following vLLM parallelisation schemes:
  • Data parallel
  • Tensor parallel
  • Expert parallel
  • Pipeline parallel

Checking if the modeling backend is Transformers is as simple as:

from vllm import LLM
llm = LLM(model=...)  # Name or path of your model
llm.apply_model(lambda model: print(type(model)))

If the printed type starts with Transformers... then it's using the Transformers model implementation!

If a model has a vLLM implementation but you would prefer to use the Transformers implementation via the Transformers modeling backend, set model_impl="transformers" for offline inference or --model-impl transformers for the online serving.

!!! note For vision-language models, if you are loading with dtype="auto", vLLM loads the whole model with config's dtype if it exists. In contrast the native Transformers will respect the dtype attribute of each backbone in the model. That might cause a slight difference in performance.

Custom models

If a model is neither supported natively by vLLM nor Transformers, it can still be used in vLLM!

For a model to be compatible with the Transformers modeling backend for vLLM it must:

• be a Transformers compatible custom model (see Transformers - Customizing models):
  • The model directory must have the correct structure (e.g. config.json is present).
  • config.json must contain auto_map.AutoModel.
• be a Transformers modeling backend for vLLM compatible model (see Writing custom models):
  • Customisation should be done in the base model (e.g. in MyModel, not MyModelForCausalLM).

If the compatible model is:

• on the Hugging Face Model Hub, simply set trust_remote_code=True for offline-inference or --trust-remote-code for the openai-compatible-server.
• in a local directory, simply pass directory path to model=<MODEL_DIR> for offline-inference or vllm serve <MODEL_DIR> for the openai-compatible-server.

This means that, with the Transformers modeling backend for vLLM, new models can be used before they are officially supported in Transformers or vLLM!

Writing custom models

This section details the necessary modifications to make to a Transformers compatible custom model that make it compatible with the Transformers modeling backend for vLLM. (We assume that a Transformers compatible custom model has already been created, see Transformers - Customizing models).

To make your model compatible with the Transformers modeling backend, it needs:

1. kwargs passed down through all modules from MyModel to MyAttention.
   • If your model is encoder-only:
     1. Add is_causal = False to MyAttention.
   • If your model is mixture-of-experts (MoE):
     1. Your sparse MoE block must have an attribute called experts.
     2. The class of experts (MyExperts) must either:
        • Inherit from nn.ModuleList (naive).
        • Or contain all 3D nn.Parameters (packed).
     3. MyExperts.forward must accept hidden_states, top_k_index, top_k_weights.
2. MyAttention must use ALL_ATTENTION_FUNCTIONS to call attention.
3. MyModel must contain _supports_attention_backend = True.

from transformers import PreTrainedModel
from torch import nn

class MyAttention(nn.Module):
    is_causal = False  # Only do this for encoder-only models

    def forward(self, hidden_states, **kwargs):
        ...
        attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
        attn_output, attn_weights = attention_interface(
            self,
            query_states,
            key_states,
            value_states,
            **kwargs,
        )
        ...

# Only do this for mixture-of-experts models
class MyExperts(nn.ModuleList):
    def forward(self, hidden_states, top_k_index, top_k_weights):
        ...

# Only do this for mixture-of-experts models
class MySparseMoEBlock(nn.Module):
    def __init__(self, config):
        ...
        self.experts = MyExperts(config)
        ...

    def forward(self, hidden_states: torch.Tensor):
        ...
        hidden_states = self.experts(hidden_states, top_k_index, top_k_weights)
        ...

class MyModel(PreTrainedModel):
    _supports_attention_backend = True

Here is what happens in the background when this model is loaded:

1. The config is loaded.
2. MyModel Python class is loaded from the auto_map in config, and we check that the model is_backend_compatible().
3. MyModel is loaded into one of the Transformers modeling backend classes in vllm/model_executor/models/transformers which sets self.config._attn_implementation = "vllm" so that vLLM's attention layer is used.

That's it!

For your model to be compatible with vLLM's tensor parallel and/or pipeline parallel features, you must add base_model_tp_plan and/or base_model_pp_plan to your model's config class:

from transformers import PretrainedConfig

class MyConfig(PretrainedConfig):
    base_model_tp_plan = {
        "layers.*.self_attn.k_proj": "colwise",
        "layers.*.self_attn.v_proj": "colwise",
        "layers.*.self_attn.o_proj": "rowwise",
        "layers.*.mlp.gate_proj": "colwise",
        "layers.*.mlp.up_proj": "colwise",
        "layers.*.mlp.down_proj": "rowwise",
    }
    base_model_pp_plan = {
        "embed_tokens": (["input_ids"], ["inputs_embeds"]),
        "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
        "norm": (["hidden_states"], ["hidden_states"]),
    }

• base_model_tp_plan is a dict that maps fully qualified layer name patterns to tensor parallel styles (currently only "colwise" and "rowwise" are supported).
• base_model_pp_plan is a dict that maps direct child layer names to tuples of lists of strs:
  • You only need to do this for layers which are not present on all pipeline stages
  • vLLM assumes that there will be only one nn.ModuleList, which is distributed across the pipeline stages
  • The list in the first element of the tuple contains the names of the input arguments
  • The list in the last element of the tuple contains the names of the variables the layer outputs to in your modeling code

Plugins

Some model architectures are supported via vLLM plugins. These plugins extend vLLM's capabilities through the plugin system.

For other model architectures not natively supported, in particular for Encoder-Decoder models, we recommend following a similar pattern by implementing support through the plugin system.

Loading a Model

Hugging Face Hub

By default, vLLM loads models from Hugging Face (HF) Hub. To change the download path for models, you can set the HF_HOME environment variable; for more details, refer to their official documentation.

To determine whether a given model is natively supported, you can check the config.json file inside the HF repository. If the "architectures" field contains a model architecture listed below, then it should be natively supported.

Models do not need to be natively supported to be used in vLLM. The Transformers modeling backend enables you to run models directly using their Transformers implementation (or even remote code on the Hugging Face Model Hub!).

!!! tip The easiest way to check if your model is really supported at runtime is to run the program below:

```python
from vllm import LLM

# For generative models (runner=generate) only
llm = LLM(model=..., runner="generate")  # Name or path of your model
output = llm.generate("Hello, my name is")
print(output)

# For pooling models (runner=pooling) only
llm = LLM(model=..., runner="pooling")  # Name or path of your model
output = llm.encode("Hello, my name is")
print(output)
```

If vLLM successfully returns text (for generative models) or hidden states (for pooling models), it indicates that your model is supported.

Otherwise, please refer to Adding a New Model for instructions on how to implement your model in vLLM. Alternatively, you can open an issue on GitHub to request vLLM support.

Download a model

If you prefer, you can use the Hugging Face CLI to download a model or specific files from a model repository:

# Download a model
hf download HuggingFaceH4/zephyr-7b-beta

# Specify a custom cache directory
hf download HuggingFaceH4/zephyr-7b-beta --cache-dir ./path/to/cache

# Download a specific file from a model repo
hf download HuggingFaceH4/zephyr-7b-beta eval_results.json

List the downloaded models

Use the Hugging Face CLI to manage models stored in local cache:

# List cached models
hf scan-cache

# Show detailed (verbose) output
hf scan-cache -v

# Specify a custom cache directory
hf scan-cache --dir ~/.cache/huggingface/hub

Delete a cached model

Use the Hugging Face CLI to interactively delete downloaded model from the cache:

# The `delete-cache` command requires extra dependencies to work with the TUI.
# Please run `pip install huggingface_hub[cli]` to install them.

# Launch the interactive TUI to select models to delete
$ hf delete-cache
? Select revisions to delete: 1 revisions selected counting for 438.9M.
  ○ None of the following (if selected, nothing will be deleted).
Model BAAI/bge-base-en-v1.5 (438.9M, used 1 week ago)
❯ ◉ a5beb1e3: main # modified 1 week ago

Model BAAI/bge-large-en-v1.5 (1.3G, used 1 week ago)
  ○ d4aa6901: main # modified 1 week ago

Model BAAI/bge-reranker-base (1.1G, used 4 weeks ago)
  ○ 2cfc18c9: main # modified 4 weeks ago

Press <space> to select, <enter> to validate and <ctrl+c> to quit without modification.

# Need to confirm after selected
? Select revisions to delete: 1 revision(s) selected.
? 1 revisions selected counting for 438.9M. Confirm deletion ? Yes
Start deletion.
Done. Deleted 1 repo(s) and 0 revision(s) for a total of 438.9M.

Using a proxy

Here are some tips for loading/downloading models from Hugging Face using a proxy:

• Set the proxy globally for your session (or set it in the profile file):

export http_proxy=http://your.proxy.server:port
export https_proxy=http://your.proxy.server:port

• Set the proxy for just the current command:

https_proxy=http://your.proxy.server:port hf download <model_name>

# or use vllm cmd directly
https_proxy=http://your.proxy.server:port  vllm serve <model_name>

• Set the proxy in Python interpreter:

import os

os.environ["http_proxy"] = "http://your.proxy.server:port"
os.environ["https_proxy"] = "http://your.proxy.server:port"

ModelScope

To use models from ModelScope instead of Hugging Face Hub, set an environment variable:

export VLLM_USE_MODELSCOPE=True

And use with trust_remote_code=True.

from vllm import LLM

llm = LLM(model=..., revision=..., runner=..., trust_remote_code=True)

# For generative models (runner=generate) only
output = llm.generate("Hello, my name is")
print(output)

# For pooling models (runner=pooling) only
output = llm.encode("Hello, my name is")
print(output)

Feature Status Legend

• ✅︎ indicates that the feature is supported for the model.

• 🚧 indicates that the feature is planned but not yet supported for the model.

• ⚠️ indicates that the feature is available but may have known issues or limitations.

List of Text-only Language Models

Generative Models

See this page for more information on how to use generative models.

Text Generation

These models primarily accept the LLM.generate API. Chat/Instruct models additionally support the LLM.chat API.

!!! note Grok2 requires tokenizer.tok.json with tiktoken installed. You can optionally override MoE router renormalization with moe_router_renormalize.

Some models are supported only via the Transformers modeling backend. The purpose of the table below is to acknowledge models which we officially support in this way. The logs will say that the Transformers modeling backend is being used, and you will see no warning that this is fallback behaviour. This means that, if you have issues with any of the models listed below, please make an issue and we'll do our best to fix it!

!!! note Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.

List of Multimodal Language Models

The following modalities are supported depending on the model:

• Text
• Image
• Video
• Audio

Any combination of modalities joined by + are supported.

• e.g.: T + I means that the model supports text-only, image-only, and text-with-image inputs.

On the other hand, modalities separated by / are mutually exclusive.

• e.g.: T / I means that the model supports text-only and image-only inputs, but not text-with-image inputs.

See this page on how to pass multi-modal inputs to the model.

!!! tip For hybrid-only models such as Llama-4, Step3, Mistral-3 and Qwen-3.5, a text-only mode can be enabled by setting all supported multimodal modalities to 0 (--language-model-only) so that their multimodal modules will not be loaded to free up more GPU memory for KV cache.

!!! note vLLM currently supports adding LoRA adapters to the language backbone for most multimodal models. Additionally, vLLM now experimentally supports adding LoRA to the tower and connector modules for some multimodal models. See this page.

Generative Models

See this page for more information on how to use generative models.

Text Generation

These models primarily accept the LLM.generate API. Chat/Instruct models additionally support the LLM.chat API.

Some models are supported only via the Transformers modeling backend. The purpose of the table below is to acknowledge models which we officially support in this way. The logs will say that the Transformers modeling backend is being used, and you will see no warning that this is fallback behaviour. This means that, if you have issues with any of the models listed below, please make an issue and we'll do our best to fix it!

^{^} You need to set the architecture name via --hf-overrides to match the one in vLLM.</br> ^E Pre-computed embeddings can be inputted for this modality.</br> ⁺ Multiple items can be inputted per text prompt for this modality. ^* Only specific variants of the model support this modality (see notes below).</br>

!!! note Gemma3nForConditionalGeneration is only supported on V1 due to shared KV caching and it depends on timm>=1.0.17 to make use of its MobileNet-v5 vision backbone.

Performance is not yet fully optimized mainly due to:

- Both audio and vision MM encoders use `transformers.AutoModel` implementation.
- There's no PLE caching or out-of-memory swapping support, as described in [Google's blog](https://developers.googleblog.com/en/introducing-gemma-3n/). These features might be too model-specific for vLLM, and swapping in particular may be better suited for constrained setups.

!!! note For Gemma4ForConditionalGeneration: - audio input is only supported by the gemma-4-E2B and gemma-4-E4B variants. - The model does not ingest videos directly. However, vLLM’s Gemma 4 implementation supports video inputs by handling video processing internally. Users can send videos directly in the message structure to vLLM, where they are converted into text and image frames before being passed to the model.

!!! note For InternVLChatModel, only InternVL2.5 with Qwen2.5 text backbone (OpenGVLab/InternVL2.5-1B etc.), InternVL3 and InternVL3.5 have video inputs support currently.

!!! note To use TIGER-Lab/Mantis-8B-siglip-llama3, you have to pass --hf_overrides '{"architectures": ["MantisForConditionalGeneration"]}' when running vLLM.

!!! note The official openbmb/MiniCPM-V-2 doesn't work yet, so we need to use a fork (HwwwH/MiniCPM-V-2) for now. For more details, please see: https://github.com/vllm-project/vllm/pull/4087#issuecomment-2250397630

Transcription

Speech2Text models trained specifically for Automatic Speech Recognition.

!!! note VoxtralForConditionalGeneration requires mistral-common[audio] to be installed.

Realtime Transcription

Speech models that support streaming transcription via the /v1/realtime WebSocket endpoint.

!!! note VoxtralRealtimeGeneration requires mistral-common[audio] to be installed, and must be served with --tokenizer-mode mistral.

`Qwen3ASRRealtimeGeneration` is not auto-detected from `config.json`.
You must pass `--hf-overrides '{"architectures":["Qwen3ASRRealtimeGeneration"]}'`
when serving.

Pooling Models

See this page for more information on how to use pooling models.

!!! important Since some model architectures support both generative and pooling tasks, you should explicitly specify --runner pooling to ensure that the model is used in pooling mode instead of generative mode.

See the link below for more information on the models supported for specific pooling tasks.

• Classification Usages
• Embedding Usages
• Reward Usages
• Token Classification Usages
• Token Embedding Usages
• Scoring Usages
• Specific Model Examples

Model Support Policy

At vLLM, we are committed to facilitating the integration and support of third-party models within our ecosystem. Our approach is designed to balance the need for robustness and the practical limitations of supporting a wide range of models. Here’s how we manage third-party model support:

1. Community-Driven Support: We encourage community contributions for adding new models. When a user requests support for a new model, we welcome pull requests (PRs) from the community. These contributions are evaluated primarily on the sensibility of the output they generate, rather than strict consistency with existing implementations such as those in transformers. Call for contribution: PRs coming directly from model vendors are greatly appreciated!

2. Best-Effort Consistency: While we aim to maintain a level of consistency between the models implemented in vLLM and other frameworks like transformers, complete alignment is not always feasible. Factors like acceleration techniques and the use of low-precision computations can introduce discrepancies. Our commitment is to ensure that the implemented models are functional and produce sensible results.

   !!! tip When comparing the output of model.generate from Hugging Face Transformers with the output of llm.generate from vLLM, note that the former reads the model's generation config file (i.e., generation_config.json) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.

3. Issue Resolution and Model Updates: Users are encouraged to report any bugs or issues they encounter with third-party models. Proposed fixes should be submitted via PRs, with a clear explanation of the problem and the rationale behind the proposed solution. If a fix for one model impacts another, we rely on the community to highlight and address these cross-model dependencies. Note: for bugfix PRs, it is good etiquette to inform the original author to seek their feedback.

4. Monitoring and Updates: Users interested in specific models should monitor the commit history for those models (e.g., by tracking changes in the main/vllm/model_executor/models directory). This proactive approach helps users stay informed about updates and changes that may affect the models they use.

5. Selective Focus: Our resources are primarily directed towards models with significant user interest and impact. Models that are less frequently used may receive less attention, and we rely on the community to play a more active role in their upkeep and improvement.

Through this approach, vLLM fosters a collaborative environment where both the core development team and the broader community contribute to the robustness and diversity of the third-party models supported in our ecosystem.

Note that, as an inference engine, vLLM does not introduce new models. Therefore, all models supported by vLLM are third-party models in this regard.

We have the following levels of testing for models:

1. Strict Consistency: We compare the output of the model with the output of the model in the HuggingFace Transformers library under greedy decoding. This is the most stringent test. Please refer to models tests for the models that have passed this test.
2. Output Sensibility: We check if the output of the model is sensible and coherent, by measuring the perplexity of the output and checking for any obvious errors. This is a less stringent test.
3. Runtime Functionality: We check if the model can be loaded and run without errors. This is the least stringent test. Please refer to functionality tests and examples for the models that have passed this test.
4. Community Feedback: We rely on the community to provide feedback on the models. If a model is broken or not working as expected, we encourage users to raise issues to report it or open pull requests to fix it. The rest of the models fall under this category.