ContextQMD
Back to Mistral Rs

Supported models

docs/src/content/docs/reference/supported-models.md

0.9.213.1 KB
Original Source
<!-- Generated from the loader registry by mistralrs-core model_metadata. Do not edit by hand. -->

Is my model supported?

mistral.rs auto-detects the architecture from a repo's config.json. To check yours:

  1. Open the model's config.json on Hugging Face and read the architectures field (e.g. "Qwen3ForCausalLM", "Gemma4ForConditionalGeneration").
  2. Find the matching row below. Each architecture covers every checkpoint that reports that class, including future fine-tunes and sizes, so the families and examples here are a sample, not the full list.
  3. If the architecture is not listed, check the GGUF compatibility reference or request model support. Use --arch only when the checkpoint matches a known architecture.
bash
mistralrs run -m <model>     # interactive
mistralrs serve -m <model>   # OpenAI-compatible server

Expand a listed example to copy a ready-to-run command. One loader often serves several brand names (Qwen 3.5 and 3.6 share Qwen3_5; LFM2 and LFM2.5 share Lfm2) - the Model families column lists them. Behavior that differs from the defaults is collected in model family notes.

The Architecture column is the config.json architectures value. Per-family quantization, thinking, gated-repo, and tool-calling details live in model family notes.

Text models

ArchitectureModel familiesExample
MistralForCausalLMMistral<details><summary><code>mistralai/Mistral-7B-Instruct-v0.3</code></summary><code>mistralrs run -m mistralai/Mistral-7B-Instruct-v0.3</code></details>
GemmaForCausalLMGemma<details><summary><code>google/gemma-7b-it</code></summary><code>mistralrs run -m google/gemma-7b-it</code></details>
MixtralForCausalLMMixtral<details><summary><code>mistralai/Mixtral-8x7B-Instruct-v0.1</code></summary><code>mistralrs run -m mistralai/Mixtral-8x7B-Instruct-v0.1</code></details>
LlamaForCausalLMLlama 2, Llama 3.x<details><summary><code>meta-llama/Llama-3.1-8B-Instruct</code></summary><code>mistralrs run -m meta-llama/Llama-3.1-8B-Instruct</code></details>
PhiForCausalLMPhi-2<details><summary><code>microsoft/phi-2</code></summary><code>mistralrs run -m microsoft/phi-2</code></details>
Phi3ForCausalLMPhi-3, Phi-3.5<details><summary><code>microsoft/Phi-3-medium-4k-instruct</code></summary><code>mistralrs run -m microsoft/Phi-3-medium-4k-instruct</code></details>
Qwen2ForCausalLMQwen2, Qwen2.5<details><summary><code>Qwen/Qwen2.5-7B-Instruct</code> (2.5), <code>Qwen/Qwen2-7B-Instruct</code> (2)</summary><code>mistralrs run -m Qwen/Qwen2.5-7B-Instruct</code>
<code>mistralrs run -m Qwen/Qwen2-7B-Instruct</code></details>
Gemma2ForCausalLMGemma 2<details><summary><code>google/gemma-2-9b-it</code></summary><code>mistralrs run -m google/gemma-2-9b-it</code></details>
Starcoder2ForCausalLMStarcoder2<details><summary><code>bigcode/starcoder2-7b</code></summary><code>mistralrs run -m bigcode/starcoder2-7b</code></details>
PhiMoEForCausalLMPhi-3.5-MoE<details><summary><code>microsoft/Phi-3.5-MoE-instruct</code></summary><code>mistralrs run -m microsoft/Phi-3.5-MoE-instruct</code></details>
DeepseekV2ForCausalLMDeepSeek-V2<details><summary><code>deepseek-ai/DeepSeek-V2-Chat</code></summary><code>mistralrs run -m deepseek-ai/DeepSeek-V2-Chat</code></details>
DeepseekV3ForCausalLMDeepSeek-V3, DeepSeek-R1<details><summary><code>deepseek-ai/DeepSeek-V3</code> (V3), <code>deepseek-ai/DeepSeek-R1</code> (R1)</summary><code>mistralrs run -m deepseek-ai/DeepSeek-V3</code>
<code>mistralrs run -m deepseek-ai/DeepSeek-R1</code></details>
Qwen3ForCausalLMQwen3<details><summary><code>Qwen/Qwen3-4B</code></summary><code>mistralrs run -m Qwen/Qwen3-4B</code></details>
Glm4ForCausalLMGLM-4<details><summary><code>zai-org/GLM-4-32B-0414</code></summary><code>mistralrs run -m zai-org/GLM-4-32B-0414</code></details>
Glm4MoeLiteForCausalLMGLM-4.7-Flash<details><summary><code>zai-org/GLM-4.7-Flash</code></summary><code>mistralrs run -m zai-org/GLM-4.7-Flash</code></details>
Glm4MoeForCausalLMGLM-4.7<details><summary><code>zai-org/GLM-4.7</code></summary><code>mistralrs run -m zai-org/GLM-4.7</code></details>
Qwen3MoeForCausalLMQwen3 MoE<details><summary><code>Qwen/Qwen3-30B-A3B</code></summary><code>mistralrs run -m Qwen/Qwen3-30B-A3B</code></details>
SmolLM3ForCausalLMSmolLM3<details><summary><code>HuggingFaceTB/SmolLM3-3B</code></summary><code>mistralrs run -m HuggingFaceTB/SmolLM3-3B</code></details>
GraniteMoeHybridForCausalLMGranite 4.0<details><summary><code>ibm-granite/granite-4.0-micro</code></summary><code>mistralrs run -m ibm-granite/granite-4.0-micro</code></details>
GptOssForCausalLMGPT-OSS<details><summary><code>openai/gpt-oss-20b</code> (20b), <code>openai/gpt-oss-120b</code> (120b)</summary><code>mistralrs run -m openai/gpt-oss-20b</code>
<code>mistralrs run -m openai/gpt-oss-120b</code></details>
HunYuanDenseV1ForCausalLMHunYuan<details><summary><code>tencent/Hunyuan-7B-Instruct</code></summary><code>mistralrs run -m tencent/Hunyuan-7B-Instruct</code></details>
HunYuanMoEV1ForCausalLMHunYuan MoE<details><summary><code>tencent/Hunyuan-A13B-Instruct</code></summary><code>mistralrs run -m tencent/Hunyuan-A13B-Instruct</code></details>
Qwen3NextForCausalLMQwen3-Next, Qwen3-Coder-Next<details><summary><code>Qwen/Qwen3-Next-80B-A3B-Instruct</code></summary><code>mistralrs run -m Qwen/Qwen3-Next-80B-A3B-Instruct</code></details>
Qwen3_5ForCausalLMQwen3.5No published example
Lfm2ForCausalLMLFM2, LFM2.5<details><summary><code>LiquidAI/LFM2.5-1.2B-Instruct</code> (LFM2.5), <code>LiquidAI/LFM2-1.2B</code> (LFM2)</summary><code>mistralrs run -m LiquidAI/LFM2.5-1.2B-Instruct</code>
<code>mistralrs run -m LiquidAI/LFM2-1.2B</code></details>
Lfm2MoeForCausalLMLFM2 MoE, LFM2.5 MoE<details><summary><code>LiquidAI/LFM2.5-8B-A1B</code> (LFM2.5), <code>LiquidAI/LFM2-8B-A1B</code> (LFM2)</summary><code>mistralrs run -m LiquidAI/LFM2.5-8B-A1B</code>
<code>mistralrs run -m LiquidAI/LFM2-8B-A1B</code></details>

Multimodal models

ArchitectureModel familiesExample
Phi3VForCausalLMPhi-3.5-Vision<details><summary><code>microsoft/Phi-3.5-vision-instruct</code></summary><code>mistralrs run -m microsoft/Phi-3.5-vision-instruct</code></details>
Idefics2ForConditionalGenerationIdefics2<details><summary><code>HuggingFaceM4/idefics2-8b</code></summary><code>mistralrs run -m HuggingFaceM4/idefics2-8b</code></details>
LlavaNextForConditionalGenerationLLaVA-NeXT<details><summary><code>llava-hf/llava-v1.6-mistral-7b-hf</code></summary><code>mistralrs run -m llava-hf/llava-v1.6-mistral-7b-hf</code></details>
LlavaForConditionalGenerationLLaVA 1.5<details><summary><code>llava-hf/llava-1.5-7b-hf</code></summary><code>mistralrs run -m llava-hf/llava-1.5-7b-hf</code></details>
Lfm2VlForConditionalGenerationLFM2-VL, LFM2.5-VL<details><summary><code>LiquidAI/LFM2.5-VL-1.6B</code> (1.6B), <code>LiquidAI/LFM2.5-VL-450M</code> (450M)</summary><code>mistralrs run -m LiquidAI/LFM2.5-VL-1.6B</code>
<code>mistralrs run -m LiquidAI/LFM2.5-VL-450M</code></details>
MllamaForConditionalGenerationLlama 3.2 Vision<details><summary><code>meta-llama/Llama-3.2-11B-Vision-Instruct</code></summary><code>mistralrs run -m meta-llama/Llama-3.2-11B-Vision-Instruct</code></details>
Qwen2VLForConditionalGenerationQwen2-VL<details><summary><code>Qwen/Qwen2-VL-7B-Instruct</code></summary><code>mistralrs run -m Qwen/Qwen2-VL-7B-Instruct</code></details>
Idefics3ForConditionalGenerationIdefics3, SmolVLM<details><summary><code>HuggingFaceM4/Idefics3-8B-Llama3</code></summary><code>mistralrs run -m HuggingFaceM4/Idefics3-8B-Llama3</code></details>
MiniCPMOMiniCPM-o<details><summary><code>openbmb/MiniCPM-o-2_6</code></summary><code>mistralrs run -m openbmb/MiniCPM-o-2_6</code></details>
Phi4MMForCausalLMPhi-4-multimodal<details><summary><code>microsoft/Phi-4-multimodal-instruct</code></summary><code>mistralrs run -m microsoft/Phi-4-multimodal-instruct</code></details>
Qwen2_5_VLForConditionalGenerationQwen2.5-VL<details><summary><code>Qwen/Qwen2.5-VL-7B-Instruct</code></summary><code>mistralrs run -m Qwen/Qwen2.5-VL-7B-Instruct</code></details>
Gemma3ForConditionalGenerationGemma 3<details><summary><code>google/gemma-3-12b-it</code></summary><code>mistralrs run -m google/gemma-3-12b-it</code></details>
Mistral3ForConditionalGenerationMistral Small 3<details><summary><code>mistralai/Mistral-Small-3.2-24B-Instruct-2506</code></summary><code>mistralrs run -m mistralai/Mistral-Small-3.2-24B-Instruct-2506</code></details>
Llama4ForConditionalGenerationLlama 4<details><summary><code>meta-llama/Llama-4-Scout-17B-16E-Instruct</code></summary><code>mistralrs run -m meta-llama/Llama-4-Scout-17B-16E-Instruct</code></details>
Gemma3nForConditionalGenerationGemma 3n<details><summary><code>google/gemma-3n-E4B-it</code></summary><code>mistralrs run -m google/gemma-3n-E4B-it</code></details>
Qwen3VLForConditionalGenerationQwen3-VL<details><summary><code>Qwen/Qwen3-VL-4B-Instruct</code></summary><code>mistralrs run -m Qwen/Qwen3-VL-4B-Instruct</code></details>
Qwen3VLMoeForConditionalGenerationQwen3-VL MoE<details><summary><code>Qwen/Qwen3-VL-235B-A22B-Instruct</code></summary><code>mistralrs run -m Qwen/Qwen3-VL-235B-A22B-Instruct</code></details>
Qwen3_5ForConditionalGenerationQwen 3.5, Qwen 3.6<details><summary><code>Qwen/Qwen3.5-27B</code> (3.5), <code>Qwen/Qwen3.6-27B</code> (3.6)</summary><code>mistralrs run -m Qwen/Qwen3.5-27B</code>
<code>mistralrs run -m Qwen/Qwen3.6-27B</code></details>
Qwen3_5MoeForConditionalGenerationQwen 3.5 MoE, Qwen 3.6 MoE<details><summary><code>Qwen/Qwen3.5-35B-A3B</code> (3.5), <code>Qwen/Qwen3.6-35B-A3B</code> (3.6)</summary><code>mistralrs run -m Qwen/Qwen3.5-35B-A3B</code>
<code>mistralrs run -m Qwen/Qwen3.6-35B-A3B</code></details>
VoxtralRealtimeForConditionalGenerationVoxtral<details><summary><code>mistralai/Voxtral-Mini-4B-Realtime-2602</code></summary><code>mistralrs run -m mistralai/Voxtral-Mini-4B-Realtime-2602</code></details>
Gemma4ForConditionalGenerationGemma 4<details><summary><code>google/gemma-4-E4B-it</code> (E4B), <code>google/gemma-4-26B-A4B-it</code> (26B-A4B MoE), <code>google/gemma-4-31B-it</code> (31B dense)</summary><code>mistralrs run -m google/gemma-4-E4B-it</code>
<code>mistralrs run -m google/gemma-4-26B-A4B-it</code>
<code>mistralrs run -m google/gemma-4-31B-it</code></details>
MuseGlimmerForConditionalGenerationMuse Glimmer<details><summary><code>meta-models/Muse-Glimmer-30B</code></summary><code>mistralrs run -m meta-models/Muse-Glimmer-30B</code></details>
DiffusionGemmaForBlockDiffusionDiffusionGemma<details><summary><code>google/diffusiongemma-26B-A4B-it</code></summary><code>mistralrs run -m google/diffusiongemma-26B-A4B-it</code></details>

Image generation

ArchitectureModel familiesExample
FluxFLUX.1<details><summary><code>black-forest-labs/FLUX.1-schnell</code></summary><code>mistralrs run -m black-forest-labs/FLUX.1-schnell</code></details>
FluxOffloadedFLUX.1 (offloaded)<details><summary><code>black-forest-labs/FLUX.1-schnell</code></summary><code>mistralrs run -m black-forest-labs/FLUX.1-schnell</code></details>

Speech

ArchitectureModel familiesExample
DiaDia<details><summary><code>nari-labs/Dia-1.6B</code></summary><code>mistralrs run -m nari-labs/Dia-1.6B</code></details>

Embedding

ArchitectureModel familiesExample
Gemma3TextModelEmbeddingGemma<details><summary><code>google/embeddinggemma-300m</code></summary><code>mistralrs run -m google/embeddinggemma-300m</code></details>
Qwen3ForCausalLMQwen3 Embedding<details><summary><code>Qwen/Qwen3-Embedding-0.6B</code></summary><code>mistralrs run -m Qwen/Qwen3-Embedding-0.6B</code></details>

Format and quantization notes

Text, multimodal, speech, and embedding models support ISQ at load time. Diffusion models (FLUX) do not; they load at native precision. See GGUF support for GGUF compatibility; availability of UQFF, GPTQ, and AWQ artifacts varies by model on Hugging Face.

Speculative decoding

ModeTarget architectureAssistant checkpoint familyGuide
MTPGemma4Gemma 4 assistant checkpoints, PagedAttention requiredSpeculative decoding (MTP)
MTPQwen3_5, Qwen3.8Built-in mtp.* head (--mtp), PagedAttention requiredSpeculative decoding (MTP)