Summary

Introduction

Getting Started

• Installation
• Cargo Features
• CLI Reference
• CLI TOML Configuration
• Troubleshooting

SDKs & APIs

• Rust SDK
• Python SDK
• Python Installation
• HTTP Server
• OpenResponses API

Models

• Supported Models
• Multimodal Models
• Image Generation
• Speech Models
• Embeddings

Text Model Guides

• DeepSeek V2
• DeepSeek V3
• Gemma 2
• Gemma 3
• Gemma 3n
• GLM4
• GLM-4.7-Flash
• GLM-4.7 MoE
• GPT-OSS
• Qwen 3
• Qwen 3 Next
• SmolLM3

Multimodal Model Guides

• Gemma 4
• Idefics 2
• Idefics 3
• LLaVA
• Llama 3.2 Vision
• Llama 4
• MiniCPM-O 2.6
• Mistral 3
• Phi 3.5 MoE
• Phi 3.5 Vision
• Phi 4 Multimodal
• Qwen 2-VL
• Qwen 3-VL
• Qwen 3.5
• Video Input

Other Model Guides

• FLUX (Diffusion)
• Voxtral (Speech-to-Text)
• Dia (Speech)
• EmbeddingGemma
• Qwen3 Embedding

Quantization & Optimization

• Quantization Overview
• ISQ (In-Situ Quantization)
• UQFF Format
  • UQFF Layout
• Topology
• Importance Matrix

Adapters & Model Customization

• Adapter Models
• LoRA/X-LoRA Examples
• Non-Granular Scalings
• AnyMoE
• MatFormer

Performance & Hardware

• Device Mapping
• PagedAttention
• Speculative Decoding
• Flash Attention
• MLA (Multi-head Latent Attention)
• Distributed Inference
  • NCCL Backend
  • Ring Backend

Features

• Tool Calling
• Web Search
• Chat Templates
• Sampling Options
• TOML Selector
• Multi-Model Support

MCP (Model Context Protocol)

• MCP Client
• MCP Server
• MCP Configuration
• MCP Transports
• MCP Advanced Usage

Reference

• Configuration
• Engine Internals