mistral.rs

import { Card, CardGrid, LinkCard } from '@astrojs/starlight/components';

Start with a task

<CardGrid> <LinkCard title="Quickstart" href="/mistral.rs/quickstart/" description="Install the CLI, chat with a model, and serve an API in ten minutes." /> <LinkCard title="Serve an OpenAI-compatible API" href="/mistral.rs/guides/serve/openai-compatible-apis/" description="Expose a model through OpenAI-compatible endpoints and the Anthropic Messages API." /> <LinkCard title="Run any model" href="/mistral.rs/guides/models/run-any-model/" description="Hugging Face ids, local files, GGUF, and quantization flags." /> <LinkCard title="Quantize a model" href="/mistral.rs/guides/quantization/quantize-a-model/" description="Fit larger models on the same hardware with prebuilt UQFF (Universal Quantized File Format) files or in-situ quantization." /> <LinkCard title="Build agents" href="/mistral.rs/guides/agents/" description="Tool calling, file inputs, web search, code execution, shell, OpenAI-compatible Skills, Model Context Protocol (MCP), and sessions." /> <LinkCard title="Use OpenAI-compatible Skills" href="/mistral.rs/guides/agents/skills/" description="Upload Skills, reference them from Responses requests, and run them with shell execution." /> <LinkCard title="Attach files to requests" href="/mistral.rs/guides/agents/file-inputs/" description="Use OpenAI-compatible uploads, input_file parts, Chat file parts, and shell/code workdir mounts." /> <LinkCard title="Use Python" href="/mistral.rs/guides/python/getting-started/" description="Call models from Python without running a separate server." /> <LinkCard title="Use Rust" href="/mistral.rs/guides/rust/getting-started/" description="Embed mistral.rs in a Rust application." /> <LinkCard title="Browse examples" href="/mistral.rs/examples/" description="Runnable HTTP, Python, and Rust examples organized by topic." /> </CardGrid>

Features

<CardGrid> <Card title="Auto-detection" icon="rocket"> `mistralrs run -m <model>` works without flags. The binary infers the architecture and chat template from the Hugging Face repository, and selects the accelerator from the detected hardware. [Quickstart](/mistral.rs/quickstart/) </Card> <Card title="Quantization" icon="puzzle"> `--quant` picks the best format at a given level, preferring prebuilt [UQFF](/mistral.rs/reference/uqff-format/) and falling back to in-situ quantization. [Quantization guide](/mistral.rs/guides/quantization/quantize-a-model/) </Card> <Card title="Tuning and diagnostics" icon="seti:config"> `mistralrs tune` recommends quantization and device mapping from the model config and detected hardware, without loading the model. `mistralrs doctor` checks hardware, build features, and Hugging Face connectivity. [CLI reference](/mistral.rs/reference/cli/) </Card> <Card title="Agents" icon="open-book"> Local agent runtime with a server-side tool loop, file inputs, web search, code execution, shell, OpenAI-compatible Skills, MCP, generated media, and sessions. [Agents & tools](/mistral.rs/guides/agents/) </Card> <Card title="Multi-GPU" icon="seti:hex"> Scale a model across GPUs and nodes: NCCL tensor parallelism, layer/P2P device mapping, multi-node NCCL, and the ring backend. [Distributed inference](/mistral.rs/guides/perf/distributed-inference/) </Card> <Card title="Web UI" icon="laptop"> Browser chat with reasoning blocks, tool, code, and shell call visualization, and inline search results. [Web UI](/mistral.rs/guides/serve/with-web-ui/) </Card> </CardGrid>

Reference

<CardGrid> <LinkCard title="CLI reference" href="/mistral.rs/reference/cli/" description="Every command and flag, generated from the binary." /> <LinkCard title="HTTP API" href="/mistral.rs/reference/http-api/" description="Endpoints, schemas, and streaming events." /> <LinkCard title="Supported models" href="/mistral.rs/reference/supported-models/" description="Model families and capability tables." /> <LinkCard title="Troubleshooting" href="/mistral.rs/reference/troubleshooting/" description="Symptom-indexed fixes for build and runtime issues." /> </CardGrid>

Community

• Discord
• Issues on GitHub
• Contributing guide