TOML configuration

mistralrs from-config -f <path> reads a TOML file. The top-level command field selects serve or run. Every key maps to a CLI flag of the same subcommand; the mapping is listed per table below. For per-flag semantics, see the generated CLI reference.

Minimal example

command = "serve"

[server]
host = "0.0.0.0"
port = 1234

[[models]]
model_id = "Qwen/Qwen3-4B"

[models.quantization]
quant = "4"

mistralrs from-config -f this.toml runs the server.

Top-level fields

[global] section

-v/--verbose has no TOML equivalent; use RUST_LOG instead.

[runtime] section

[server] section (serve only)

:::caution The default host = "0.0.0.0" binds on all interfaces, exposing the server to your network. There is no built-in authentication. Set host = "127.0.0.1" for local-only access, or put an authenticating reverse proxy in front before exposing it. :::

The MCP client configuration (mcp_config) lives under [runtime], not [server]: it applies to run as well as serve.

[paged_attn] section

[sandbox] section

OS-level isolation for the code-execution subprocess. Mechanics and threat model: sandbox reference.

[[models]] array

Each entry defines one loaded model.

Each [[models]] entry can carry nested sections whose field shapes mirror the corresponding CLI flags:

Multi-model example

command = "serve"
default_model_id = "Qwen/Qwen3-4B"

[server]
host = "0.0.0.0"
port = 1234

[runtime]
enable_search = true
search_embedding_model = "embedding-gemma"

[[models]]
model_id = "Qwen/Qwen3-4B"

[models.quantization]
quant = "4"

[[models]]
model_id = "google/gemma-4-E4B-it"

[models.quantization]
quant = "4"

Validation

Invalid configs abort startup with a message identifying the problem:

• At least one entry in [[models]].
• default_model_id matches a model_id in [[models]].
• cpu is consistent across all models when set.
• search_embedding_model requires enable_search = true (or agent = true).
• code_exec_python, code_exec_timeout, and code_exec_workdir each require enable_code_execution = true (or agent = true).

CLI usage notes

Flag interactions that hold on the command line and as TOML keys:

• quant (CLI --quant, TOML key quant) is the front door: it tries a prebuilt UQFF (Universal Quantized File Format) first and falls back to ISQ (in-situ quantization). It conflicts with isq (--isq, the explicit ISQ level) and from_uqff (--from-uqff). mistralrs tune rejects quant = "auto" (--quant auto) because tune is the recommender.
• --calibration-file conflicts with --imatrix.
• --xlora conflicts with --lora. --xlora-order and --tgt-non-granular-index require --xlora; --xlora alone is accepted.
• --matformer-slice-name requires --matformer-config-path.
• mistralrs run: --image, --video, and --audio require -i/--input.
• mistralrs bench: --prompt-len and --depth accept comma-separated values for sweeps.
  • Each --prompt-len value produces a prefill measurement at that prompt length.
  • Each --depth value produces a decode measurement that prefills depth tokens and then generates --gen-len tokens.
  • --depth must be greater than 0 when --gen-len is greater than 0.

Server behavior notes

• CORS and body limit. Not exposed as CLI flags or TOML keys. Defaults: any origin; methods GET, POST, PUT, DELETE; allowed headers Content-Type, Authorization, x-api-key, anthropic-version, anthropic-beta, x-request-id; exposed headers x-request-id; 50 MB request body limit. Configure programmatically through MistralRsServerRouterBuilder in mistralrs-server-core.
• Authentication. mistral.rs does not implement authentication. Put a reverse proxy (nginx, Caddy, Traefik) in front for auth and TLS. OpenAI-protocol clients always send Authorization: Bearer ... because the OpenAI SDK requires an API key; mistral.rs does not validate the header.
• Logging and metrics. Access logs are written to normal server stdout/stderr by default, with request ids and route/status/latency metadata. GET /metrics exposes Prometheus HTTP metrics by default. See observability.
• Payload logging. -v enables debug detail and -vv trace-level file/cache internals; RUST_LOG module filters (e.g. RUST_LOG=mistralrs_core=debug,tower_http=info) override both. -l <path> logs all requests and responses to a file.