llama.cpp

Pi supports the llama.cpp router server. The router discovers multiple GGUF models and loads or unloads them on demand.

Use a current llama.cpp build with router support. Follow the build instructions or install a prebuilt release for your platform.

Start the router

Start llama-server without --model or -m. Passing a model starts single-model mode instead of router mode.

llama-server \
  --models-dir ~/models \
  --no-models-autoload \
  --jinja \
  --host 127.0.0.1 \
  --port 8080 \
  -ngl 999 \
  -c 32768

Important options:

• --models-dir ~/models discovers local GGUF files.
• --no-models-autoload keeps loading explicit through /llama.
• --jinja enables compatible chat templates and tool calling.
• -ngl 999 offloads as many layers as possible to the GPU.
• -c 32768 sets the context window for each loaded model. Omit it to use the model's native context, which may require substantially more memory.

A single-file model can sit directly in the model directory. Put multimodal and multi-shard models in separate subdirectories:

~/models/
├── llama-3.2-1b-Q4_K_M.gguf
├── gemma-3-4b-it-Q4_K_M/
│   ├── gemma-3-4b-it-Q4_K_M.gguf
│   └── mmproj-F16.gguf
└── large-model-Q4_K_M/
    ├── large-model-Q4_K_M-00001-of-00003.gguf
    ├── large-model-Q4_K_M-00002-of-00003.gguf
    └── large-model-Q4_K_M-00003-of-00003.gguf

Restart the router after manually adding files. For per-model context sizes and other options, use llama.cpp model presets.

Configure Pi

Start Pi and configure the provider:

/login llama.cpp

Enter the router URL and optional API key. The default URL is http://127.0.0.1:8080.

Environment variables can configure the same values without /login:

export LLAMA_BASE_URL=http://127.0.0.1:8080
export LLAMA_API_KEY=optional-secret
pi

If the server uses an API key, start llama-server with the matching --api-key value. Keep --host 127.0.0.1 for local-only access.

Manage models

Run:

/llama

• Select an unloaded model to load it.
• Select a loaded model to unload it.
• Select Download model…, search Hugging Face, then choose a repository and quantization. Exact owner/repository[:quant] values also work.
• Press Escape during a load or download to confirm cancellation.

Hugging Face search uses HF_TOKEN when set, then checks $HF_TOKEN_PATH, $HF_HOME/token, $XDG_CACHE_HOME/huggingface/token, and ~/.cache/huggingface/token. Search also works without authentication, subject to lower rate limits. Pi warns before downloading gated repositories and links to their access page. The llama.cpp server performs the download, so its process must also have HF_TOKEN when the selected repository requires access.

If other models are loaded, Pi asks whether to unload them first or keep them loaded. Pi does not silently unload models and never deletes model files. The router may be shared with other clients, so /llama always displays the router's current state.

Only loaded models appear in /model. After loading a model, run /model to select it for the current Pi session.

If the router disconnects, /llama shows Retry and Close. Retry reconnects and refreshes model state without replaying the interrupted operation.

Troubleshooting

Check that the router is reachable:

curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/models

• No models in /llama: Check --models-dir, the directory layout, and restart the router.
• Model missing from /model: Load it with /llama first.
• Load fails or uses too much memory: Lower -c or unload another model.
• Server is not in router mode: Start it without --model, -m, or -hf.