ContextQMD
Back to Claude Scientific Skills

llama.cpp

skills/pi-agent/references/llama-cpp.md

2.57.03.0 KB
Original Source

llama.cpp

Source: https://pi.dev/docs/latest/llama-cpp

Pi supports the llama.cpp router server, which discovers multiple GGUF models and loads or unloads them on demand. Use a current llama.cpp build with router support.

Start the Router

Start llama-server without --model, -m, or -hf. Passing a model starts single-model mode instead of router mode.

bash
llama-server \
  --models-dir ~/models \
  --no-models-autoload \
  --jinja \
  --host 127.0.0.1 \
  --port 8080 \
  -ngl 999 \
  -c 32768
  • --models-dir discovers local GGUF files.
  • --no-models-autoload keeps loading explicit through /llama.
  • --jinja enables compatible chat templates and tool calling.
  • -ngl 999 offloads as many layers as possible to the GPU.
  • -c 32768 sets the context window per loaded model; omit to use the model's native context (may need much more memory).

Single-file models sit directly in the models directory. Multimodal and multi-shard models go in their own subdirectories with all components included. Restart the router after manually adding files; use llama.cpp model presets for per-model context sizes.

Configure Pi

text
/login llama.cpp

Enter the router URL (default http://127.0.0.1:8080) and optional API key. Environment variables configure the same values without /login:

bash
export LLAMA_BASE_URL=http://127.0.0.1:8080
export LLAMA_API_KEY=optional-secret
pi

If the server uses an API key, start llama-server with the matching --api-key. Keep --host 127.0.0.1 for local-only access.

Manage Models with /llama

  • Select an unloaded model to load it; select a loaded model to unload it.
  • Select Download model… to search Hugging Face, then choose a repository and quantization. Exact owner/repository[:quant] values also work.
  • Escape during a load or download prompts to confirm cancellation.
  • If the router disconnects, /llama offers Retry and Close; Retry reconnects and refreshes state without replaying the interrupted operation.

Hugging Face search uses HF_TOKEN when set, then $HF_TOKEN_PATH, $HF_HOME/token, $XDG_CACHE_HOME/huggingface/token, and ~/.cache/huggingface/token. Search works unauthenticated with lower rate limits. Pi warns before downloading gated repositories. The llama.cpp server performs the download, so its process also needs HF_TOKEN for gated repositories.

Pi never silently unloads models and never deletes model files; it asks whether to unload others first. The router may be shared with other clients, so /llama always displays the router's current state.

Only loaded models appear in /model. Load with /llama, then select the model with /model.

Troubleshooting

bash
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/models
  • No models in /llama: check --models-dir, the directory layout, and restart the router.
  • Model missing from /model: load it with /llama first.
  • Load fails or uses too much memory: lower -c or unload another model.
  • Server not in router mode: start it without --model, -m, or -hf.