llama.cpp Patches for Llamafile

This directory contains patches that adapt llama.cpp for use with Llamafile and Cosmopolitan libc. These patches enable llama.cpp to run as a portable, single-file executable across Windows, macOS, Linux, and BSD without installation.

Directory Structure

llama.cpp.patches/
├── README.md              # This file
├── apply-patches.sh       # Script to apply all patches to llama.cpp submodule
├── fetch-ui-assets.sh     # Downloads + validates the prebuilt web UI (see Server Integration)
├── renames.sh             # Script for file renames/moves (if any)
├── llamafile-files/       # Additional files to copy into llama.cpp
│   ├── BUILD.mk           # Makefile for building llama.cpp with cosmocc
│   ├── README.llamafile   # License and modification notes
│   └── common/
│       └── license.cpp    # Llama.cpp's license file (cmake creates this at build time)
└── patches/               # Patch files for upstream sources

Applying Patches

To apply all patches to the llama.cpp submodule:

./llama.cpp.patches/apply-patches.sh

To reset the submodule to its clean state:

cd llama.cpp && git reset --hard && git clean -fdx

Patch Index

Windows/macOS ABI Compatibility (GGML_CALL)

GPU backends (CUDA, Vulkan, Metal) are compiled as shared libraries (.dll/.so/.dylib) using native compilers, but the llamafile host binary is built with Cosmopolitan libc which uses System V AMD64 ABI everywhere — including on Windows. When the host calls function pointers inside backend interface structs, the calling convention must match.

The GGML_CALL macro (defined as __attribute__((__ms_abi__)) when GGML_MULTIPLATFORM is set) annotates all function pointers in the backend interface structs and their implementations, so the correct calling convention is used on every platform.

Cross-Module Memory Management

When GPU backends (CUDA, Vulkan, Metal) are loaded as dynamic libraries, memory allocated by the DSO must be freed by the DSO's allocator, not the main executable's.

Cosmopolitan Libc Compatibility

These patches address compatibility issues when building with Cosmopolitan libc (cosmocc).

Threading and Signal Handling

Cosmopolitan libc has specific behaviors with condition variables and signals that require workarounds.

HTTPS / TLS Support

Upstream llama.cpp gets TLS from cpp-httplib's OpenSSL backend (CPPHTTPLIB_OPENSSL_SUPPORT, satisfied by system OpenSSL or vendored BoringSSL/LibreSSL at cmake time). None of those is available in the cosmocc make build, so llamafile instead enables cpp-httplib's Mbed TLS backend (CPPHTTPLIB_MBEDTLS_SUPPORT) against the mbedtls fork already vendored in third_party/mbedtls — the same TLS stack llamafile <= 0.9.3 used. third_party/mbedtls/include/ maps the canonical <mbedtls/*.h> include paths onto the fork's headers, and BUILD.mk sets the macro on every object that can reach httplib.h (the macro changes httplib class layouts, so all TUs must agree) and links mbedtls.a into llama-server. This enables HTTPS model downloads (-hf, --model-url), https clients in server-models, and TLS serving via --ssl-cert-file/--ssl-key-file.

TinyBLAS Integration

Llamafile uses TinyBLAS as a lightweight replacement for cuBLAS, enabling GPU support without CUDA SDK dependencies.

Optional IQ-Quant Exclusion (CUDA)

The IQ ("importance") quantization formats (IQ1_S, IQ2_XXS/XS/S, IQ3_S/XXS, IQ4_NL/XS) pull in a large amount of CUDA template instantiation that inflates compile time and binary size. These patches gate the IQ code paths behind #ifndef GGML_CUDA_NO_IQ_QUANTS — the MMQ/MMVQ matmul kernels, the f32 → IQ4_NL copy, and the IQ dequant cases in ggml_get_to_bf16_cuda/ggml_get_to_fp16_cuda — so a build can compile them out by defining GGML_CUDA_NO_IQ_QUANTS. (ggml_get_to_fp32_cuda's IQ cases are not guarded, so the float dequant-template instantiations still compile — a minor size cost, harmless because ggml_backend_cuda_device_supports_op gates the same IQ ops, so those tensors fall back to CPU in a minimized build regardless.) When the macro is undefined (the default), behavior is unchanged.

CPU Performance Optimizations (llamafile #975)

These patches restore llamafile's optimized CPU kernels (TinyBLAS matmul, AVX-512 flash-attention helpers) on top of upstream's CPU backend, and tune CPU-only defaults. The hooks call into symbols exported from llamafile/sgemm.cpp and are compiled only when GGML_USE_LLAMAFILE is defined.

Llamafile File Handling

These patches integrate llamafile's file handling APIs for loading models from bundled zip archives and .llamafile containers.

Server Integration

The web UI moved upstream from prebuilt tools/server/public/* assets to a Svelte/PWA project under tools/ui/, embedded at CMake time via tools/ui/embed.cpp. cosmocc has no JS toolchain, so fetch-ui-assets.sh (run by apply-patches.sh / make setup) downloads the prebuilt site tarball dist.tar.gz (plus its .sha256) from the ggml-org/llama-ui Hugging Face bucket — picking the newest bNNNN tag <= our pinned build — verifies it, and extracts the whole static site into llama.cpp/tools/ui/dist/. The modern site is no longer four flat files (bundle.css/bundle.js/index.html/loading.html) but a hashed tree (_app/immutable/bundle.HASH.js, a service worker, manifest, icons, splash screens). To keep the embedded payload small, the script also builds a dist/_gzip/ mirror with every file gzip-compressed under its original name.

At build time, llama.cpp/BUILD.mk compiles tools/ui/embed.cpp with cosmoc++ (its APE output runs on the host) and runs it against the dist/ directory (the new embed <out_cpp> <out_h> [<asset_dir>] interface): embed.cpp recursively bakes every file — auto-detecting dist/_gzip/ and emitting gzip-encoded assets keyed by relative path — into o/$(MODE)/llama.cpp/tools/ui/ui.{cpp,h}, which is compiled like any other C++ source and linked into llama-server and llamafile. Upstream's server-http.cpp (unpatched) registers a route per embedded asset (index.html at /) and serves gzip assets with Content-Encoding: gzip. If the download fails (offline, version not yet on HF) the script leaves dist/ empty — embed.cpp then emits a no-op llama_ui_find_asset and the LLAMA_UI_HAS_ASSETS guard keeps the UI routes unregistered, so the REST API still works.

Note that embed.cpp only takes that graceful no-asset path when dist/ is empty: once the directory is non-empty it enforces a required-asset set (its required_check[] table — index.html, loading.html, manifest.webmanifest, sw.js, build.json, version.json, and the hashed bundle*.js/bundle*.css/workbox*.js) and returns a hard error if any is missing. So fetch-ui-assets.sh validates the same set after extracting (ui_missing_assets); if a downloaded tarball is partial or has drifted, it clears dist/ and falls back to a UI-less build rather than letting the embed step abort the whole build. Because embed.cpp is an upstream file, that list can change on a llama.cpp bump — when it does, the asset list in ui_missing_assets must be updated to match.

Bug Fixes

Creating New Patches

Files in llama.cpp are usually modified in-place for development and testing. Once they are ready to be committed, you can update all files in the llama.cpp.patches directory by running the following:

# echo y answers the prompt; the subshell restores the cwd even on failure
( cd llama.cpp && echo y | ../tools/generate_patches.sh --output-dir ../llama.cpp.patches )

Patch filenames will automatically reflect the file path with underscores replacing slashes (e.g., common_arg.cpp.patch for common/arg.cpp).