polarquant-cpu

C reference + AVX2 + NEON kernels and a GGUF converter for the on-device PolarQuant Q4 weight format (block_q4_polar, GGML type tag Q4_POLAR=45).

The standalone static library here is the behavioural source of truth for the kernels. Drop-in patches for the Apothic-AI/llama.cpp-1bit-turboquant fork live under fork-integration/ (separate quants-polar.{h,c} + .patch deltas for ggml-common.h, ggml.h, ggml-cpu.c, ggml-quants.c, and ggml/src/ggml-cpu/CMakeLists.txt).

What is in here

Block format (locked)

#define QK_POLAR 128
#define QJL_RESIDUAL_BYTES (QK_POLAR / 8)   // 16 bytes

typedef struct __attribute__((packed)) {
    polar_fp16_t d;                          // 2  bytes (per-block L2 norm)
    uint8_t      qs[QK_POLAR / 2];           // 64 bytes (4-bit codes, 2 per byte)
    uint8_t      qjl[QJL_RESIDUAL_BYTES];    // 16 bytes (1-bit residual per block)
} block_q4_polar;

// 82 bytes/block.  5.125 bpw with QJL, 4.125 bpw without.

qs: low nibble = even-index code, high nibble = odd-index code (matches the layout llama.cpp's existing 4-bit kernels assume so SIMD unpacking ports cleanly).

qjl[0] bit 0 holds the per-block residual sign; bytes 1..15 are reserved for a future per-coordinate residual without breaking the on-disk size.

Build + test

cmake -B build -S .
cmake --build build -j
ctest --test-dir build --output-on-failure

Centroid regeneration

The committed centroid header is the bit-for-bit output of:

python scripts/gen_centroids.py > include/polarquant/polar_centroids.h

The Lloyd-Max iteration is deterministic (16 levels, 100 iterations, fixed initial boundaries on [-4, 4]). gen_centroids.py mirrors packages/training/scripts/quantization/polarquant/polar_quant.py::_compute_lloyd_max_centroids exactly.

GGUF converter

python scripts/polarquant_to_gguf.py \
  --sidecar  /path/to/polarquant_artifacts.safetensors \
  --base-model /path/to/base/hf/model_dir \
  --output   /path/to/out.gguf

Reads the sidecar's <layer>.codes (int8), <layer>.norms (fp16), optional <layer>.qjl (uint8) tensors; packs each layer into block_q4_polar records; and writes a GGUF where every quantized tensor is typed Q4_POLAR=45. Header metadata:

The decoder is expected to verify these against its compile-time constants and refuse to load on any mismatch.

Test

python scripts/test_converter.py

Synthesizes a 128x128 fp32 weight, runs the vendored PolarQuant encoder over it, drives the converter, and reads the GGUF back via gguf.GGUFReader (with Q4_POLAR=45 patched into the enum to mirror what the upstream registration step will do).

Validation results

The per-block reconstruction error (~9-10%) is not a quality knob. PolarQuant Q4's downstream perplexity claim (PPL Δ ≤ +0.05 vs FP16) is end-to-end and runs once a real Q4_POLAR model GGUF is built through the integration flow described in fork-integration/README.md.

Architecture-specific kernels

The encoder stays scalar because it runs once per weight tensor at GGUF convert time, not in the inference hot path. The decoder + dot are SIMD-dispatched. The dispatcher itself (src/polar_dispatch.c) is #if-guarded by POLARQUANT_HAVE_AVX2 / POLARQUANT_HAVE_NEON (set by CMakeLists.txt) and falls back to the scalar reference when neither is available — useful for non-x86 / non-aarch64 dev hosts.

In-fork integration

The Apothic llama.cpp fork integration (registers Q4_POLAR=45, wires the type-traits dispatch) is staged in fork-integration/:

• quants-polar.{h,c} — drop-in for ggml/src/ggml-cpu/, scalar + AVX2 + NEON.
• *.patch — the deltas for ggml-common.h, ggml.h, ggml-cpu.c, ggml-quants.c, and ggml/src/ggml-cpu/CMakeLists.txt.
• fork-integration/README.md — the order of operations + the test-quantize-fns gate the vendor must run before we bump the pin in compile-libllama.mjs.

This standalone library remains the behavioural source of truth (it has the unit tests + parity gates). The in-fork file is a transcription with llama.cpp's own typedefs (ggml_fp16_t, block_q8_0). Math is identical; only the type names differ.

QJL residual sign vector parity

The Python reference uses torch.randint(seed=42), which is not portable across torch versions. Both the standalone library and the in-fork TU (fork-integration/quants-polar.c) use the deterministic C xorshift32 stream defined in src/polar_qjl.c. The GGUF converter at scripts/polarquant_to_gguf.py is responsible for recomputing the QJL bits against the same xorshift32 stream when packing the sidecar, so encoder + decoder + converter all agree on the same 128-bit sign vector.

Related files in this repo

• docs/porting/on-device-quantization-porting-plan.md -- the design spec this implementation follows ("PolarQuant block_q4_polar GGML quant type").
• packages/training/scripts/quantization/polarquant/polar_quant.py -- the bit-exact Python reference for the Lloyd-Max centroid solver, the Hadamard rotation, and the QJL residual.
• packages/training/scripts/quantization/polarquant_apply.py -- the orchestrator that produces the safetensors sidecar this converter consumes.
• packages/app-core/scripts/aosp/compile-libllama.mjs -- the toolchain that will build the libllama.so carrying the eventual Q4_POLAR kernel registration.