doc/en/kt-kernel/Native-Precision-Tutorial.md
This tutorial demonstrates how to run native precision MoE model inference using SGLang integrated with KT-Kernel. KTransformers v0.5.1+ supports multiple native precision formats, enabling efficient inference across various model architectures.
KTransformers supports multiple native precision formats via the --kt-method parameter:
| kt-method | Precision Format | Description | Instruction Set |
|---|---|---|---|
BF16 | BF16 Native | Zero precision loss, original weights | AMX + AVX512 |
FP8 | FP8 Blockwise | Block-wise scale quantization | AVX512 |
FP8_PERCHANNEL | FP8 Per-Channel | Per-channel scale quantization | AVX512 |
RAWINT4 | INT4 Native | Same INT4 weights for CPU and GPU | AVX512 |
| Model(sorted by lexicographical order) | kt-method | Precision |
|---|---|---|
| DeepSeek-V3/R1/V3.2 | FP8 | FP8 |
| GLM-4.7 | FP8_PERCHANNEL, BF16 | FP8, BF16 |
| Kimi-K2-Thinking | RAWINT4 | INT4 Native |
| MiniMax-M2/M2.1 | FP8 | FP8 |
| Qwen3-235B-A22B | FP8, BF16 | FP8, BF16 |
| Qwen3-30-A3B | FP8, BF16 | FP8, BF16 |
| Qwen3-Next-80B-A3B | FP8, BF16 | FP8, BF16 |
Minimum Configuration:
Recommended Configuration:
Before starting, ensure you have:
SGLang installed
Install the kvcache-ai fork of SGLang (one of):
# Option A: One-click install (from ktransformers root)
./install.sh
# Option B: pip install
pip install sglang-kt
KT-Kernel installed
Follow the kt-kernel installation guide:
git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers/kt-kernel
./install.sh
Verify the installation:
kt version
CUDA toolkit - CUDA 12.0+ recommended
Hugging Face CLI - For downloading models:
pip install -U huggingface-hub
For now, only MiniMax-M2/M2.1, DeepSeek-V3/R1-0528/V3.2, Kimi-K2-Thinking can run with kt-cli.
DeepSeek-V3.2
kt run V3.2 --kt-enable-dynamic-expert-update
GLM-4.7
python -m sglang.launch_server \
--host 0.0.0.0 \
--port 30000 \
--model /path/to/GLM-4.7/ \
--kt-weight-path /path/to/GLM-4.7/ \
--kt-cpuinfer 100 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 15 \
--kt-method BF16 \
--kt-enable-dynamic-expert-update \
--attention-backend flashinfer \
--mem-fraction-static 0.80 \
--chunked-prefill-size 16384 \
--max-running-requests 2 \
--max-total-tokens 32768 \
--trust-remote-code \
--served-model-name GLM-4.7 \
--enable-mixed-chunk \
--tensor-parallel-size 8 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--watchdog-timeout 3000 \
--kt-gpu-prefill-token-threshold 1024
GLM-4.7-FP8
python -m sglang.launch_server \
--host 0.0.0.0 \
--port 30000 \
--model /path/to/GLM-4.7-FP8/ \
--kt-weight-path /path/to/GLM-4.7-FP8/ \
--kt-cpuinfer 100 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 80 \
--kt-method FP8_PERCHANNEL \
--kt-enable-dynamic-expert-update \
--attention-backend flashinfer \
--mem-fraction-static 0.75 \
--chunked-prefill-size 16384 \
--max-running-requests 4 \
--max-total-tokens 100000 \
--trust-remote-code \
--served-model-name GLM-4.7 \
--enable-mixed-chunk \
--tensor-parallel-size 8 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--watchdog-timeout 3000 \
--fp8-gemm-backend triton \
--kt-gpu-prefill-token-threshold 2048
Qwen3-235B-A22B
python -m sglang.launch_server \
--host 0.0.0.0 \
--port 30000 \
--model /path/to/Qwen3-235B-A22B \
--kt-weight-path /path/to/Qwen3-235B-A22B \
--kt-cpuinfer 100 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 20 \
--kt-method FP8 \
--kt-enable-dynamic-expert-update \
--kt-expert-placement-strategy uniform \
--attention-backend flashinfer \
--mem-fraction-static 0.80 \
--chunked-prefill-size 16384 \
--max-running-requests 4 \
--max-total-tokens 100000 \
--trust-remote-code \
--served-model-name Qwen3-235B-A22B \
--enable-mixed-chunk \
--tensor-parallel-size 8 \
--enable-p2p-check \
--kt-gpu-prefill-token-threshold 2048
| Parameter | Description |
|---|---|
--kt-method | Precision format: BF16, FP8_PERCHANNEL, FP8, RAWINT4, AMXINT4 |
--kt-cpuinfer | Number of CPU inference threads (set to ~90% of physical cores) |
--kt-threadpool-count | Number of thread pools (set to NUMA node count) |
--kt-num-gpu-experts | Number of experts kept on GPU per layer |
--kt-enable-dynamic-expert-update | Enable dynamic expert placement updates during Layerwise Prefill |
--kt-expert-placement-strategy | Expert placement strategy |
--kt-gpu-prefill-token-threshold | Token threshold for triggering Layerwise Prefill |
--chunked-prefill-size | Maximum tokens per prefill batch |
--max-total-tokens | Maximum total tokens in KV cache |
Once the server is running (default: http://localhost:30000), you can interact with the model:
kt chat
This opens an interactive terminal chat session. Type your messages and press Enter to send. Use Ctrl+C to exit.
The server exposes an OpenAI-compatible API at http://localhost:30000/v1.
curl example (streaming):
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "MODEL_NAME",
"messages": [{"role": "user", "content": "Hello! What can you help me with?"}],
"stream": true
}'
Python example:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="none")
response = client.chat.completions.create(
model="MODEL_NAME",
messages=[{"role": "user", "content": "Explain quantum computing in simple terms."}],
stream=True
)
for chunk in response:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
See CPU-GPU Expert Scheduling Tutorial for details.
KTransformers implements an adaptive dual prefill mechanism based on input token count:
| Mode | Trigger Condition | Computation |
|---|---|---|
| CPU-GPU Hybrid | num_tokens < threshold | GPU + CPU |
| Layerwise Prefill | num_tokens >= threshold | GPU (CPU weights transferred to GPU) |
Set the kt-gpu-prefill-token-threshold parameter for best performance based on your workload.
Layerwise prefill requires extra VRAM. If you encounter OOM, adjust these parameters:
| Parameter | VRAM Impact |
|---|---|
--kt-num-gpu-experts | Reduces expert weight VRAM usage |
--chunked-prefill-size | Reduces prefill extra VRAM allocation |
--max-total-tokens | Reduces KV cache VRAM usage |
--mem-fraction-static | Adjusts static memory fraction |
Tips:
chunked-prefill-size to verify configuration--kt-num-gpu-experts if GPU memory is limited--enable-p2p-check is enabled--fp8-gemm-backend triton may be required