doc/en/kt-kernel/Kimi-K2-Thinking-Native.md
This tutorial demonstrates how to run Kimi-K2 model inference using SGLang integrated with KT-Kernel for CPU-GPU heterogeneous inference. This setup enables efficient deployment of large MoE models by offloading experts to CPU.
Minimum Configuration:
Tested Configuration:
Before starting, ensure you have:
# Option A: One-click install (from ktransformers root)
./install.sh
# Option B: pip install
pip install sglang-kt
pip install huggingface-hub
# Create a directory for models
mkdir -p /path/to/models
cd /path/to/models
# Download Kimi-K2-Thinking (INT4 for both CPU and GPU)
huggingface-cli download moonshotai/Kimi-K2-Thinking \
--local-dir /path/to/kimi-k2-thinking
Note: Replace /path/to/models with your actual storage path throughout this tutorial.
Start the SGLang server with KT-Kernel integration for CPU-GPU heterogeneous inference.
python -m sglang.launch_server \
--host 0.0.0.0 \
--port 30001 \
--model /path/to/kimi-k2-thinking \
--kt-weight-path /path/to/kimi-k2-thinking \
--kt-cpuinfer 96 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 8 \
--kt-method RAWINT4 \
--kt-gpu-prefill-token-threshold 400 \
--kt-max-deferred-experts-per-token 1 \
--trust-remote-code \
--mem-fraction-static 0.94 \
--served-model-name Kimi-K2-Thinking \
--enable-mixed-chunk \
--tensor-parallel-size 2 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--chunked-prefill-size 65536 \
--max-total-tokens 65536 \
--attention-backend flashinfer
It takes about 2~3 minutes to start the server.
See KT-Kernel Parameters for detailed parameter tuning guidelines.
| Parameter | Description |
|---|---|
--kt-method RAWINT4 | CPU and GPU use the same INT4 weight. Set --model and --kt-weight-path to the same directory. |
--kt-num-gpu-experts | Number of experts kept on GPU for decoding. |
--kt-gpu-prefill-token-threshold | Token count threshold for prefill strategy. Below: hybrid CPU+GPU. Above: layerwise GPU prefill. |
--chunked-prefill-size | Maximum tokens per prefill batch. |
--max-total-tokens | Maximum total tokens in KV cache. |
--kt-gpu-prefill-token-thresholdThis parameter controls the prefill strategy:
Layerwise prefill requires extra VRAM (~9GB + incremental cost with prefill length). If you encounter OOM, adjust these parameters based on your use case and hardware (refer to the recommended parameters table below):
| Parameter | VRAM Impact |
|---|---|
--kt-num-gpu-experts | Reduces expert weight VRAM usage |
--chunked-prefill-size | Reduces prefill extra VRAM allocation |
--max-total-tokens | Reduces KV cache VRAM usage |
Tip: Test with an input of length chunked-prefill-size to verify your configuration won't OOM during prefill.
| GPU Config | kt-num-gpu-experts | max-total-tokens | chunked-prefill-size |
|---|---|---|---|
| 1x RTX 4090 (48GB) | 0 | 30000 | 30000 |
| 2x RTX 4090 (48GB) | 8 | 65536 | 65536 |
| 4x RTX 4090 (48GB) | 30 | 80000 | 65536 |
| 8x RTX 4090 (48GB) | 80 | 100000 | 65536 |
Tip: If your prefill and total length requirements are low (e.g., processing short texts), you can reduce max-total-tokens and chunked-prefill-size to free up VRAM for a larger kt-num-gpu-experts, which improves decode performance.
The following prefill throughput (tokens/s) benchmarks were measured with single concurrency:
| GPU Config | 2048 tokens | 8192 tokens | 32768 tokens |
|---|---|---|---|
| 1x RTX 4090 (48GB) | 53 | 184 | 290* |
| 2x RTX 4090 (48GB) | 85 | 294 | 529 |
| 4x RTX 4090 (48GB) | 118 | 415 | 818 |
| 8x RTX 4090 (48GB) | 130 | 435 | 1055 |
Once the server is running, you can send inference requests using the OpenAI-compatible API.
curl -s http://localhost:30001/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Kimi-K2-Thinking",
"stream": false,
"messages": [
{"role": "user", "content": "hi"}
]
}'
{
"id": "cd0905562bf44513947284f80cc5634b",
"object": "chat.completion",
"created": 1764921457,
"model": "Kimi-K2-Thinking",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": " <think> The user says \"hi\". This is a very simple greeting. I should respond in a friendly and helpful manner. Since I'm an AI assistant, I should be professional but approachable.\n\nPossible responses:\n1. \"Hello! How can I help you today?\"\n2. \"Hi there! What can I do for you?\"\n3. \"Hello! It's nice to hear from you. What would you like to talk about?\"\n4. \"Hi! I'm here to assist you with any questions you might have.\"\n\nI think option 1 is the most standard and professional. It's direct, friendly, and opens the door for the user to ask their question. I should keep it concise.\n\nLet me go with: \"Hello! How can I help you today?\" </think> Hello! How can I help you today?",
"reasoning_content": null,
"tool_calls": null
},
"logprobs": null,
"finish_reason": "stop",
"matched_stop": 163586
}
],
"usage": {
"prompt_tokens": 26,
"total_tokens": 189,
"completion_tokens": 163,
"prompt_tokens_details": null,
"reasoning_tokens": 0
},
"metadata": {
"weight_version": "default"
}
}
Add the following parameters to the SGLang launch command above to enable tool calling support:
--tool-call-parser kimi_k2 --reasoning-parser kimi_k2
With these parameters enabled, you can use claude-code-router to connect Kimi-K2-Thinking as a local backend for Claude Code.