ContextQMD
Back to Ktransformers

Running DeepSeek-V4-Flash with SGLang and KT-Kernel

doc/en/DeepSeek-V4-Flash.md

0.7.09.0 KB
Original Source

Running DeepSeek-V4-Flash with SGLang and KT-Kernel

This tutorial demonstrates how to run DeepSeek-V4-Flash model inference using SGLang integrated with KT-Kernel for CPU-GPU heterogeneous inference. The hybrid path splits MXFP4 routed experts between CPU (KT-Kernel cpuinfer) and GPU (sglang kt-num-gpu-experts), enabling deployment on consumer-grade hardware.

Table of Contents

Hardware Requirements

Validated Configuration (this tutorial):

  • GPU: 1× NVIDIA RTX 5090 (32GB VRAM, SM_120)
  • CPU: x86 CPU with AVX2 and FMA; AVX512/AMX improves throughput but is not required
  • RAM: ≥200GB system memory
  • Storage: ~340GB for model weights

Supported consumer GPU architectures:

ArchCompute CapMXFP4 MoENSA sparse MLAValidated
Consumer Blackwell (RTX 5090)SM_120triton_kernelsTriton fallback✓
Ada Lovelace (RTX 4090)SM_89triton_kernelsTriton fallback✓
Ampere (RTX 3090)SM_86triton_kernelsTriton fallback✓

Docker Quick Start

Use Docker when you want to run the prebuilt environment without cloning the repository or compiling from source. Install the NVIDIA driver, Docker, and NVIDIA Container Toolkit on the host first.

Pull the image from Docker Hub:

bash
sudo docker pull approachingai/ktransformers:DSV4-specific

After downloading the model, enter the model directory and start the service:

bash
cd /path/to/DeepSeek-V4-Flash-0731

sudo docker run --gpus all \
  --ipc host \
  --cap-add SYS_NICE \
  -p 30000:30000 \
  -v "$PWD":/model:ro \
  approachingai/ktransformers:DSV4-specific

The server listens on http://localhost:30000 and exposes an OpenAI-compatible API. After the startup logs report readiness, verify it with:

bash
curl http://localhost:30000/v1/models

Docker Runtime Configuration

The image exposes the following environment variables. The launch command above uses these defaults:

VariableDefaultDescription
CUDA_VISIBLE_DEVICESall GPUs exposed by DockerCUDA ordinals visible to the container; use with TP.
TP1SGLang tensor-parallel degree; it must not exceed the visible GPU count.
MEM_FRACTION0.90Fraction of GPU memory reserved by the server.
CHUNKED_PREFILL_SIZE4096Maximum token count in a prefill chunk.
CONTEXT_LENGTH16384Maximum model context length.
MAX_RUNNING_REQUESTS2Maximum concurrent running requests.
KT_GPU_PREFILL_TOKEN_THRESHOLD2048Layerwise GPU-prefill threshold. Set 0 to disable it.
SWA_FULL_TOKENS_RATIO0.4SWA KV-cache ratio sized for the default 4,096-token prefill chunk.

Layerwise prefill is enabled by default for prompts of 2,048 tokens or longer. Its slots are allocated lazily by the first qualifying request, so that request can have a one-time setup cost. Set KT_GPU_PREFILL_TOKEN_THRESHOLD=0 only when you intentionally want to disable layerwise prefill to reduce peak VRAM use.

For tensor parallelism, select the CUDA ordinals and set a matching degree. For example, this starts two ranks on GPUs 0 and 1:

bash
sudo docker run --gpus all \
  --ipc host \
  --cap-add SYS_NICE \
  -p 30000:30000 \
  -e CUDA_VISIBLE_DEVICES=0,1 \
  -e TP=2 \
  -v "$PWD":/model:ro \
  approachingai/ktransformers:DSV4-specific

Prerequisites

The remaining sections describe the native source installation path. Docker users can skip them.

  1. KT-Kernel installed:

    bash
    git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule update --init --recursive
cd kt-kernel && ./install.sh

  2. SGLang installed (kvcache-ai fork):

    bash
    ./install.sh   # from ktransformers root

  3. CUDA 12.8+ and flashinfer ≥ 0.6.9 (flashinfer-python and flashinfer-cubin must be the same version):

    bash
    pip install --upgrade flashinfer-python flashinfer-cubin

    This upgrade is required (even though sglang-kt pins flashinfer_python==0.6.3) because V4-Flash's MXFP4 MoE module imports mxfp8_quantize, trtllm_fp4_block_scale_routed_moe, etc., which only exist in flashinfer ≥ 0.6.9.

  4. transformers==4.57.1 (V4-Flash is incompatible with the 5.x series):

    bash
    pip install "transformers==4.57.1"

    transformers 5.x adds default-valued fields to PretrainedConfig that make DeepSeekV4Config's dataclass declaration raise TypeError: non-default argument 'quantization_config' follows default argument at import time. sglang-kt's pyproject does not pin transformers, so a fresh pip install will pull the latest 5.x and break server startup; pinning explicitly to 4.57.1 is required until the upstream fix lands.

  5. tilelang (manual install — required for the NSA sparse-MLA tilelang indexer path used on non-Hopper GPUs):

    bash
    pip install tilelang "apache-tvm-ffi<0.1.12"

    sglang-kt's pyproject does not declare tilelang as a dependency, so pip install ./python[all] will not pull it in. Validated with tilelang==0.1.8.

    Note: Constrain apache-tvm-ffi<0.1.12. The standalone apache-tvm-ffi 0.1.12 wheel collides with the TVM FFI runtime bundled inside tilelang, so importing tilelang aborts with TypeAttr __ffi_repr__ is already registered for type index 130 and the SGLang scheduler dies on startup. apache-tvm-ffi==0.1.11 does not register the conflicting attribute and starts cleanly; pin until the upstream duplicate-registration fix lands.

Step 1: Download Model Weights

Download the model from Hugging Face:

bash
mkdir -p /path/to/models
huggingface-cli download deepseek-ai/DeepSeek-V4-Flash-0731 \
  --local-dir /path/to/models/DeepSeek-V4-Flash-0731

Step 2: Launch SGLang Server

Launch Command (Single RTX 5090 Example)

bash
export FLASHINFER_CUDA_ARCH_LIST=12.0a
export TORCH_CUDA_ARCH_LIST="12.0+PTX"

python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/models/DeepSeek-V4-Flash-0731 \
  --kt-weight-path /path/to/models/DeepSeek-V4-Flash-0731 \
  --kt-method MXFP4 \
  --kt-num-gpu-experts 10 \
  --kt-cpuinfer 60 \
  --kt-threadpool-count 2 \
  --kt-gpu-prefill-token-threshold 4096 \
  --kt-enable-dynamic-expert-update \
  --tensor-parallel-size 1 \
  --context-length 16384 \
  --attention-backend flashinfer \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 2048 \
  --max-prefill-tokens 2048 \
  --max-running-requests 2 \
  --watchdog-timeout 1200 \
  --disable-shared-experts-fusion \
  --trust-remote-code \
  --cuda-graph-bs 1 \
  --cuda-graph-max-bs 1 \
  --disable-radix-cache \
  --skip-server-warmup

Decode throughput: 20+ tok/s on a single RTX 5090.

It takes about 4-5 minutes to start the server (weight load + CUDA Graph capture).

See KT-Kernel Parameters for detailed parameter tuning guidelines.

Optional: Enable MTP (Multi-Token Prediction) Speculative Decoding

V4-Flash ships a NextN draft head that can be run as EAGLE-style speculative decoding for ~1.2× throughput on single-request decode (validated 26.5 → 32.74 tok/s on 8× RTX 5090, 90% accept rate at chain depth 1).

Append the following flags to the launch command above:

bash
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-moe-runner-backend auto \

Step 3: Send Inference Requests

Decode

bash
curl -s -X POST http://127.0.0.1:30000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Explain quantum computing in detail:",
    "sampling_params": {"temperature": 0.0, "max_new_tokens": 256}
  }'

Interactive Chat (kt chat)

The kt CLI ships with an OpenAI-compatible chat client that talks to the SGLang server's /v1/chat/completions endpoint:

bash
kt chat --host 127.0.0.1 --port 30000 --temperature 0.7 --max-tokens 2048

See KT-Kernel Parameters for the complete parameter reference.