DeepSeek-V4

Deployment

For all methods and hardware platforms, see the official SGLang installation guide. The two paths below match the Python / Docker toggle in the command panel.

pip install --upgrade pip
pip install uv
uv pip install sglang

Then run the Python output of the command panel below in that environment.

For how to launch the image, see Install → Method 3: Using Docker. A minimal example (substitute the inner sglang serve ... with whatever the command generator below produces):

NVIDIA GPUs

A single image — lmsysorg/sglang:latest — covers the datacenter GPUs in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000).

docker pull lmsysorg/sglang:latest

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<your-hf-token>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    sglang serve <use args below>

AMD GPUs (ROCm)

AMD uses the daily-updated lmsysorg/sglang-rocm images. You can find the latest images on Docker Hub. We recommend the ROCm 7.2 version.

For example:

• MI355X → lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710
• MI300X → lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi30x-20260623

docker pull lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710

docker run \
    --device=/dev/kfd --device=/dev/dri \
    --group-add video \
    --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
    --shm-size 32g --ipc=host \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<your-hf-token>" \
    lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710 \
    sglang serve <use args below>

Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:

• Low-Latency — fastest reply for a single user. Pick for chat.
• Balanced — good speed with several users at once. Use for typical multi-user serving.
• High-Throughput — most tokens per second across many users. Best for batch jobs.

import { Deployment } from "/src/snippets/_deployment.jsx"; import { config } from "/src/snippets/configs/deepseek-ai/deepseek-v4.jsx"; import { benchmarks } from "/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx";

Playground

The Playground is where you experiment with SGLang features beyond the verified matrix. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.

The knobs come in two flavors:

• Built-in SGLang features — parallelism overrides (TP / CP / DP-Attention — DP-Attention's value is the DP degree, with off to disable), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, HiCache tiers, and HiSparse hierarchical sparse attention (decode-role only — the card appears once PD-Disagg mode is set to decode).
• DeepSeek-V4 specific features — MegaMoE W4A8 / W4A4 fused kernel (Blackwell only; Hopper SM90 uses a separate all-FP8 MegaMoE path — see Configuration Tips below).

Lines highlighted green are added by your overrides; lines with red strikethrough were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's Verified badge; any actual change flips it to Not Verified until the new configuration is run end-to-end and submitted back.

import { Playground } from "/src/snippets/_playground.jsx";

1. Model Introduction

DeepSeek-V4 is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an MIT License. The 0731 Flash refresh adds a checkpoint with a bundled DSpark draft head:

The Instruct checkpoints ship as FP4 MoE experts + FP8 attention / dense (one mixed-precision checkpoint covers every FP4-capable GPU). Matching *-Base repos ship pure FP8 mixed and are for further pre-training only — not for chat or tool calling.

Highlights: hybrid CSA + HCA attention (~27% inference FLOPs / ~10% KV cache vs DSv3.2 at 1M context), manifold-constrained hyper-connections (mHC), Muon optimizer, 1M-token context (32T+ pre-training tokens), three reasoning modes (Non-think / Think High / Think Max — use ≥ 384K context for Think Max), and a dedicated encoding_dsv4.encode_messages Python encoder + DSML tool-call grammar.

Recommended generation: temperature=1.0, top_p=1.0.

Resources: HuggingFace · Flash Official (0731) · Flash · Pro · ModelScope · Flash · Pro.

2. Configuration Tips

Concurrency & DeepEP dispatch buffer

Must hold: max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK. Violating it blows DeepEP's dispatch buffer at steady-state load (deep_ep.cpp:1105). When tuning, move --cuda-graph-max-bs-decode, --max-running-requests, and the env together.

The generator currently picks values on the conservative side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised.

Speculative decoding

The original Flash and Pro recipes use EAGLE. Flash Official (0731) uses the bundled DSpark draft head; see DSpark for its launch and tuning notes.

For the original Flash and Pro checkpoints:

• low-latency: steps=3, draft-tokens=4 → largest win at bs=1.
• balanced: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch.
• high-throughput: MTP disabled — at saturation the verify step costs more than it saves.
• MTP runs on the v2 speculative path.

Compressed attention state dtype

DeepSeek-V4 uses hybrid compressed attention for long-context efficiency. SGLANG_DSV4_COMPRESS_STATE_DTYPE controls the dtype of the C4 / C128 compressed attention state pools. Supported values are float32 / fp32 (default: float32) and bfloat16 / bf16. For BF16 on the offline compression path:

SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16 \
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4-Flash \
  <other args>

This BF16 setting applies only to the compressed attention state pools and reduces the GPU memory footprint of each compressed-state slot. It does not change model weight precision or the main KV cache dtype. With automatic pool sizing and no explicit capacity cap, the same memory budget holds more slots, and the startup log shows larger c4_state and c128_state pool sizes. Keep the default float32 setting for the most conservative behavior.

EPLB + Waterfill (Experimental)

For recorded/static EPLB reproduction, first record an expert-distribution file by following Capture expert selection distribution in MoE models. For reproduction runs, use the generated expert_distribution_recorder_*.pt as the initial expert location. Please checkout to latest main branch for this feature.

For non-PD reproduction, use:

--moe-a2a-backend deepep \
--deepep-mode auto \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill

For PD-Disagg reproduction, use normal mode on the prefill server and low_latency mode on the decode server. Add the same --init-expert-location flag to both commands:

# prefill
--moe-a2a-backend deepep \
--deepep-mode normal \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill

# decode
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill

You can also add --ep-num-redundant-experts and --eplb-algorithm to customize EPLB placement.

Waterfill also supports MegaMOE. Use --moe-a2a-backend megamoe --enable-waterfill to keep the MegaMOE backend while applying Waterfill to the fused shared expert slot.

FP4 Indexer (Experimental)

DeepSeek-V4 uses the default indexer path unless --enable-deepseek-v4-fp4-indexer is set. Enable this flag to use the experimental FP4 C4 indexer on SM100 GPUs with DeepGEMM FP4 indexer support. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial.

# Please use the latest main branch for this feature.
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4-Flash \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --enable-deepseek-v4-fp4-indexer

NVFP4 Hybrid Checkpoints

The nvidia/DeepSeek-V4-Pro-NVFP4 and nvidia/DeepSeek-V4-Flash-NVFP4 checkpoints quantize MoE experts to NVFP4 while keeping attention and dense layers in FP8. It requires --moe-runner-backend flashinfer_trtllm_routed which will be automatically selected if not provided.

sglang serve \
  --model-path nvidia/DeepSeek-V4-Pro-NVFP4 \
  --tp 8

or

sglang serve \
  --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \
  --tp 8

Requires Blackwell (SM100+). The MTP layer in this checkpoint stays MXFP4-packed and is routed through the Mxfp4FlashinferTrtllmMoEMethod path automatically.

Hopper (H100 / H200) note

Two options are available for running DeepSeek-V4 on Hopper:

• Original FP4 checkpoints — apply the W4A16 MoE kernels (Marlin) as the command generator picks for Hopper cells. This path works on both H100 and H200 and is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16).
• Converted FP8 checkpoints (H100 and H200 only) — pre-repackaged FP8 weights at sgl-project/DeepSeek-V4-Flash-FP8 and sgl-project/DeepSeek-V4-Pro-FP8 unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes).

On these FP8 checkpoints you can additionally enable the all-FP8 MegaMoE path on SM90 for higher long-context / large-decode throughput — see the SM90 (Hopper) FP8 MegaMoE note in Configuration Tips below.

PD-Disagg recipes on H200 may require docker run --privileged --ulimit memlock=-1 (or --device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK) so mooncake can discover the IB HCAs; without IB exposure mooncake silently falls back to TCP, which can lead to garbled KV transfer on large checkpoints.

RTX PRO 6000 (SM120 / Blackwell Desktop) note

RTX PRO 6000 (96 GB) runs Flash only with the FlashInfer MXFP4 MoE runner. V4-Pro doesn't fit on 8× 96 GB; the Deploy panel greys out unsupported recipes. HiCache and MegaMoE are not supported on RTX PRO 6000.

AMD (MI300X / MI355X) note

• Model checkpoints — for correct accuracy, the FP4 model uses the stock deepseek-ai/DeepSeek-V4-{Flash,Pro}, and the FP8 model uses the repackaged sgl-project/DeepSeek-V4-{Flash,Pro}-FP8.
• Supported models — MI300X supports DeepSeek-V4-Flash in FP8; MI355X supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
• TP / DP setting — both TP=4 and TP=8 are supported. At low concurrency we recommend TP-only; at high concurrency use TP + DP, which additionally needs --dp 8 --enable-dp-attention --enable-prefill-delayer --prefill-delayer-max-delay-ms 5000.
• MTP — speculative decoding is supported; add --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4.
• Kernels — uses the Unified KV attention and the flydsl MoE.

MegaMoE

MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput on MoE layers. To enable it, use the MegaMoE chip in the Playground below — the playground will swap --moe-a2a-backend deepep for --moe-a2a-backend megamoe and add the relevant env vars automatically.

Two variants are exposed:

• W4A8 — default MegaMoE kernel (FP4 weights, FP8 activations).
• W4A4 — adds SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1 and SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1 to run the custom W4A4 kernel (FP4 activations). Higher throughput with negligible accuracy drop (~89.5 GPQA on Pro).

Notes:

• The W4A8 / W4A4 variants above are Blackwell-only (B200 / B300 / GB200 / GB300). On Hopper (SM90, H100 / H200) use the all-FP8 MegaMoE path described below instead.
• MegaMoE is only wired into the high-throughput recipe on Blackwell (per sgl-project/sglang#26451). The chip is hidden on low-latency and balanced — switch to high-throughput to expose it.
• When running MegaMoE, don't set --moe-runner-backend manually.
• Adjust SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK based on your workload and memory usage. Setting higher number of tokens for MegaMoE requires more HBM space (recommended: 8320 for high-throughput).

SM90 (Hopper) FP8 MegaMoE (Experimental)

On SM90 (Hopper, H100 / H200), the all-FP8 MegaMoE path routes MoE through the DeepGEMM mega_moe runner for higher long-context / large-decode throughput on the FP8 checkpoints. Unlike the Blackwell W4A8 / W4A4 variants above, experts stay in FP8 — keep SGLANG_DSV4_FP4_EXPERTS=0. It requires a sgl-deep-gemm build with SM90 FP8 MegaMoE support. Please use the latest image for this feature.

Enable the MegaMoE path with --moe-a2a-backend megamoe — or equivalently set SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1, which auto-configures the same backend:

SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 \
SGLANG_DSV4_FP4_EXPERTS=0 \
sglang serve \
  --model-path sgl-project/DeepSeek-V4-Flash-FP8 \
  --tp 8 \
  --moe-a2a-backend megamoe \
  --chunked-prefill-size 4096

SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK caps the number of tokens the MegaMoE path processes per rank (i.e. per GPU); the MegaMoE path is only used for batches at or below this cap. The right value depends on your parallelism / token-split scheme, and larger values reserve more HBM.

GB300 PD-Disagg cross-pod MNNVL

On some GB300 clusters with cross-pod KV transfer over NVLink, mooncake may fail with nvlink_transport.cpp:497 Requested address ... not found!. If this happens, prepend MC_FORCE_MNNVL=1 NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1 to both prefill and decode sglang serve commands.

3. Advanced Usage

3.1 Reasoning

Enable the deepseek-v4 reasoning parser (toggle Reasoning Parser in the Parsers card of the Playground above) to separate thinking from the final answer into reasoning_content vs content.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    max_tokens=2048,
    extra_body={"chat_template_kwargs": {"thinking": True}},
    stream=True,
)

thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta

    if getattr(delta, "reasoning_content", None):
        if not thinking_started:
            print("=============== Thinking =================", flush=True)
            thinking_started = True
        has_thinking = True
        print(delta.reasoning_content, end="", flush=True)

    if delta.content:
        if has_thinking and not has_answer:
            print("\n=============== Content =================", flush=True)
            has_answer = True
        print(delta.content, end="", flush=True)

print()

We are asked: "What is 15% of 240?" This is a simple percentage problem. I need to provide a step-by-step solution. The user wants the solution explained step by step. I'll calculate 15% of 240: 0.15 * 240 = 36. I'll break it down into steps: understand what percent means, convert percentage to decimal or fraction, then multiply. I'll present the answer clearly.</think>To find 15% of 240, follow these steps:

**Step 1: Understand the meaning of percent**
"Percent" means "per hundred," so 15% means 15 out of every100, or \( \frac{15}{100} \).

**Step2: Convert the percentage to a decimal or fraction**
\( 15\% = \frac{15}{100} = 0.15 \)

**Step3: Multiply by the given number**
Multiply the decimal form by 240:
\( 0.15 \times 240 \)

**Step4: Perform the multiplication**
\( 0.15 \times 240 = 36 \)

**Answer:** 15% of 240 is **36**.

3.2 Tool Calling

Enable the deepseekv4 tool-call parser (toggle Tool Call Parser in the Parsers card of the Playground above) to surface structured tool calls via message.tool_calls.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "The city name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash",
    messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
    tools=tools,
    extra_body={"chat_template_kwargs": {"thinking": True}},
    stream=True,
)

thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta

    if getattr(delta, "reasoning_content", None):
        if not thinking_started:
            print("=============== Thinking =================", flush=True)
            thinking_started = True
        has_thinking = True
        print(delta.reasoning_content, end="", flush=True)

    if getattr(delta, "tool_calls", None):
        if has_thinking and thinking_started:
            print("\n=============== Content =================\n", flush=True)
            thinking_started = False
        for tool_call in delta.tool_calls:
            index = tool_call.index
            if index not in tool_calls_accumulator:
                tool_calls_accumulator[index] = {"name": None, "arguments": ""}
            if tool_call.function:
                if tool_call.function.name:
                    tool_calls_accumulator[index]["name"] = tool_call.function.name
                if tool_call.function.arguments:
                    tool_calls_accumulator[index]["arguments"] += tool_call.function.arguments

    if delta.content:
        print(delta.content, end="", flush=True)

for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()

The user wants to know the weather in Beijing. I'll use the get_weather function with Beijing as the location. I don't need to specify a unit, so I'll just use the default.</think>

<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="location" string="true">Beijing</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

3.3 HiCache (Hierarchical KV Caching)

HiCache enables multi-tier KV cache offloading (GPU → CPU → Storage), significantly expanding effective context capacity for long-context and multi-turn scenarios. Combined with UnifiedRadixTree, it provides intelligent prefix caching across all tiers.

To enable HiCache, open the HiCache card in the Playground above and flip Enable:

• L2 (GPU + CPU) — leave Storage on auto (default). Cold KV pages spill to CPU pinned memory only.
• L3 (GPU + CPU + Storage) — pick a Storage backend (file / mooncake / hf3fs / nixl); the Playground emits the canonical page_first_direct mem-layout + direct IO backend + wait_complete prefetch policy, matching the HiCache best-practices recipe.

For AMD devices,

• L2 (GPU + CPU) — leave Storage on auto (default). Cold KV pages spill to CPU pinned memory only. Use direct IO backend + page_first_direct or layer-first mem-layout.
• L3 (GPU + CPU + Storage) — pick a Storage backend (file); the Playground emits the canonical page_first_direct mem-layout + direct IO backend + wait_complete prefetch policy, matching the HiCache best-practices recipe.

The Write policy knob defaults to write_through (the upstream default); switch to write_back / write_through_selective to trade durability for write speed when the storage tier is slow.

For more details, see the HiCache documentation.

3.4 DSpark (Speculative Decoding)

Flash Official (0731) bundles a DSpark draft head in deepseek-ai/DeepSeek-V4-Flash-0731. The target and draft weights therefore come from the same checkpoint: enable DSpark with --speculative-algorithm DSPARK and do not set a separate --speculative-draft-model-path.

Unlike the EAGLE recipes for the original Flash and Pro checkpoints, this recipe omits --speculative-num-steps, --speculative-eagle-topk, and --speculative-num-draft-tokens. SGLang reads the DSpark shape from the checkpoint.

The verified 4×GB300 FP4 low-latency command is:

sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algorithm DSPARK \
  --mem-fraction-static 0.90 \
  --chunked-prefill-size 4096 \
  --swa-full-tokens-ratio 0.1 \
  --host 0.0.0.0 \
  --port 30000

Keep --mem-fraction-static 0.90 on this topology to leave enough headroom for the batch-256 verify graph. The first cold start can take 10–15 minutes while FlashInfer autotunes and SGLang captures the draft and verify graphs; later starts reuse the cache. This path is verified end-to-end on 4×GB300 with SGLang v0.5.16.

Tune proposed draft tokens. --speculative-dspark-block-size N asks DSpark to propose N tokens per step; the target verifies a window of N + 1. If the flag is omitted, SGLang reads the value from the checkpoint. The current 0731 checkpoint resolves to five proposed tokens, which is the verified default. Use the DSpark Proposed Draft Tokens slider in the Playground to sweep one through five.

Larger blocks can improve decode latency when acceptance stays high, but they also increase verification work and graph memory. Start from the checkpoint default, then sweep downward under the real prompt-length and concurrency distribution. The gain is usually largest for short interactive traffic and narrows as prefill dominates. Track P50/P99 TTFT and TPOT, total throughput, accepted length, GPU memory, and stop rate rather than choosing from acceptance alone.

For every candidate, compare with the same recipe without --speculative-algorithm DSPARK. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each bench_serving leg its own --flush-cache. Leave --speculative-draft-attention-backend unset unless a separate profiling run justifies an override.

DSpark currently requires CUDA, pp_size == 1, and DP Attention disabled. It is not compatible with PD disaggregation on current SGLang releases; turn DSpark off before selecting a prefill or decode role. The DP-Attention and MI355X Flash Official recipes therefore run target-only. If a larger draft block or concurrency causes graph-capture OOM, lower --mem-fraction-static, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.