HiSparse: Hierarchical Sparse Attention

HiSparse reduces per-request GPU memory consumption during the decode phase by maintaining only a small "hot" KV buffer on GPU while keeping complete KV data in CPU pinned memory. Combined with PD disaggregation, it enables significantly higher decode concurrency.

Prerequisites: HiSparse only works with models that use DeepSeek Sparse Attention (DSA) architectures (e.g., DeepSeek-V3.2, GLM-5). These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires PD disaggregation mode and is enabled on the decode instance only.

Why HiSparse?

In long-context LLM inference, each decoding request holds a full-length KV cache on GPU, limiting the number of concurrent requests a decode instance can serve. HiSparse addresses this by:

• Reducing GPU memory per request: Each request occupies only a fixed-size device buffer (e.g., 4KB tokens) instead of the full sequence length.
• On-demand swap-in: A CUDA kernel dynamically loads the top-k most relevant KV entries from host memory based on attention scores.
• Transparent to prefill: HiSparse is entirely a decode-side optimization; the prefill instance requires no changes.

Design Overview

Decode Workflow

Each decode step follows this flow:

1. Forward decode — generate the next token
2. Top-k selection — select the most relevant token positions via attention scores
3. Swap-in — the CUDA kernel loads top-k KV entries from host to device buffer:
   • Short sequences (seq_len ≤ device_buffer_size): fast path, all KV already in buffer
   • Long sequences: hit detection → LRU reordering → miss handling (host → device copy)
4. Decode attention — compute attention using the top-k device locations
5. Eager backup — asynchronously copy the previous token's KV from device to host

PD Disaggregation Integration (Direct-to-Host)

In PD disaggregation mode, the prefill instance transfers KV cache directly into the decode instance's host pool via RDMA, bypassing the GPU entirely on the decode side. This eliminates the transient GPU memory spike during KV transfer and removes the staging DMA step.

Prefill GPU  ──RDMA──▶  Decode Host Pool (CPU pinned memory)
                              │
                              ▼
                     alloc device buffer (4KB)
                              │
                              ▼
                     swap-in kernel (on-demand top-k)

Server Arguments

HiSparse Config Parameters

Pass as a JSON string via --hisparse-config:

Example: --hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10}'

Deployment

HiSparse currently requires PD disaggregation mode and is enabled only on the decode instance.

Prefill Instance

python3 -m sglang.launch_server \
    --model-path /path/to/model \
    --trust-remote-code \
    --port 8000 --host 0.0.0.0 \
    --context-length 81920 \
    --chunked-prefill-size 65536 \
    --tp-size 8 --dp-size 8 --enable-dp-attention \
    --mem-fraction-static 0.85 \
    --disaggregation-mode prefill \
    --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
    --nnodes 1 --node-rank 0

Decode Instance (with HiSparse)

python3 -m sglang.launch_server \
    --model-path /path/to/model \
    --trust-remote-code \
    --port 8000 --host 0.0.0.0 \
    --context-length 81920 \
    --tp-size 8 --dp-size 8 --enable-dp-attention \
    --mem-fraction-static 0.85 \
    --kv-cache-dtype bfloat16 \
    --nsa-decode-backend flashmla_sparse \
    --disaggregation-mode decode \
    --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
    --dist-init-addr 127.0.0.1:5757 \
    --nnodes 1 --node-rank 0 \
    --enable-hisparse \
    --hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10}'

Benchmark

python3 -m sglang.bench_serving \
    --backend sglang \
    --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \
    --dataset-name random \
    --random-input 40000 \
    --random-output 20000 \
    --num-prompts 200 \
    --max-concurrency 200 \
    --request-rate 40 \
    --random-range-ratio 1.0 \
    --host 127.0.0.1 \
    --port 20000 \
    --model /path/to/model \
    --flush-cache \

Key Notes

• The prefill instance does not need --enable-hisparse; it is unaware of HiSparse.
• On the decode instance, the following flags are required for HiSparse:
  • --kv-cache-dtype bfloat16 — currently only bfloat16 KV cache is supported (more dtypes planned).
  • --nsa-decode-backend flashmla_sparse — currently only flashmla_sparse backend is supported.
  • --enable-hisparse — enables HiSparse.
  • --hisparse-config — HiSparse configuration (top_k, device_buffer_size, host_to_device_ratio).
    • host_to_device_ratio should be configured based on the host machine's available memory. For example:
      • ~1 TB host memory → host_to_device_ratio: 5
      • ~2 TB host memory → host_to_device_ratio: 10

Acknowledgments

We would like to thank the SGLang team and community for the implementation and generous support, especially Zhiqiang Xie, Zhangheng Huang, Tingwei Huang, Shangming Cai, Teng Ma, and many others. We also thank the Alibaba Cloud TairKVCache team and the AntGroup SCT Inference team for their valuable contributions.