docs/source/en/model_doc/deepseek_v4.md
This model was released on {release_date} and added to Hugging Face Transformers on 2026-05-02.
DeepSeek-V4 is the next-generation MoE language model from DeepSeek (paper). The architecture replaces DeepSeek-V3's Multi-head Latent Attention (MLA) with a hybrid local + long-range design, swaps residual connections for Manifold-Constrained Hyper-Connections (mHC), and bootstraps the first few MoE layers with a static token-id → expert-id hash table.
This implementation covers DeepSeek-V4-Flash, DeepSeek-V4-Pro, and their -Base pretrained siblings. All four
share the same architecture; they differ only in width / depth / expert count and weights.
Each decoder block is one of three attention types, dispatched by config.layer_types[i]:
"sliding_attention"): only the local window of sliding_window tokens, no
long-range branch. Matches V3's "Full Attention" style for the bootstrap layers."compressed_sparse_attention", CSA — paper §2.3.1): a low-compression
pool (compress_rate_csa, default m=4) with overlapping windows, plus a Lightning Indexer (eqs. 13–17)
that scores queries against the pool and gathers the top index_topk blocks per query before they reach core
attention."heavily_compressed_attention", HCA — paper §2.3.2): a high-compression
pool (compress_rate_hca, default m'=128) with non-overlapping windows. No indexer — every pooled entry
contributes to attention.All three types share the same backbone:
num_key_value_heads = 1; kv_proj produces a single KV head and the same
tensor is read as both key and value.qk_rope_head_dim = head_dim * partial_rotary_factor channels of each head. The same rotation is applied with
position -i to the attention output's rope slice (eq. 26) so the contribution of each KV entry stays a function
of the relative distance to the query.o_groups head-groups → o_lora_rank
per group → hidden_size, computed by [DeepseekV4GroupedLinear] (o_a_proj) followed by o_b_proj. Cuts the
per-token cost of the wide attention output without losing expressivity.sliding_window ("Additional Branch of Sliding Window Attention",
§2.3.1) preserves local fine-grained dependencies; the long-range compressor's output is concatenated with this
branch's KVs before core attention.Residual connections are replaced by mHC (Xie et al., 2026): hc_mult parallel residual streams kept in shape
[B, S, hc_mult, D] throughout each block. Two [DeepseekV4HyperConnection] modules — attn_hc and ffn_hc — mix
streams in and out around the attention / MLP sublayers via a (pre, post, comb) triplet. The comb matrix is a
doubly-stochastic projection produced by hc_sinkhorn_iters Sinkhorn–Knopp iterations on the manifold, making
signal propagation non-expansive across deep stacks. A final [DeepseekV4HyperHead] collapses the hc_mult
streams down to a single sequence before the model norm.
Routing is configured per layer by config.mlp_layer_types, with values from {"hash_moe", "moe"}:
"hash_moe": expert indices come from a frozen tid2eid[input_ids] lookup populated from the V4 checkpoint.
The learned gate weight still produces the per-expert scores that weight the selected experts; only
which-experts is static. Used for the first few bootstrap layers (default 3, override via legacy
num_hash_layers)."moe": standard top-k routed MoE. The expert affinity uses Sqrt(Softplus(·)) instead of V3's Sigmoid
("we change the activation function that computes the affinity scores from Sigmoid(·) into Sqrt(Softplus(·))",
paper §2.1), and V3's n_group / topk_group constraint is dropped. The auxiliary-loss-free strategy
(DeepSeek's noaux_tc) is preserved via the e_score_correction_bias buffer that biases the top-k argmax
without flowing gradients.Routed experts use a clamped SwiGLU (gate.clamp(max=swiglu_limit), up.clamp(min=-swiglu_limit, max=swiglu_limit),
then act_fn(gate) * up) on top of the standard Mixtral [num_experts, 2 * moe_intermediate_size, hidden_size]
expert weight layout. A single shared expert (a plain SwiGLU MLP at moe_intermediate_size width) runs in parallel
on every token.
Each DeepseekV4Attention layer extends the standard sliding-window-causal mask along the key axis with a
block_bias returned by its compressor, then feeds the concatenated mask to eager_attention_forward. The
sliding-section (left, [S, S]) is the same for every layer type; the compressor-section (right) differs by
layer type and is the actual "novel" piece introduced by V4.
The diagrams below were produced with a tiny config (sliding_window=8, CSA m=4, HCA m'=8, index_topk=2)
on a 16-token input so the full per-layer-type mask fits on screen. Green = the query/key diagonal in the
sliding section, dark = a visible standard KV position, light = masked, amber = a compressor / indexer slot
the query is allowed to attend to. Columns past the dashed line are appended by the compressor via
cat([sliding_causal_mask, block_bias], dim=-1).
Sliding-only layer ("sliding_attention"). No compressor, no right-padding — the mask is the plain
sliding-window-causal mask of shape [S, S] (window = 8). For i ≥ window the lower-left triangle is cut
off, recovering the local-only attention pattern.
CSA layer ("compressed_sparse_attention"). The compressor flattens its per-query gathered output to
[B, 1, S·k, D] and right-pads the mask by S·k columns. For query t, only the k slots at columns
[S + t·k, S + (t+1)·k) carry the indexer's picks; all other compressor columns are -inf. Queries before
the first window has closed (t < m − 1) get nothing — the indexer's -1 sentinel propagates straight to
the mask. As t grows, more compressed entries are ready and the indexer can fill all k slots.
HCA layer ("heavily_compressed_attention"). No indexer — every cached compressed entry is potentially
visible. Right-padded by T_total = entry_count["compressor"] columns. Query t may only see entry w once
its source window has closed, i.e. w < (t + 1) // m. With m=8 here, entries 0 (covers positions 0..7)
and 1 (covers 8..15) only become visible at t ≥ 7 and t ≥ 15 respectively.
These diagrams are reproducible end-to-end via:
python docs/source/en/imgs/deepseek_v4/visualize_attention_masks.py \
--svg docs/source/en/imgs/deepseek_v4
The script runs a forward pass on this tiny config, wraps each attention layer to capture the exact
post-cat([attention_mask, block_bias]) mask, remaps CSA's [S, S·k] flat-slot mask back to a
[S, T_entries] entry-visibility view (so each C_w column is a compressed entry, not a gather slot),
and writes the three SVGs above. It also prints an ANSI grid to stdout for quick terminal inspection and
dumps the indexer's per-query top-k picks so warm-up sentinels and pick choices are auditable.
Each non-sliding attention block needs to thread compressor / indexer state across forward calls. V4 ships two
cache layer types that auto-register with LAYER_TYPE_CACHE_MAPPING:
DeepseekV4HCACache: sliding-window K=V + HCA compressor buffer / pool / count (no overlap, no indexer).DeepseekV4CSACache: sliding-window K=V + CSA compressor (with overlap state) + parallel indexer
buffer / pool / count / overlap at index_head_dim.DynamicCache(config=…) builds the right cache layer per config.layer_types[i].
[[autodoc]] DeepseekV4Config
[[autodoc]] DeepseekV4Model - forward
[[autodoc]] DeepseekV4ForCausalLM - forward