Dots3-Note

Deployment

dots3.note support is in SGLang PR #33829. Until that PR is included in a tagged SGLang release, install from a build that contains the PR.

pip install -U uv
uv venv --python 3.12 && source .venv/bin/activate

git clone https://github.com/sgl-project/sglang.git
cd sglang
git fetch origin pull/33829/head && git checkout FETCH_HEAD
uv pip install -e python

Then run the Python output of the command panel below in that environment.

docker pull lmsysorg/sglang:dev-dots3-note

This image packages SGLang with the dots3.note support from PR #33829 and is the recommended way to deploy until the PR lands in a tagged SGLang release — it saves you from building the branch yourself.

For how to launch the image, see Install → Method 3: Using Docker. Substitute the inner sglang serve ... with what the command generator below produces.

Pick the hardware and the checkpoint precision. The recipe runs on a single 8-GPU Hopper node with DP8 attention × TP8 × EP8 and DeepEP as the MoE all-to-all transport. Blackwell is not supported yet.

Precision — selects the MoE path, not just the weights. The BF16 cells pin --moe-runner-backend deep_gemm with BF16 DeepEP dispatch output (JIT DeepGEMM is enabled via SGLANG_ENABLE_JIT_DEEPGEMM=1). The FP8 cells leave both at auto and let SGLang resolve the runner from the checkpoint's quantization config.

Spec Decode — NEXTN is on in every cell: 3 draft steps, 4 draft tokens per step, and the draft model path pointing at the target checkpoint itself. dots3's MTP layer is full-sharing — it carries the dots3 sliding-window attention geometry and reuses the target LM head — so no separate draft checkpoint is needed. Target verification and draft extension run on the paged, absorbed SWA-MLA FA3 path.

import { Deployment } from "/src/snippets/_deployment.jsx"; import { config } from "/src/snippets/configs/rednote/dots3-note.jsx";

1. Model Introduction

dots3.note is RedNote's native multimodal omni model, built on the dots3 language model. It accepts text, image, audio, and native video input.

• Native multimodality — a custom MoE vision transformer and a Whisper-derived audio encoder run in-process with the language model, loaded from the same checkpoint directory. Image and audio placeholders are expanded by a model-specific processor.
• Native video pipeline — the server jointly samples and interleaves frames, timestamps, and audio segments under a token budget, reproducing the training-time flattening algorithm. A generic uniform-frame video processor would silently change the modality ordering and token allocation (inference/training mismatch), so the pipeline is vendored into the serving path.
• Hybrid attention — dots3 combines MLA with full-attention and sliding-window layers of different geometry, attention gates, and optional DSA indexing on full-attention layers.
• MTP speculative decoding — a full-sharing MTP/NextN architecture exposes one recursively shared, SWA-shaped MTP layer and shares the target LM head.

Resources: Hugging Face · SGLang PR #33829

2. Configuration Tips

Hybrid KV pool. dots3 mixes full-attention and sliding-window layers, and its MTP draft layer is an ordinary SWA layer — not a full-attention one. SGLang sizes the pool accordingly, with --swa-full-tokens-ratio 0.03 setting the ratio of SWA-layer KV tokens to full-layer KV tokens (swa_tokens ≈ full_tokens × ratio). Lower it when long full-attention contexts dominate and the full pool fills first; raise it when the SWA pool is the bottleneck.

MoE runner. Leave the runner at the cell default: deep_gemm for BF16 checkpoints, auto for quantized ones. DeepEP is the all-to-all transport in every cell (--moe-a2a-backend deepep, dispatch tokens per rank tuned via SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128).

Attention backend. FA3 across the board: prefill, decode, and draft (--prefill-attention-backend fa3 --decode-attention-backend fa3 --speculative-draft-attention-backend fa3) with --page-size 64. MTP target verification uses FA3's absorbed SWA-MLA fallback, which consumes the same paged latent KV view as decode.

DSA. DSA indexing on full-attention layers is on by default. To disable it, add --json-model-override-args '{"index_topk":null}'.

CUDA graphs. The cells enable decode-side CUDA graphs only (--cuda-graph-backend-decode full --cuda-graph-backend-prefill disabled, max batch size 32) and are sized for GPUs with at least 120 GiB of memory. On smaller GPUs, switch to --cuda-graph-backend-decode disabled (and expect --deepep-mode normal to be the better fit).

Context length. --context-length 524288 is the model's window. Like other SGLang models, it bounds the longest accepted request; it does not size the KV pool.

Language-only mode. Add --language-only to skip constructing the vision and audio towers entirely — the freed memory goes to the language model. This is also the language role of an encoder/LLM-disaggregated (EPD) deployment; see EPD below.

3. Advanced Usage

3.1 Native video input

dots3.note accepts a native video_url. The server decodes the remote video in memory and applies the training-consistent flattening pipeline — interleaving timestamps, frames, and audio under a token budget, with a deterministic seed derived from the video and the question.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="dots3.note",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "video_url",
                    "video_url": {"url": "https://example.com/sample.mp4"},
                },
                {"type": "text", "text": "Summarize what happens in this video."},
            ],
        }
    ],
    extra_body={
        "seq": 131072,
        "audio_cap": 0.5,
        "audio_sr": 16000,
        "k_mode": "eval_ek",
    },
)

print(response.choices[0].message.content)

Pending update...

Per-request video preprocessing controls (all optional, passed via extra_body):

These controls are request-scoped so that evaluation jobs with different context budgets can share one server. The flattener reserves room for max_new_tokens inside the budget and falls back to visual-only processing if audio would exceed the configured token budget.

3.2 Image and audio input

Outside the native-video path, images and audio clips use the standard OpenAI multimodal message format and SGLang's multimodal serving (--enable-multimodal is in every cell). The vision and audio towers run in-process, so no extra server is needed.

3.3 Tool Calling

The cells launch with --tool-call-parser dots, so structured tool calls surface via message.tool_calls out of the box.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
resp = client.chat.completions.create(
    model="dots3.note",
    messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
    tools=tools,
)
print(resp.choices[0].message.tool_calls)

Pending update...

3.4 Encoder/LLM Disaggregation (EPD)

Dot3NoteForCausalLM supports both roles of an encoder/LLM-disaggregated deployment:

• Encoder role — serve with --encoder-only; the instance runs only the vision and audio towers.
• Language role — serve with --language-only; the instance skips tower construction, leaving the memory to the language model.

See the EPD guide for how to wire the roles together.