Inkling

Deployment

For all install methods and hardware platforms, see the official SGLang installation guide.

Inkling support isn't in a pip release yet — install from the inkling-support branch:

pip install --upgrade pip
pip install "sglang[all] @ git+https://github.com/sgl-project/sglang.git@inkling-support"

Then run the Python output of the command panel below.

<Note>The Inkling images are being published to lmsysorg/sglang — watch the tag list for status.</Note>

There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU:

docker pull lmsysorg/sglang:inkling-cu13            # CUDA 13
docker pull lmsysorg/sglang:inkling-cu12            # CUDA 12
docker pull lmsysorg/sglang:inkling-rocm700-mi35x   # AMD MI350X / MI355X

For how to launch the image, see Install → Method 3: Using Docker. Substitute the inner sglang serve ... with what the command generator below produces.

Pick your hardware to generate the launch command. Each platform ships a Balanced recipe plus an MTP (speculative decoding) tier and a Long Context (MXFP8 KV) tier where validated; the LoRA variant serves adapters on top of the frozen base model. Set MAX_LORAS to the number of distinct adapters you serve (1 is fastest for single-adapter serving).

import { Deployment } from "/src/snippets/_deployment.jsx"; import { config } from "/src/snippets/configs/thinkingmachines/inkling.jsx"; import { benchmarks } from "/src/snippets/configs/thinkingmachines/inkling-benchmarks.jsx";

Playground

The Playground is where you experiment with SGLang features beyond the verified matrix. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.

Lines highlighted green are added by your overrides; lines with red strikethrough were in the verified base but stripped by an override. Any change flips the badge to Not Verified until the new configuration is run end-to-end.

import { Playground } from "/src/snippets/_playground.jsx";

1. Model Introduction

Inkling is a Mixture-of-Experts model from Thinking Machines — 975B total parameters, 41B active per token, with a 1M-token context window and open weights (BF16 and NVFP4 checkpoints below). It handles text, image, and audio inputs natively, and exposes a variable reasoning-effort control to trade latency and cost against answer quality. This page covers serving Inkling on SGLang, including its MTP speculative-decoding path and long-context prefix caching (unified radix cache + HiCache).

Resources: HuggingFace — Inkling (BF16) · Inkling-NVFP4.

2. Configuration Tips

Multimodal. The recipes pass --enable-multimodal so the server accepts image and audio inputs alongside text — drop it for text-only serving.

Memory pool ratios. --swa-full-tokens-ratio and --mamba-full-memory-ratio (both default 0.1) size the SWA and Mamba/sconv state pools; tune them to your workload's usage.

MTP needs --enable-multi-layer-eagle. The MTP recipe drives Inkling's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage.

Reasoning effort. Pass reasoning_effort as one of the named levels below; requests that omit it default to high, and max is the strongest. Each level maps to an internal effort value (max at 0.99):

3. Advanced Usage

3.1 Reasoning

Enable the inkling reasoning parser (toggle Reasoning Parser in the Parsers card of the Playground above) to separate thinking from the final answer into reasoning_content vs content.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="thinkingmachines/Inkling-NVFP4",
    messages=[{"role": "user", "content": "What is 17 times 24?"}],
    extra_body={"chat_template_kwargs": {"thinking": True}},
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Answer:", msg.content)

Reasoning: The user is asking for the product of 17 and 24. Let me calculate that.

17 × 24

I can break this down:
17 × 20 = 340
17 × 4 = 68
340 + 68 = 408

Alternatively:
24 × 10 = 240
24 × 7 = 168
240 + 168 = 408

So the answer is 408.
Answer: 17 times 24 is **408**.

Here's a quick breakdown:
- 17 × 20 = 340
- 17 × 4 = 68
- 340 + 68 = **408**

3.2 Tool Calling

Enable the inkling tool-call parser (toggle Tool Call Parser in the Parsers card of the Playground above) to surface structured tool calls via message.tool_calls.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {"location": {"type": "string", "description": "The city name"}},
                "required": ["location"],
            },
        },
    }
]

resp = client.chat.completions.create(
    model="thinkingmachines/Inkling-NVFP4",
    messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
    tools=tools,
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Content:", msg.content)
print("Tool calls:", msg.tool_calls)

Reasoning: The user is asking for the weather in Beijing. I have a tool called `get_weather` that can get the current weather for a location. Let me call it with "Beijing" as the location.
Content:
Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_98f772f3a0044f45b80c5ba5', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)]

3.3 Multimodal Input (Image + Audio)

Inkling is multimodal: a single user message can mix text, images, and audio. Pass each media item as its own content part — image_url for images, audio_url for audio — with the url set to either an HTTP(S) link or a base64 data: URI. The server must be started with --enable-multimodal (already included in every recipe above).

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

with open("image.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()
with open("audio.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="thinkingmachines/Inkling-NVFP4",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
                {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}},
                {"type": "text", "text": "Describe the image, then transcribe the audio."},
            ],
        }
    ],
    max_tokens=1024,
)
print(resp.choices[0].message.content)

3.4 LoRA (Serving Adapters)

The LoRA deploy variant serves adapters on top of the frozen base model. Its launch command adds --enable-lora --lora-paths lora0={{ADAPTER_PATH}} --max-loras-per-batch {{MAX_LORAS}} — each adapter is registered under the name to the left of = (here lora0). Adapters can also be added/removed at runtime via the POST /load_lora_adapter endpoint. To serve several adapters, pass multiple --lora-paths name=path at launch and reference each by its name.

Pick the adapter per request by that name — either in the model field with base-model:adapter syntax (recommended), or explicitly via lora_path in extra_body:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

# Option A (recommended): "<model>:<adapter-name>" in the model field
resp = client.chat.completions.create(
    model="thinkingmachines/Inkling-NVFP4:lora0",
    messages=[{"role": "user", "content": "Summarize the changelog."}],
)

# Option B: explicit lora_path via extra_body
resp = client.chat.completions.create(
    model="thinkingmachines/Inkling-NVFP4",
    messages=[{"role": "user", "content": "Summarize the changelog."}],
    extra_body={"lora_path": "lora0"},
)

print(resp.choices[0].message.content)

3.5 HiCache (Hierarchical KV Caching)

Inkling serves on SGLang's unified radix cache: the historically separate full-attention, SWA, and Mamba/sconv caches are combined into one radix tree with typed components, and native HiCache offloads cold prefix pages across tiers (GPU HBM → host DRAM → disk / remote). This expands effective prefix-cache capacity for multi-turn and long-context workloads.

To enable HiCache, open the HiCache card in the Playground above and flip Enable, then pick a storage backend (file / mooncake / nixl) for the L3 tier. The Write policy defaults to write_through.

3.6 Long Context (MXFP8 KV)

The Long Context deploy strategy adds --kv-cache-dtype mxfp8 on top of the Balanced recipe. KV entries are stored as block-scaled MXFP8 instead of BF16, so the SWA + Mamba/sconv memory pool holds roughly 2x as many tokens on the same GPU. Use it when you're context-bound or concurrency-bound.

Blackwell only. MXFP8 KV cache requires Blackwell (B200 / B300 / GB200 / GB300), it's not offered on Hopper (H200).

The tradeoff is not just a ~5% decode latency penalty from the extra quantize/dequantize work versus BF16 KV — storing KV in MXFP8 also introduces some accuracy loss at long context lengths. Treat it as a capacity lever, not a speed one — stay on Balanced if you have headroom in the memory pool and just want lower latency or maximum output quality.

To try it, select the Long Context strategy in the Deploy panel above for any NVFP4 cell; the panel regenerates the launch command with --kv-cache-dtype mxfp8 inserted. Verified end-to-end on B200, B300, and GB300.