Cosmos3

import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx';

<DiffusionModelTags tags={["image", "video", "sound/action", "world model", "policy"]} />

1. Model Introduction

NVIDIA Cosmos3 is an omnimodal world-model family for image, video, sound, and action generation. SGLang Diffusion serves the public checkpoints with its native Cosmos3 pipeline.

Sound and action generation require the corresponding checkpoint heads. The pipeline reads the transformer and scheduler configs at startup, so Edge and distilled checkpoints do not require architecture-specific server flags. Non-distilled checkpoints use the flow-native FlowUniPCMultistepScheduler; distilled checkpoints use the fixed sigma schedule stored in the checkpoint.

The default flow_shift is 3.0 for T2I, 10.0 for non-Edge video and all action modes, and 3.0 for Edge video modes. Distilled checkpoints bake the schedule into their sigmas and do not use a request-level flow_shift.

2. Installation

Install SGLang with the diffusion dependencies:

pip install -e "python[diffusion]"

Cosmos3 guardrails are enabled by default when the package is available:

pip install "cosmos-guardrail==0.3.1"

cosmos-guardrail downloads gated NVIDIA guardrail weights, so pass a Hugging Face token if your environment needs one. If the package is not installed, SGLang skips Cosmos3 guardrails and logs a warning. To disable Cosmos3 guardrails for local experiments, set SGLANG_DISABLE_COSMOS3_GUARDRAILS=1 before starting the server.

3. Serve Cosmos3

Serve Cosmos3-Nano directly from the Hugging Face model ID:

sglang serve \
  --model-path nvidia/Cosmos3-Nano \
  --num-gpus 1

For Cosmos3-Super, split the model across multiple GPUs:

sglang serve \
  --model-path nvidia/Cosmos3-Super \
  --num-gpus 4

The server also accepts the specialized nvidia/Cosmos3-Super-Text2Image and nvidia/Cosmos3-Super-Image2Video checkpoint IDs.

Edge checkpoints

Cosmos3-Edge is a 4B dense model and can be served on one GPU:

sglang serve \
  --model-path nvidia/Cosmos3-Edge \
  --num-gpus 1

Edge is trained for 256p and 480p generation. Its default video configuration is 832x480 with guidance_scale=5.0; its default image configuration is 640x640 with guidance_scale=7.0. Supported sizes are 832x480, 480x832, 640x480, 480x640, 480x480, 640x640, 448x256, 256x448, and 256x256.

Serve the Edge DROID policy checkpoint with the same single-GPU configuration, replacing the model path with nvidia/Cosmos3-Edge-Policy-DROID.

Distilled checkpoints

The distilled Super checkpoints are 64B models. Use multiple GPUs unless the complete model and request workload fit on one GPU:

sglang serve \
  --model-path nvidia/Cosmos3-Super-Text2Image-4Step \
  --num-gpus 4

For distilled I2V, replace the model path with nvidia/Cosmos3-Super-Image2Video-4Step. SGLang detects both checkpoints from scheduler/scheduler_config.json, uses the checkpoint's fixed four-step sigma schedule, and forces guidance_scale=1.0. Do not tune num_inference_steps or flow_shift for these checkpoints.

4. OpenAI-Compatible Requests

Text to image

Cosmos3 text-to-image uses /v1/images/generations. The default Cosmos3 image response is b64_json, matching vLLM-Omni's examples.

curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "1280x720",
    "n": 1,
    "num_inference_steps": 35,
    "guidance_scale": 6.0,
    "flow_shift": 3.0,
    "seed": 0,
    "extra_args": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'

With a server running nvidia/Cosmos3-Super-Text2Image-4Step, omit the scheduler controls and use guidance_scale=1.0:

curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "640x640",
    "n": 1,
    "guidance_scale": 1.0,
    "seed": 0,
    "extra_args": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'

Text to video with sound

Use /v1/videos to create an asynchronous job, then poll the job and download the completed MP4. Set generate_sound=true to generate and mux a stereo 48 kHz audio track; omit it for a silent video.

job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string "negative_prompt=blurry, distorted, low quality" \
  --form-string "size=1280x720" \
  --form-string "num_frames=81" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=4.0" \
  --form-string "flow_shift=10.0" \
  --form-string "generate_sound=true" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

while true; do
  status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${job_id}" \
    | python -c 'import json, sys; print(json.load(sys.stdin)["status"])')
  [ "$status" = "completed" ] && break
  [ "$status" = "failed" ] && exit 1
  sleep 1
done

curl -sS -L "http://127.0.0.1:30010/v1/videos/${job_id}/content" \
  -o cosmos3_t2v.mp4

Image to video

This mirrors the official nvidia/Cosmos3-Nano Hugging Face image-to-video example:

import json
import time
from pathlib import Path

import requests
from huggingface_hub import snapshot_download

base_url = "http://127.0.0.1:30010"
model_dir = Path(snapshot_download("nvidia/Cosmos3-Nano"))
asset_dir = model_dir / "assets"

prompt = json.dumps(json.loads((asset_dir / "example_i2v_prompt.json").read_text()))
negative_prompt = json.dumps(
    json.loads((asset_dir / "negative_prompt.json").read_text())
)

data = {
    "prompt": prompt,
    "negative_prompt": negative_prompt,
    "size": "1280x720",
    "num_frames": "189",
    "fps": "24",
    "num_inference_steps": "35",
    "guidance_scale": "6.0",
    "max_sequence_length": "4096",
    "flow_shift": "10.0",
    "seed": "1111",
    "extra_params": json.dumps(
        {
            "use_resolution_template": False,
            "use_duration_template": False,
            "guardrails": True,
        }
    ),
}

with (asset_dir / "example_i2v_input.jpg").open("rb") as image:
    response = requests.post(
        f"{base_url}/v1/videos",
        data=data,
        files={"input_reference": ("example_i2v_input.jpg", image, "image/jpeg")},
        timeout=60,
    )
response.raise_for_status()
video_id = response.json()["id"]

while True:
    job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json()
    if job["status"] == "completed":
        break
    if job["status"] == "failed":
        raise RuntimeError(job.get("error") or "Video generation failed")
    time.sleep(1)

response = requests.get(f"{base_url}/v1/videos/{video_id}/content", timeout=300)
response.raise_for_status()
Path("cosmos3_i2v.mp4").write_bytes(response.content)

For the distilled I2V checkpoint, use the same API with a server running nvidia/Cosmos3-Super-Image2Video-4Step. The recommended request is 480p and does not specify scheduler controls:

job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A warehouse robot carefully places a blue box on a shelf." \
  --form "input_reference=@first_frame.png;type=image/png" \
  --form-string "size=832x480" \
  --form-string "num_frames=189" \
  --form-string "fps=24" \
  --form-string "guidance_scale=1.0" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

Poll and download this job with the same status and content endpoints used by the T2V example.

Video to video

Upload a source video with video_reference. Cosmos3 keeps latent frames [0, 1] by default and generates the remaining frames. Use condition_frame_indexes to select different latent frames, and condition_video_keep to take conditioning frames from the start or end of the source.

job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A robotic arm pours liquid into a glass on a white tabletop." \
  --form "video_reference=@robot_pouring.mp4;type=video/mp4" \
  --form-string "size=1280x704" \
  --form-string "num_frames=45" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=6.0" \
  --form-string 'condition_frame_indexes=[0,1]' \
  --form-string "condition_video_keep=first" \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

Poll and download this job with the same status and content endpoints used by the T2V example.

Action generation

For DROID policy generation, start a single-GPU server with either the Nano or Edge policy checkpoint. Cosmos3 action generation does not currently support CFG or sequence parallelism.

sglang serve \
  --model-path nvidia/Cosmos3-Nano-Policy-DROID \
  --num-gpus 1

Use nvidia/Cosmos3-Edge-Policy-DROID in the same command to serve the smaller 4B policy checkpoint.

policy and inverse_dynamics return actions, so their canonical API is the synchronous /v1/actions/generations endpoint. The following request predicts a 16-step action chunk from one observation image. action_horizon=16 maps to the model's num_frames=17 convention.

import base64
from pathlib import Path

import requests

image_b64 = base64.b64encode(Path("observation.png").read_bytes()).decode()
response = requests.post(
    "http://127.0.0.1:30010/v1/actions/generations",
    json={
        "input": {
            "task": "Put the pot to the left of the purple item.",
            "observation": {
                "image": {"b64_json": image_b64},
            },
        },
        "parameters": {
            "action_mode": "policy",
            "action_horizon": 16,
            "domain_name": "droid_lerobot",
            "height": 480,
            "width": 832,
            "fps": 5,
            "num_inference_steps": 30,
            "guidance_scale": 1.0,
            "seed": 42,
        },
    },
    timeout=300,
)
response.raise_for_status()
action = response.json()["data"][0]["action"]
print(action["shape"], action["values"])

Use GET /v1/actions/metadata to inspect the action modes, default horizon, padded action dimension, and accepted observation modalities. Msgpack requests and the /v1/actions/realtime websocket use the same action envelope.

inverse_dynamics also uses /v1/actions/generations; set action_mode="inverse_dynamics" and pass an observation video URL or server-local path as input.observation.video. Select the embodiment head with domain_name or domain_id; set raw_action_dim explicitly when it cannot be inferred from the domain name.

forward_dynamics is intentionally different: it consumes an action array and predicts video, so it remains on /v1/videos. Action-producing modes submitted to /v1/videos return HTTP 400 with the canonical action endpoint in the error message.

5. Cosmos3 Parameters

Cosmos3 supports the standard SGLang video and image fields such as size, num_frames, fps, num_inference_steps, guidance_scale, negative_prompt, and seed. For distilled checkpoints, SGLang replaces num_inference_steps with the checkpoint's fixed four-step schedule and forces guidance_scale=1.0; negative-prompt CFG and request-level flow_shift do not apply.

Top-level Cosmos3 request fields:

• max_sequence_length: maximum text token length used by the Cosmos3 tokenizer.
• flow_shift: per-request scheduler shift for non-distilled checkpoints. If omitted, SGLang uses --flow-shift, then the mode default (3.0 for T2I, 10.0 for non-Edge video and all action modes, or 3.0 for Edge video).
• guidance_interval: optional [start, end] noise interval for CFG. Non-distilled T2I defaults to [400, 1000]; video modes guide at every step.

Cosmos3 omnimodal fields are accepted as extra JSON fields or multipart form fields:

• generate_sound: generate a sound track whose duration follows num_frames / fps.
• sound_duration: explicit sound duration in seconds; takes precedence over the derived duration.
• condition_frame_indexes: V2V latent-frame indexes to keep from the source video; defaults to [0, 1].
• condition_video_keep: use the first or last source frames for V2V conditioning.
• action_mode: policy, forward_dynamics, or inverse_dynamics.
• domain_name / domain_id: select the action embodiment head.
• raw_action_dim: number of active action dimensions; inferred for known domain names.
• action: action array with shape [T, D], required by forward_dynamics.
• action_fps: action-token frame rate for temporal mRoPE; defaults to the video FPS.
• action_view_point: viewpoint used in the structured action caption.
• action_normalization: dataset normalization mode, such as quantile, meanstd, or minmax.

Put model-specific compatibility knobs in extra_params for video requests, or extra_args for image requests:

• use_duration_template: whether to append SGLang's generated duration suffix to video prompts.
• use_resolution_template: accepted for vLLM-Omni request compatibility.
• use_system_prompt: whether to add the Cosmos3 system prompt to the chat template.
• guardrails or use_guardrails: per-request guardrail toggle when the server started with guardrails enabled.