MiMo-V2.5

docker run -it --privileged \
  --shm-size=32g \
  --ipc=host \
  --network=host \
  -v /dev:/dev \
  us-docker.pkg.dev/cloud-tpu-images/jax-ai-image/tpu:jax0.8.1-rev1 bash

git clone https://github.com/sgl-project/sglang-jax.git
cd sglang-jax
pip install -e "python[tpu]"

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[
        {"role": "user", "content": "Which is larger, 9.11 or 9.9? Think carefully."}
    ]
)

print("====== Reasoning ======")
print(response.choices[0].message.reasoning_content)
print("====== Answer ======")
print(response.choices[0].message.content)

====== Reasoning ======
Comparing 9.11 and 9.9.

The integer parts are both 9. Now compare the decimal parts: 0.11 vs 0.9.

0.9 = 0.90, which is greater than 0.11.

So 9.9 > 9.11.
====== Answer ======
**9.9 is larger than 9.11.**

Here's the reasoning: When comparing decimals, line them up to the same number of decimal places:

- 9.11
- 9.90

Both have a **9** in the ones place, but in the tenths place, **9 > 1**, so 9.90 > 0.11.

**9.9 > 9.11**

response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[
        {"role": "user", "content": "Which is larger, 9.11 or 9.9? Think carefully."}
    ],
    extra_body={"chat_template_kwargs": {"thinking": False}}
)

print(response.choices[0].message.content)

## Comparing 9.11 and 9.9

**9.9 is larger.**

The key is to compare them place by place. It helps to write them with the same number of decimal places:

- **9.11** → 9.11
- **9.9** → 9.90

Both have **9** in the ones place, but in the tenths place: **9** (in 9.90) is greater than **1** (in 9.11).

So **9.90 > 9.11**.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/images/man_ironing_on_back_of_suv.png"}},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }]
)

print(response.choices[0].message.content)

Based on the image provided, here is a detailed description:

The image captures a whimsical or surreal scene set on a busy city street, likely in New York City given the iconic yellow cabs. In the center foreground, a man is sitting on a folding chair, casually crossing his legs. He is wearing a bright yellow hoodie with a graphic on the front and blue jeans. He is intently focused on ironing a white dress shirt that rests on an ironing board set up directly on the asphalt.

Behind him, a yellow SUV taxi cab is stopped or moving slowly, angled slightly away from the camera. To his left, another yellow taxi sedan is captured in motion blur, indicating it is driving past him. The background features tall city buildings with glass windows and storefronts. There are banners hanging from streetlights, and some greenery is visible in the distance. The overall impression is one of incongruity—performing a domestic chore like ironing in the middle of a chaotic urban environment.

response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "video_url", "video_url": {"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"}},
            {"type": "text", "text": "Summarize what happens in this video."}
        ]
    }]
)

print(response.choices[0].message.content)

A person wearing blue protective gloves is shown operating a microscope in a close-up shot. The individual is adjusting a knob on the side of the microscope, which moves the stage holding a glass slide, likely focusing the lens on the specimen.

response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "audio_url", "audio_url": {"url": "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/audios/Trump_WEF_2018_10s.mp3"}},
            {"type": "text", "text": "Transcribe and summarize this audio."}
        ]
    }]
)

print(response.choices[0].message.content)

**Transcript:**
"Thank you Klaus very much. It's a privilege to be here at this forum where leaders in business, science, art, diplomacy and world affairs have gathered for..."

**Summary:**
The speaker thanks Klaus for the introduction and expresses their honor at attending a forum. They highlight that the event has brought together high-level leaders from various sectors, including business, science, art, and diplomacy.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
    tools=tools
)

msg = response.choices[0].message
if msg.reasoning_content:
    print("=== Reasoning ===")
    print(msg.reasoning_content)
if msg.tool_calls:
    print("=== Tool Calls ===")
    for tc in msg.tool_calls:
        print(f"  Function: {tc.function.name}")
        print(f"  Arguments: {tc.function.arguments}")

=== Reasoning ===
The user wants to know the weather in Beijing. I have a function available called "get_weather" that can retrieve current weather for a location. Let me call that function with Beijing as the location.
=== Tool Calls ===
  Function: get_weather
  Arguments: {"location": "Beijing"}

python3 -m sglang.test.run_eval \
  --base-url http://127.0.0.1:30000 \
  --model XiaomiMiMo/MiMo-V2.5 \
  --eval-name gsm8k \
  --num-examples 200 \
  --num-threads 8 \
  --max-tokens 4096 \
  --temperature 0.0

python3 benchmark/mmmu/bench_sglang.py \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5 \
  --concurrency 16

python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5-Pro \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  27.59
Total input tokens:                      1997
Total input text tokens:                 1997
Total generated tokens:                  2798
Total generated tokens (retokenized):    2794
Request throughput (req/s):              0.36
Input token throughput (tok/s):          72.38
Output token throughput (tok/s):         101.41
Peak output token throughput (tok/s):    110.00
Peak concurrent requests:                3
Total token throughput (tok/s):          173.79
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2757.26
Median E2E Latency (ms):                 3319.10
P90 E2E Latency (ms):                    4157.47
P99 E2E Latency (ms):                    4869.32
---------------Time to First Token----------------
Mean TTFT (ms):                          162.17
Median TTFT (ms):                        68.11
P99 TTFT (ms):                           929.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.19
Median TPOT (ms):                        9.33
P99 TPOT (ms):                           9.39
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.31
Median ITL (ms):                         9.35
P95 ITL (ms):                            9.44
P99 ITL (ms):                            9.77
Max ITL (ms):                            19.80
==================================================

python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5-Pro \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  112.78
Total input tokens:                      302118
Total input text tokens:                 302118
Total generated tokens:                  195775
Total generated tokens (retokenized):    191069
Request throughput (req/s):              8.87
Input token throughput (tok/s):          2678.83
Output token throughput (tok/s):         1735.90
Peak output token throughput (tok/s):    3040.00
Peak concurrent requests:                121
Total token throughput (tok/s):          4414.73
Concurrency:                             87.80
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   9901.96
Median E2E Latency (ms):                 6525.54
P90 E2E Latency (ms):                    23567.98
P99 E2E Latency (ms):                    42109.22
---------------Time to First Token----------------
Mean TTFT (ms):                          223.69
Median TTFT (ms):                        139.45
P99 TTFT (ms):                           1082.02
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          50.63
Median TPOT (ms):                        51.66
P99 TPOT (ms):                           91.41
---------------Inter-Token Latency----------------
Mean ITL (ms):                           49.79
Median ITL (ms):                         33.69
P95 ITL (ms):                            103.37
P99 ITL (ms):                            151.34
Max ITL (ms):                            1600.00
==================================================

python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  14.72
Total input tokens:                      1997
Total input text tokens:                 1997
Total generated tokens:                  2798
Total generated tokens (retokenized):    2697
Request throughput (req/s):              0.68
Input token throughput (tok/s):          135.67
Output token throughput (tok/s):         190.09
Peak output token throughput (tok/s):    245.00
Peak concurrent requests:                3
Total token throughput (tok/s):          325.77
Concurrency:                             1.00
Accept length:                           3.08
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1469.98
Median E2E Latency (ms):                 1652.84
P90 E2E Latency (ms):                    2210.80
P99 E2E Latency (ms):                    2823.86
---------------Time to First Token----------------
Mean TTFT (ms):                          143.89
Median TTFT (ms):                        99.25
P99 TTFT (ms):                           481.01
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.87
Median TPOT (ms):                        4.30
P99 TPOT (ms):                           6.64
---------------Inter-Token Latency----------------
Mean ITL (ms):                           4.76
Median ITL (ms):                         3.46
P95 ITL (ms):                            13.52
P99 ITL (ms):                            13.84
Max ITL (ms):                            74.37
==================================================

python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  93.41
Total input tokens:                      302118
Total input text tokens:                 302118
Total generated tokens:                  195775
Total generated tokens (retokenized):    188139
Request throughput (req/s):              10.71
Input token throughput (tok/s):          3234.48
Output token throughput (tok/s):         2095.97
Peak output token throughput (tok/s):    3019.00
Peak concurrent requests:                121
Total token throughput (tok/s):          5330.45
Concurrency:                             91.04
Accept length:                           2.95
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   8503.45
Median E2E Latency (ms):                 7491.96
P90 E2E Latency (ms):                    13706.99
P99 E2E Latency (ms):                    20474.33
---------------Time to First Token----------------
Mean TTFT (ms):                          4399.20
Median TTFT (ms):                        4333.35
P99 TTFT (ms):                           8004.81
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          58.23
Median TPOT (ms):                        21.78
P99 TPOT (ms):                           747.79
---------------Inter-Token Latency----------------
Mean ITL (ms):                           20.06
Median ITL (ms):                         15.28
P95 ITL (ms):                            48.36
P99 ITL (ms):                            96.99
Max ITL (ms):                            969.61
==================================================

python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5 \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1

============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  25.73
Total input tokens:                      661
Total input text tokens:                 631
Total input vision tokens:               30
Total generated tokens:                  4220
Total generated tokens (retokenized):    0
Request throughput (req/s):              0.39
Input token throughput (tok/s):          25.69
Output token throughput (tok/s):         164.03
Peak output token throughput (tok/s):    1.00
Peak concurrent requests:                2
Total token throughput (tok/s):          189.73
Concurrency:                             1.00
Accept length:                           2.94
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2570.74
Median E2E Latency (ms):                 2411.92
P90 E2E Latency (ms):                    3711.62
P99 E2E Latency (ms):                    4949.74
---------------Time to First Token----------------
Mean TTFT (ms):                          0.00
Median TTFT (ms):                        0.00
P99 TTFT (ms):                           0.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.31
Median TPOT (ms):                        6.17
P99 TPOT (ms):                           17.18
---------------Inter-Token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            0.00
P99 ITL (ms):                            0.00
Max ITL (ms):                            0.00
==================================================

python3 -m sglang.test.run_eval \
  --base-url http://127.0.0.1:30000 \
  --model XiaomiMiMo/MiMo-V2.5-Pro \
  --eval-name gsm8k \
  --num-examples 200 \
  --num-threads 4

python3 -m sglang.bench_serving \
  --backend sglang \
  --model XiaomiMiMo/MiMo-V2.5-Pro \
  --host 0.0.0.0 \
  --port 30000 \
  --dataset-name random \
  --random-input-len <INPUT_LEN> \
  --random-output-len 1 \
  --random-range-ratio 1.0 \
  --flush-cache \
  --seed 12345 \
  --num-prompts 10000