Datasets Benchmark

Get started with the Benchmark API

Create Dataset Benchmark Task

python

POST /api/v2/serve/evaluate/execute_benchmark_task

shell

DBGPT_API_KEY=dbgpt
SPACE_ID={YOUR_SPACE_ID}

curl -X POST "http://localhost:5670/api/v2/serve/evaluate/execute_benchmark_task" \
-H "Authorization: Bearer $DBGPT_API_KEY" \
-H "accept: application/json" \
-H "Content-Type: application/json" \
-d '{
    "scene_value": "Falcon_benchmark_01",
    "model_list": ["DeepSeek-V3.1", "Qwen3-235B-A22B"]
}'

The Benchmark Request Object

scene_key string Required

The scene type of the evaluation, e.g. support app, recall

scene_value string Required

The scene value of the benchmark, e.g. The marking evaluation task name

model_list object Required

The model name list of the benchmark will execute, e.g. ["DeepSeek-V3.1","Qwen3-235B-A22B"] Notice: The model name configured on the db-gpt platform needs to be entered.

temperature float

The temperature of the llm model, Default is 0.7

max_tokens int

The max tokens of the llm model, Default is None

The Benchmark Result

status string

The benchmark status，e.g. success, failed, running

Query Benchmark Task List

python

GET /api/v2/serve/evaluate/benchmark_task_list

shell

DBGPT_API_KEY=dbgpt
SPACE_ID={YOUR_SPACE_ID}

curl -X GET "http://localhost:5670/api/v2/serve/evaluate/benchmark_task_list?page=1&page_size=20" \ 
-H "Authorization: Bearer $DBGPT_API_KEY" \
-H "accept: application/json" \
-H "Content-Type: application/json"

The Benchmark Task List Request Object

page string Required

Query task list page number, Default is 1

page_size string Required

Query task list page size, Default is 20

The Benchmark Task List Result

json

{
    "success": true,
    "err_code": null,
    "err_msg": null,
    "data": {
        "items": [
            {
                "evaluate_code": "1ec15dcbf5d54124bd5a5d23992af35d",
                "scene_key": "dataset",
                "scene_value": "local_benchmark_task_for_Qwen",
                "datasets_name": "Falcon评测集",
                "input_file_path": "2025_07_27_public_500_standard_benchmark_question_list.xlsx",
                "output_file_path": "/DB-GPT/pilot/benchmark_meta_data/result/1ec15dcbf5d54124bd5a5d23992af35d/202510201650_multi_round_benchmark_result.xlsx",
                "model_list": [
                    "Qwen3-Coder-480B-A35B-Instruct"
                ],
                "context": {
                    "benchmark_config": "{\"file_parse_type\":\"EXCEL\", \"format_type\":\"TEXT\", \"content_type\":\"SQL\", \"benchmark_mode_type\":\"EXECUTE\", \"scene_key\":\"dataset\", \"temperature\":0.6, \"max_tokens\":6000}"
                },
                "user_name": null,
                "user_id": null,
                "sys_code": "benchmark_system",
                "parallel_num": 1,
                "state": "running",
                "temperature": null,
                "max_tokens": null,
                "log_info": null,
                "gmt_create": "2025-10-20 16:50:46",
                "gmt_modified": "2025-10-20 16:50:46",
                "cost_time": null,
                "round_time": 1
            }
        ],
        "total_count": 80,
        "total_pages": 4,
        "page": 1,
        "page_size": 20
    }
}

evaluate_code string

The benchmark task unique code

scene_key string

The benchmark task scene, e.g. dataset

scene_value string

The benchmark task name

datasets_name string

The benchmark execute dataset name

input_file_path string

The benchmark dataset file path

output_file_path string

The benchmark execute result file path

model_list object

The benchmark execute model list

context object

The benchmark task context

user_name string

The benchmark task user name

user_id string

The benchmark task user id

sys_code string

The benchmark task system code, e.g. benchmark_system

parallel_num int

The benchmark task execute parallel num

state string

The benchmark task state, e.g. running, success, failed

temperature float

The benchmark task LLM temperature

max_tokens int

The benchmark task LLM max tokens

log_info int

If benchmark task execute error, It will show error message,

gmt_create string

Task create time

gmt_modified string

Task Finish time

cost_time int

Benchmark Task cost time

round_time int

Benchmark Task execute round time

Benchmark Compare Result

python

GET /api/v2/serve/evaluate/benchmark/result/{evaluate_code}

shell

DBGPT_API_KEY=dbgpt
SPACE_ID={YOUR_SPACE_ID}

curl -X GET "http://localhost:5670/api/v2/serve/evaluate/benchmark/result/{evaluate_code}" \
-H "Authorization: Bearer $DBGPT_API_KEY" \
-H "accept: application/json" \
-H "Content-Type: application/json"

The Benchmark Request Object

evaluate_code string Required

The benchMark task unique code

The Benchmark Result

json

{
    "success": true,
    "err_code": null,
    "err_msg": null,
    "data": {
        "evaluate_code": "c827a274b4084f5dbce4c630f5267239",
        "scene_value": "Falcon评测集_benchmark",
        "summaries": [
            {
                "roundId": 1,
                "llmCode": "Qwen3-Coder-480B-A35B-Instruct",
                "right": 136,
                "wrong": 269,
                "failed": 95,
                "exception": 0,
                "accuracy": 0.272,
                "execRate": 0.81,
                "outputPath": "/DB-GPT/pilot/benchmark_meta_data/result/c827a274b4084f5dbce4c630f5267239/202510181449_multi_round_benchmark_result.xlsx"
            }
        ]
    }
}

roundId string

The benchmark task execute round time

llmCode string

The benchmark task execute model name

right int The benchmark task execute right question number

wrong int The benchmark task execute wrong question number

failed int The benchmark task execute failed question number

exception int The benchmark task execute exception question number

accuracy float The benchmark task question list execute accuracy rate

execRate float The benchmark task question list executable rate

outputPath string The benchmark task execute result output file path