Back to Docker

docker model bench

_vendor/github.com/docker/model-runner/cmd/cli/docs/reference/model_bench.md

18.09-release1.4 KB
Original Source

docker model bench

<!---MARKER_GEN_START-->

Benchmark a model's performance showing tokens per second at different concurrency levels.

This command runs a series of benchmarks with 1, 2, 4, and 8 concurrent requests by default, measuring the tokens per second (TPS) that the model can generate.

Options

NameTypeDefaultDescription
--concurrencyintSlice[1,2,4,8]Concurrency levels to test
--durationduration30sDuration to run each concurrency test
--jsonboolOutput results in JSON format
--promptstringWrite a comprehensive 100 word summary on whales and their impact on society.Prompt to use for benchmarking
--timeoutduration5m0sTimeout for each individual request
<!---MARKER_GEN_END-->