docs/articles/guides/console-args.md
BenchmarkSwitcher supports various console arguments, to make it work you need to pass the args to switcher:
class Program
{
static void Main(string[] args)
=> BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
}
Note: the docs that you are currently reading might get outdated, to get the most up-to-date info about supported console arguments run the benchmarks with --help.
The --filter or just -f allows you to filter the benchmarks by their full name (namespace.typeName.methodName) using glob patterns.
Examples:
-f 'System.Memory*'-f '*'-f '*ClassA*' '*ClassB*'Note: If you would like to join all the results into a single summary, you need to put --join. For example: -f '*ClassA*' '*ClassB*' --join
The --list allows you to print all of the available benchmark names. Available options are:
flat - prints list of the available benchmarks: --list flatBenchmarkDotNet.Samples.Algo_Md5VsSha256.Md5
BenchmarkDotNet.Samples.Algo_Md5VsSha256.Sha256
BenchmarkDotNet.Samples.IntroArguments.Benchmark
BenchmarkDotNet.Samples.IntroArgumentsSource.SingleArgument
BenchmarkDotNet.Samples.IntroArgumentsSource.ManyArguments
BenchmarkDotNet.Samples.IntroArrayParam.ArrayIndexOf
BenchmarkDotNet.Samples.IntroArrayParam.ManualIndexOf
BenchmarkDotNet.Samples.IntroBasic.Sleep
[...]
tree - prints tree of the available benchmarks: --list treeBenchmarkDotNet
└─Samples
├─Algo_Md5VsSha256
│ ├─Md5
│ └─Sha256
├─IntroArguments
│ └─Benchmark
├─IntroArgumentsSource
│ ├─SingleArgument
│ └─ManyArguments
├─IntroArrayParam
│ ├─ArrayIndexOf
│ └─ManualIndexOf
├─IntroBasic
│ ├─Sleep
[...]
The --list option works with the --filter option. Examples:
--list flat --filter *IntroSetupCleanup* prints:BenchmarkDotNet.Samples.IntroSetupCleanupGlobal.Logic
BenchmarkDotNet.Samples.IntroSetupCleanupIteration.Benchmark
BenchmarkDotNet.Samples.IntroSetupCleanupTarget.BenchmarkA
BenchmarkDotNet.Samples.IntroSetupCleanupTarget.BenchmarkB
BenchmarkDotNet.Samples.IntroSetupCleanupTarget.BenchmarkC
BenchmarkDotNet.Samples.IntroSetupCleanupTarget.BenchmarkD
--list tree --filter *IntroSetupCleanup* prints:BenchmarkDotNet
└─Samples
├─IntroSetupCleanupGlobal
│ └─Logic
├─IntroSetupCleanupIteration
│ └─Benchmark
└─IntroSetupCleanupTarget
├─BenchmarkA
├─BenchmarkB
├─BenchmarkC
└─BenchmarkD
You can also filter the benchmarks by categories:
--anyCategories - runs all benchmarks that belong to any of the provided categories--allCategories- runs all benchmarks that belong to all provided categories-m, --memory - enables MemoryDiagnoser and prints memory statistics-t, --threading - enables ThreadingDiagnoser and prints threading statistics-d, --disasm- enables DisassemblyDiagnoser and exports diassembly of benchmarked code. When you enable this option, you can use:
--disasmDepth - Sets the recursive depth for the disassembler.--disasmDiff - Generates diff reports for the disassembler.The --runtimes or just -r allows you to run the benchmarks for selected Runtimes. Available options are:
$Path, you can override it with --monoPath.--ilcPackages, --ilCompilerVersion.Example: run the benchmarks for .NET 4.7.2 and .NET 8.0:
dotnet run -c Release -- --runtimes net472 net8.0
Example: run the benchmarks for .NET Core 3.1 and latest .NET SDK installed on your PC:
dotnet run -c Release -f netcoreapp3.1 -- --runtimes clr core
But same command executed with -f net6.0 is going to run the benchmarks for .NET 6.0:
dotnet run -c Release -f net6.0 -- --runtimes clr core
--launchCount - how many times we should launch process with target benchmark. The default is 1.--warmupCount - how many warmup iterations should be performed. If you set it, the minWarmupCount and maxWarmupCount are ignored. By default calculated by the heuristic.--minWarmupCount - minimum count of warmup iterations that should be performed. The default is 6.--maxWarmupCount - maximum count of warmup iterations that should be performed. The default is 50.--iterationTime - desired time of execution of an iteration. Used by Pilot stage to estimate the number of invocations per iteration. 500ms by default.--iterationCount - how many target iterations should be performed. By default calculated by the heuristic.--minIterationCount - minimum number of iterations to run. The default is 15.--maxIterationCount - maximum number of iterations to run. The default is 100.--invocationCount - invocation count in a single iteration. By default calculated by the heuristic.--unrollFactor - how many times the benchmark method will be invoked per one iteration of a generated loop. 16 by default--runOncePerIteration - run the benchmark exactly once per iteration. False by default.Example: run single warmup iteration, from 9 to 12 actual workload iterations.
dotnet run -c Release -- --warmupCount 1 --minIterationCount 9 --maxIterationCount 12
If you want to have a possibility to specify custom default Job settings programmatically and optionally overwrite it with console line arguments, then you should create a global config with single job marked as .AsDefault and pass it to BenchmarkSwitcher together with the console line arguments.
Example: run single warmup iteration by default.
static void Main(string[] args)
=> BenchmarkSwitcher
.FromAssembly(typeof(Program).Assembly)
.Run(args, GetGlobalConfig());
static IConfig GetGlobalConfig()
=> DefaultConfig.Instance
.With(Job.Default
.WithWarmupCount(1)
.AsDefault()); // the KEY to get it working
Now, the default settings are: WarmupCount=1 but you might still overwrite it from console args like in the example below:
dotnet run -c Release -- --warmupCount 2
Benchmark.NET supports parsing parameters via response files. for example you can create file run.rsp with following content
--warmupCount 1
--minIterationCount 9
--maxIterationCount 12
and run it using dotnet run -c Release -- @run.rsp. It would be equivalent to running following command line
dotnet run -c Release -- --warmupCount 1 --minIterationCount 9 --maxIterationCount 12
To perform a Mann–Whitney U Test and display the results in a dedicated column you need to provide the Threshold:
--statisticalTest- Threshold for Mann–Whitney U Test. Examples: 5%, 10ms, 100ns, 1sExample: run Mann–Whitney U test with relative ratio of 5% for all benchmarks for .NET 6.0 (base) vs .NET 8.0 (diff). .NET 6.0 will be baseline because it was first.
dotnet run -c Release -- --filter * --runtimes net6.0 net8.0 --statisticalTest 5%
-j short-i--runtimes net472 net8.0 Mono--runtimes net8.0 net10.0-m-d--counters CacheMisses+InstructionRetired-f * -j Dry-f System.Memory*-f ClassA ClassB-f *.ClassA.* *.ClassB.*BenchmarkName and show the results in single summary:
--filter *.BenchmarkName --join--runOncePerIteration--invocationCount 100 --iterationCount 5 --warmupCount 1--iterationTime 250 --maxIterationCount 15 --minIterationCount 9--filter * --runtimes net6.0 net8.0 --statisticalTest 5%--envVars ENV_VAR_KEY_1:value_1 ENV_VAR_KEY_2:value_2-h Mean Ratio-j, --job (Default: Default) Dry/Short/Medium/Long or Default-r, --runtimes Full target framework moniker for .NET Core and .NET. For Mono just 'Mono'. For NativeAOT please append target runtime version (example: 'nativeaot7.0'). First one will be marked as baseline!-e, --exporters GitHub/StackOverflow/RPlot/CSV/JSON/HTML/XML-m, --memory (Default: false) Prints memory statistics-t, --threading (Default: false) Prints threading statistics--exceptions (Default: false) Prints exception statistics-d, --disasm (Default: false) Gets disassembly of benchmarked code-p, --profiler Profiles benchmarked code using selected profiler. Available options: EP/ETW/CV/NativeMemory-f, --filter Glob patterns-h, --hide Hides columns by name-i, --inProcess (Default: false) Run benchmarks in Process-a, --artifacts Valid path to accessible directory--outliers (Default: RemoveUpper) DontRemove/RemoveUpper/RemoveLower/RemoveAll--affinity Affinity mask to set for the benchmark process--allStats (Default: false) Displays all statistics (min, max & more)--allCategories Categories to run. If few are provided, only the benchmarks which belong to all of them are going to be executed--anyCategories Any Categories to run--attribute Run all methods with given attribute (applied to class or method)--join (Default: false) Prints single table with results for all benchmarks--keepFiles (Default: false) Determines if all auto-generated files should be kept or removed after running the benchmarks.--noOverwrite (Default: false) Determines if the exported result files should not be overwritten (be default they are overwritten).--counters Hardware Counters--cli Path to dotnet cli (optional).--packages The directory to restore packages to (optional).--coreRun Path(s) to CoreRun (optional).--monoPath Optional path to Mono which should be used for running benchmarks.--clrVersion Optional version of private CLR build used as the value of COMPLUS_Version env var.--ilCompilerVersion Optional version of Microsoft.DotNet.ILCompiler which should be used to run with NativeAOT. Example: "7.0.0-preview.3.22123.2"--ilcPackages Optional path to shipping packages produced by local dotnet/runtime build. Example: 'D:\projects\runtime\artifacts\packages\Release\Shipping'--launchCount How many times we should launch process with target benchmark. The default is 1.--warmupCount How many warmup iterations should be performed. If you set it, the minWarmupCount and maxWarmupCount are ignored. By default calculated by the heuristic.--minWarmupCount Minimum count of warmup iterations that should be performed. The default is 6.--maxWarmupCount Maximum count of warmup iterations that should be performed. The default is 50.--iterationTime Desired time of execution of an iteration in milliseconds. Used by Pilot stage to estimate the number of invocations per iteration. 500ms by default--iterationCount How many target iterations should be performed. By default calculated by the heuristic.--minIterationCount Minimum number of iterations to run. The default is 15.--maxIterationCount Maximum number of iterations to run. The default is 100.--invocationCount Invocation count in a single iteration. By default calculated by the heuristic.--unrollFactor How many times the benchmark method will be invoked per one iteration of a generated loop. 16 by default--strategy The RunStrategy that should be used. Throughput/ColdStart/Monitoring.--platform The Platform that should be used. If not specified, the host process platform is used (default). AnyCpu/X86/X64/Arm/Arm64/LoongArch64.--runOncePerIteration (Default: false) Run the benchmark exactly once per iteration.--info (Default: false) Print environment information.--apples (Default: false) Runs apples-to-apples comparison for specified Jobs.--list (Default: Disabled) Prints all of the available benchmark names. Flat/Tree--disasmDepth (Default: 1) Sets the recursive depth for the disassembler.--disasmFilter Glob patterns applied to full method signatures by the the disassembler.--disasmDiff (Default: false) Generates diff reports for the disassembler.--logBuildOutput Log Build output.--generateBinLog Generate msbuild binlog for builds--buildTimeout Build timeout in seconds.--wakeLock Prevents the system from entering sleep or turning off the display. None/System/Display.--stopOnFirstError (Default: false) Stop on first error.--statisticalTest Threshold for Mann–Whitney U Test. Examples: 5%, 10ms, 100ns, 1s--disableLogFile Disables the logfile.--maxWidth Max parameter column width, the default is 20.--envVars Colon separated environment variables (key:value)--memoryRandomization Specifies whether Engine should allocate some random-sized memory between iterations. It makes [GlobalCleanup] and [GlobalSetup] methods to be executed after every iteration.--wasmEngine Full path to a java script engine used to run the benchmarks, used by Wasm toolchain.--wasmArgs (Default: --expose_wasm) Arguments for the javascript engine used by Wasm toolchain.--customRuntimePack Path to a custom runtime pack. Only used for wasm/MonoAotLLVM currently.--AOTCompilerPath Path to Mono AOT compiler, used for MonoAotLLVM.--AOTCompilerMode (Default: mini) Mono AOT compiler mode, either 'mini' or 'llvm'--wasmDataDir Wasm data directory--wasmCoreCLR (Default: false) Use CoreCLR runtime pack (Microsoft.NETCore.App.Runtime.browser-wasm) instead of the Mono runtime pack for WASM benchmarks.--noForcedGCs Specifying would not forcefully induce any GCs.--noOverheadEvaluation Specifying would not run the evaluation overhead iterations.--resume (Default: false) Continue the execution if the last run was stopped.--help Display this help screen.--version Display version information.