Quantization Tool

This tool can be used to quantize selected ONNX models. Support is based on operators in the model. Please refer to https://onnxruntime.ai/docs/performance/quantization.html for usage details and https://github.com/microsoft/onnxruntime-inference-examples/tree/main/quantization for examples.

Static Quantization Tool

Build

Please add --enable_pybind and --build_wheel to the build command to acquire the python tools.

bash

cd onnxruntime
.\build.bat --config RelWithDebInfo --build_shared_lib --parallel --cmake_generator "Visual Studio 17 2022" --enable_pybind --build_wheel

Model and Data

The static quantization tool expects the directory structure of model and data.

ps1

work_dir\resnet18-v1-7
├───model.onnx
├───test_data_set_0
│   ├───input_0.pb
│   └───input_1.pb
├───test_data_set_1
│   ├───input_0.pb
│   └───input_1.pb
├───test_data_set_2
│   ├───input_0.pb
│   └───input_1.pb
├───test_data_set_3
│   ├───input_0.pb
│   └───input_1.pb
├───test_data_set_4
│   ├───input_0.pb
│   └───input_1.pb
├───test_data_set_5
│   ├───input_0.pb
│   └───input_1.pb
├───test_data_set_6
│   ├───input_0.pb
│   └───input_1.pb
├───test_data_set_7
│   ├───input_0.pb
│   └───input_1.pb
├───test_data_set_8
│   ├───input_0.pb
│   └───input_1.pb
└───test_data_set_9
    ├───input_0.pb
    └───input_1.pb

Note that the indexing must fully align the order of model inputs (i.e., input_0.pb is expected to be the data for the 1st model input, input_1.pb for the 2nd, and so on).

Usage

Install the python tools built in onnxruntime

ps1

cd work_dir
python -m venv ort_env
ort_env\Scripts\activate
python -m pip install <path-to-built-folder>\RelWithDebInfo\RelWithDebInfo\dist\<name-of-the-wheel>.whl

# The following command yields model_quant.onnx under the same directory "resnet18-v1-7"
python -m onnxruntime.quantization.static_quantize_runner -i resnet18-v1-7\model.onnx -o resnet18-v1-7\model_quant.onnx

work_dir\resnet18-v1-7
├───model.onnx
├───model_quant.onnx
├───test_data_set_0
│   ...
└───test_data_set_9

Quantization Arguments

Please refer to static_quantize_runner.py for more detailed arguments.

ps1

python -m onnxruntime.quantization.static_quantize_runner -i resnet18-v1-7\model.onnx -o resnet18-v1-7\model_quant.onnx --activation_type qint8 --weight_type qint16
python -m onnxruntime.quantization.static_quantize_runner -i resnet18-v1-7\model.onnx -o resnet18-v1-7\model_quant.onnx --activation_type qint16 --weight_type qint16 --quantize_bias
python -m onnxruntime.quantization.static_quantize_runner -i resnet18-v1-7\model.onnx -o resnet18-v1-7\model_quant.onnx --activation_type qint16 --weight_type qint8 --per_channel

Tensor Quant Overrides Json Format

With --tensor_quant_overrides, the tool can consume the json file with quantization override information.

ps1

python -m onnxruntime.quantization.static_quantize_runner -i resnet18-v1-7\model.onnx -o resnet18-v1-7\model_quant.onnx --tensor_quant_overrides <path-to-json>\encoding.json

The tool expects the encoding.json with the format:

json

{
    "conv1_1": [
        {
            "scale": 0.005,
            "zero_point": 12
        }
    ]
}

Each key is the name of a tensor in the onnx model.
- e.g. "conv1_1"
For each tensor, a list of dictionary should be provided
- For per-tensor quantization, the list contains a single dictionary.
- For per-channel quantization, the list contains a dictionary for each channel in the tensor.
- Each dictionary contain the information required for quantization including:
  - scale (float)
  - zero_point (int)