Throughput Benchmark Sample

.. meta:: :description: Learn how to estimate performance of a model using Asynchronous Inference Request API in throughput mode (Python, C++).

This sample demonstrates how to estimate performance of a model using Asynchronous Inference Request API in throughput mode. This sample does not have other configurable command-line arguments. Feel free to modify sample's source code to try out different options.

The reported results may deviate from what :doc:benchmark_app <benchmark-tool> reports. One example is model input precision for computer vision tasks. benchmark_app sets uint8, while the sample uses default model precision which is usually float32.

Before using the sample, refer to the following requirements:

• The sample accepts any file format supported by core.read_model.
• The sample has been validated with: yolo-v3-tf and face-detection-0200 models.
• To build the sample, use instructions available at :ref:Build the Sample Applications <build-samples> section in "Get Started with Samples" guide.

How It Works ####################

The sample compiles a model for a given device, randomly generates input data, performs asynchronous inference multiple times for a given number of seconds. Then, it processes and reports performance results.

.. tab-set::

.. tab-item:: Python :sync: python

  .. scrollbox::

     .. doxygensnippet:: samples/python/benchmark/throughput_benchmark/throughput_benchmark.py
        :language: python

.. tab-item:: C++ :sync: cpp

  .. scrollbox::

     .. doxygensnippet:: samples/cpp/benchmark/throughput_benchmark/main.cpp
        :language: cpp

You can see the explicit description of each sample step at :doc:Integration Steps <../../../openvino-workflow/running-inference> section of "Integrate OpenVINO™ Runtime with Your Application" guide.

Running ####################

.. tab-set::

.. tab-item:: Python :sync: python

  .. code-block:: console

     python throughput_benchmark.py <path_to_model> <device_name>(default: CPU)

.. tab-item:: C++ :sync: cpp

  .. code-block:: console

     throughput_benchmark <path_to_model> <device_name>(default: CPU)

To run the sample, you need to specify a model. You can get a model specific for your inference task from one of model repositories, such as TensorFlow Zoo, HuggingFace, or TensorFlow Hub.

Example ++++++++++++++++++++

1. Download a pre-trained model.

2. You can convert it by using:

   .. tab-set::

   .. tab-item:: Python :sync: python

   Copy

     .. code-block:: python
   
        import openvino as ov
   
        ov_model = ov.convert_model('./models/googlenet-v1')
        # or, when model is a Python model object
        ov_model = ov.convert_model(googlenet-v1)
   

   .. tab-item:: CLI :sync: cli

   Copy

     .. code-block:: console
   
        ovc ./models/googlenet-v1
   

3. Perform benchmarking, using the googlenet-v1 model on a CPU:

   .. tab-set::

   .. tab-item:: Python :sync: python

   Copy

     .. code-block:: console
   
        python throughput_benchmark.py ./models/googlenet-v1.xml
   

   .. tab-item:: C++ :sync: cpp

   Copy

     .. code-block:: console
   
        throughput_benchmark ./models/googlenet-v1.xml
   

Sample Output ####################

.. tab-set::

.. tab-item:: Python :sync: python

  The application outputs performance results.

  .. code-block:: console

     [ INFO ] OpenVINO:
     [ INFO ] Build ................................. <version>
     [ INFO ] Count:          2817 iterations
     [ INFO ] Duration:       10012.65 ms
     [ INFO ] Latency:
     [ INFO ]     Median:     13.80 ms
     [ INFO ]     Average:    14.10 ms
     [ INFO ]     Min:        8.35 ms
     [ INFO ]     Max:        28.38 ms
     [ INFO ] Throughput: 281.34 FPS

.. tab-item:: C++ :sync: cpp

  The application outputs performance results.

  .. code-block:: console

     [ INFO ] OpenVINO:
     [ INFO ] Build ................................. <version>
     [ INFO ] Count:      1577 iterations
     [ INFO ] Duration:   15024.2 ms
     [ INFO ] Latency:
     [ INFO ]        Median:     38.02 ms
     [ INFO ]        Average:    38.08 ms
     [ INFO ]        Min:        25.23 ms
     [ INFO ]        Max:        49.16 ms
     [ INFO ] Throughput: 104.96 FPS

Additional Resources ####################

• :doc:Integrate the OpenVINO™ Runtime with Your Application <../../../openvino-workflow/running-inference>
• :doc:Get Started with Samples <get-started-demos>
• :doc:Using OpenVINO Samples <../openvino-samples>
• :doc:Convert a Model <../../../openvino-workflow/model-preparation/convert-model-to-ir>
• Throughput Benchmark Python Sample on Github <https://github.com/openvinotoolkit/openvino/blob/master/samples/python/benchmark/throughput_benchmark/README.md>__
• Throughput Benchmark C++ Sample on Github <https://github.com/openvinotoolkit/openvino/blob/master/samples/cpp/benchmark/throughput_benchmark/README.md>__