ONNX Runtime on Azure Kubernetes Service

Deploying Facial Emotion Recognition (FER+) using Docker Images for ONNX Runtime with TensorRT

This example shows how to deploy an image classification neural network using ONNX Runtime on GPU compute SKUs in Azure. This example makes use of the Facial Expression Recognition (FER) dataset and Open Neural Network eXchange format (ONNX) on the Azure Machine Learning platform.

Tutorial Roadmap:

Obtain an external ONNX model for facial emotion recognition (FER+) from the ONNX model zoo
Register our model in our Azure Machine Learning workspace
Write a scoring file and environment file to evaluate our model with ONNX Runtime
Build a container image using the ONNX Runtime + TensorRT base image from Microsoft Container Registry (MCR)
Deploy to the cloud using an AKS cluster with GPU and use it to make predictions using ONNX Runtime Python APIs

Note: You can also use the same notebook and code to deploy to a CPU cluster in addition to your GPU cluster. ONNX Runtime APIs remain unchanged across hardware endpoints. The default option for a secondary deployment with CPU is False, but you can change the variable below to True for a performance comparison.

python

deploy_with_cpu = False

0. Prerequisites

Throughout this tutorial, we will be referring to ONNX, a neural network exchange format used to represent deep learning models. With ONNX, AI developers can more easily move models between state-of-the-art tools (CNTK, PyTorch, Caffe, MXNet, TensorFlow) and choose the combination that is best for them. ONNX is developed and supported by a community of partners including Microsoft AI, Facebook, and Amazon. For more information, explore the ONNX website and open source files.

ONNX Runtime is the runtime engine that enables evaluation of trained machine learning (traditional ML and Deep Learning) models with high performance and low resource utilization. We use the CPU version of ONNX Runtime in this tutorial, but will soon be releasing an additional tutorial for deploying this model using ONNX Runtime GPU.

Install Azure ML SDK and create a new workspace

If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, please follow the Azure ML configuration notebook to set up your environment.

Install additional packages needed for this Notebook

You need to install the popular plotting library matplotlib, the image manipulation library opencv, and the onnx library in the conda environment where Azure Machine Learning SDK is installed.

(myenv) $ pip install matplotlib onnx opencv-python

Debugging tip: Make sure that to activate your virtual environment (myenv) before you re-launch this notebook using the jupyter notebook comand. Choose the respective Python kernel for your new virtual environment using the Kernel > Change Kernel menu above. If you have completed the steps correctly, the upper right corner of your screen should state Python [conda env:myenv] instead of Python [default].

1. Obtain a model from the ONNX Model Zoo

For more information on the Facial Emotion Recognition (FER+) model, you can explore the notebook explaining how to deploy FER+ with ONNX Runtime on an ACI Instance.

python

# urllib is a built-in Python library to download files from URLs

# Objective: retrieve the latest version of the ONNX Emotion FER+ model files from the
# ONNX Model Zoo and save it in the same folder as this tutorial

import urllib.request

onnx_model_url = "https://www.cntk.ai/OnnxModels/emotion_ferplus/opset_7/emotion_ferplus.tar.gz"

urllib.request.urlretrieve(onnx_model_url, filename="emotion_ferplus.tar.gz")

# the ! magic command tells our jupyter notebook kernel to run the following line of 
# code from the command line instead of the notebook kernel

# We use tar and xvcf to unzip the files we just retrieved from the ONNX model zoo

!tar xvzf emotion_ferplus.tar.gz

2. Registering your model with Azure ML

Load Azure ML workspace

We begin by instantiating a workspace object from the existing workspace created earlier in the configuration notebook.

python

# Check core SDK version number
import azureml.core

print("SDK version:", azureml.core.VERSION)

python

from azureml.core import Workspace

# read existing workspace from config.json
ws = Workspace.from_config()

print(ws.name, ws.location, ws.resource_group, sep = '\n')

Register your ONNX model with Azure ML

python

model_dir = "emotion_ferplus" # replace this with the location of your model files

# leave as is if it's in the same folder as this notebook

python

from azureml.core.model import Model

# register the new model from local folder
model = Model.register(model_path = model_dir + "/" + "model.onnx",
                       model_name = "onnx_emotion",
                       tags = {"onnx": "demo"},
                       description = "FER+ emotion recognition CNN from ONNX Model Zoo",
                       workspace = ws)

# Alternative: uncomment the line below and point to the model file in the workspace's model registry
# model = Model(name="onnx_emotion", workspace=ws)

Optional: Displaying your registered models

This step is not required, so feel free to skip it.

python

models = ws.models
for name, m in models.items():
    print("Name:", name,"\tVersion:", m.version, "\tDescription:", m.description, m.tags)

3. Specify our Score and Environment Files

We are now going to deploy our ONNX Model on AKS with inference in ONNX Runtime. We begin by writing a score.py file, which will help us run the model in our Azure Kubernetes Cluster, and then specify our environment by writing a yml file. You will also notice that we import the onnxruntime library to do runtime inference on our ONNX models (passing in input and evaluating out model's predicted output). More information on the API and commands can be found in the ONNX Runtime documentation.

Write Score File

A score file is what tells our Azure cloud service what to do. After initializing our model using azureml.core.model, we start an ONNX Runtime inference session to evaluate the data passed in on our function calls.

python

%%writefile score.py
import json
import numpy as np
import onnxruntime
import sys
import os
from azureml.core.model import Model
import time

def init():
    global session, input_name, output_name
    model = Model.get_model_path(model_name = 'onnx_emotion')
    
    # Load the model in onnx runtime to start the session    
    session = onnxruntime.InferenceSession(model, None)
    input_name = session.get_inputs()[0].name
    output_name = session.get_outputs()[0].name 
    
def run(input_data):
    '''Purpose: evaluate test input in Azure Cloud using onnxruntime.
        We will call the run function later from our Jupyter Notebook 
        so our azure service can evaluate our model input in the cloud. '''

    try:
        # load in our data, convert to readable format
        data = np.array(json.loads(input_data)['data']).astype('float32')
        
        # pass input data to do model inference with ONNX Runtime
        start = time.time()
        r = session.run([output_name], {input_name : data})
        end = time.time()
        
        result = emotion_map(postprocess(r[0]))
        
        result_dict = {"result": result,
                      "time_in_sec": [end - start]}
    except Exception as e:
        result_dict = {"error": str(e)}
    
    return json.dumps(result_dict)

def emotion_map(classes, N=1):
    """Take the most probable labels (output of postprocess) and returns the 
    top N emotional labels that fit the picture."""
    
    emotion_table = {'neutral':0, 'happiness':1, 'surprise':2, 'sadness':3, 
                     'anger':4, 'disgust':5, 'fear':6, 'contempt':7}
    
    emotion_keys = list(emotion_table.keys())
    emotions = []
    for i in range(N):
        emotions.append(emotion_keys[classes[i]])
    return emotions

def softmax(x):
    """Compute softmax values (probabilities from 0 to 1) for each possible label."""
    x = x.reshape(-1)
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)

def postprocess(scores):
    """This function takes the scores generated by the network and 
    returns the class IDs in decreasing order of probability."""
    prob = softmax(scores)
    prob = np.squeeze(prob)
    classes = np.argsort(prob)[::-1]
    return classes

Write Environment File

python

from azureml.core.conda_dependencies import CondaDependencies 

myenv = CondaDependencies.create(pip_packages=["numpy", "azureml-core"])

with open("myenv.yml","w") as f:
    f.write(myenv.serialize_to_string())

4. Create the Container Image

This step will take a few minutes if the container image is built for the first time.

python

from azureml.core.image import ContainerImage
from azureml.core.model import Model

gpu_image_config = ContainerImage.image_configuration(execution_script = "score.py",
                                                  runtime = "python",
                                                  conda_file = "myenv.yml",
                                                  description = "Emotion ONNX Runtime container",
                                                  tags = {"demo": "onnx"})

# Use the ONNX Runtime + TensorRT base image
gpu_image_config.base_image = "mcr.microsoft.com/azureml/onnxruntime:latest-tensorrt"

gpu_image = ContainerImage.create(name = "onnximage.trt",
                              # this is the model object
                              models = [model],
                              image_config = gpu_image_config,
                              workspace = ws)

# Alternative: Re-use an image that you have already built from the workspace image registry
# gpu_image = ContainerImage(name = "onnximage.trt", workspace = ws)

gpu_image.wait_for_creation(show_output = True)

In case you need to debug your code, the next line of code accesses the log file.

python

print(gpu_image.image_build_log_uri)

This step is for a container image to target a CPU cluster.

python

if deploy_with_cpu:
    
    cpu_image_config = ContainerImage.image_configuration(execution_script = "score.py",
                                                      runtime = "python",
                                                      conda_file = "myenv.yml",
                                                      description = "Emotion ONNX Runtime container",
                                                      tags = {"demo": "onnx"})

    # use the ONNX Runtime CPU base image
    cpu_image_config.base_image = "mcr.microsoft.com/azureml/onnxruntime:latest"

    cpu_image = ContainerImage.create(name = "onnximage.cpu",
                                      models = [model],
                                      image_config = cpu_image_config,
                                      workspace = ws)
    
    # Alternative: Re-use an image that you have already built from the workspace image registry
    # cpu_image = ContainerImage(name = "onnximage.cpu", workspace = ws)

    cpu_image.wait_for_creation(show_output = True)

In case you need to debug your code, the next line of code accesses the log file.

python

if deploy_with_cpu:
    print(cpu_image.image_build_log_uri)

We're all done specifying what we want our virtual machine to do. Let's configure and deploy our container image.

5. Deploy the container image

Create a Azure Kubernetes Service (AKS) Compute Target for GPU.

python

# create the AKS service with GPU nodes
from azureml.core import Workspace
from azureml.core.compute import AksCompute, ComputeTarget
from azureml.core.compute_target import ComputeTargetException
from azureml.core.webservice import Webservice, AksWebservice
from azureml.core.image import Image
from azureml.core.model import Model

gpu_aks_name = 'your-gpu-cluster'

try:
    gpu_aks_target = ComputeTarget(workspace = ws, name=gpu_aks_name)
    print('Found existing cluster, use it.')
except ComputeTargetException:
    # Create a configuration (can also provide parameters to customize)
    prov_config = AksCompute.provisioning_configuration(vm_size='Standard_NC6', location='East US2' )
    # Create the cluster
    gpu_aks_target = ComputeTarget.create(workspace = ws, 
                                  name = gpu_aks_name, 
                                  provisioning_configuration = prov_config)

python

%%time
gpu_aks_target.wait_for_completion(show_output = True)
print(gpu_aks_target.provisioning_state)
print(gpu_aks_target.provisioning_errors)

(Optional) Configure another target for a CPU AKS cluster.

python

# CPU-cell

if deploy_with_cpu: 
    
    cpu_aks_name = 'your-cpu-cluster'

    try:
        cpu_aks_target = ComputeTarget(workspace = ws, name=cpu_aks_name)
        print('Found existing cluster, use it.')

    except ComputeTargetException:
        # Create a configuration (can also provide parameters to customize)
        prov_config = AksCompute.provisioning_configuration(vm_size='Standard_D3', location='East US2' )
        # Create the cluster
        cpu_aks_target = ComputeTarget.create(workspace = ws, 
                                      name = cpu_aks_name, 
                                      provisioning_configuration = prov_config)
    
    cpu_aks_target.wait_for_completion(show_output = True)
    print(cpu_aks_target.provisioning_state)
    print(cpu_aks_target.provisioning_errors)

Set the web service configuration. In this case, we're using the default.

python

gpu_aks_config = AksWebservice.deploy_configuration()

if deploy_with_cpu:
    cpu_aks_config = AksWebservice.deploy_configuration()

Create an AKS service for our GPU AKS cluster.

python

%%time
gpu_aks_service_name ='gpu-aks-service'

gpu_aks_service = Webservice.deploy_from_image(workspace = ws, 
                                           name = gpu_aks_service_name,
                                           image = gpu_image,
                                           deployment_config = gpu_aks_config,
                                           deployment_target = gpu_aks_target)
gpu_aks_service.wait_for_deployment(show_output = True)
print(gpu_aks_service.state)

python

if gpu_aks_service.state != 'Healthy':
    # run this command for debugging.
    print(gpu_aks_service.get_logs())
    
    # If your deployment fails, make sure to delete your aci_service before trying again!
    # gpu_aks_service.delete()

(Optional) Create an AKS service for our CPU AKS cluster.

python

# CPU-cell

if deploy_with_cpu:
    cpu_aks_service_name ='cpu-aks-service'

    cpu_aks_service = Webservice.deploy_from_image(workspace = ws, 
                                               name = cpu_aks_service_name,
                                               image = cpu_image,
                                               deployment_config = cpu_aks_config,
                                               deployment_target = cpu_aks_target)
    
    cpu_aks_service.wait_for_deployment(show_output = True)
    print(cpu_aks_service.state)

python

if deploy_with_cpu:
    
    if cpu_aks_service.state != 'Healthy':
        # run this command for debugging.
        print(cpu_aks_service.get_logs())

        # If your deployment fails, make sure to delete your aks_service before trying again!
        # cpu_aks_service.delete()

Success!

If you've made it this far, you've deployed a working AKS Cluster with a facial emotion recognition model running in the cloud using Azure ML. Congratulations!

Let's see how well our model deals with our test images.

Testing and Evaluation

Useful Helper Functions

We preprocess and postprocess our data (see score.py file) using the helper functions specified in the ONNX FER+ Model page in the Model Zoo repository.

python

def emotion_map(classes, N=1):
    """Take the most probable labels (output of postprocess) and returns the 
    top N emotional labels that fit the picture."""
    
    emotion_table = {'neutral':0, 'happiness':1, 'surprise':2, 'sadness':3, 
                     'anger':4, 'disgust':5, 'fear':6, 'contempt':7}
    
    emotion_keys = list(emotion_table.keys())
    emotions = []
    for i in range(N):
        emotions.append(emotion_keys[classes[i]])
    return emotions

def softmax(x):
    """Compute softmax values (probabilities from 0 to 1) for each possible label."""
    x = x.reshape(-1)
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)

def postprocess(scores):
    """This function takes the scores generated by the network and 
    returns the class IDs in decreasing order of probability."""
    prob = softmax(scores)
    prob = np.squeeze(prob)
    classes = np.argsort(prob)[::-1]
    return classes

Try classifying your own images!

python

# Preprocessing functions take your image and format it so it can be passed
# as input into our ONNX model
import cv2

def rgb2gray(rgb):
    """Convert the input image into grayscale"""
    return np.dot(rgb[...,:3], [0.299, 0.587, 0.114])

def resize_img(img):
    """Resize image to MNIST model input dimensions"""
    img = cv2.resize(img, dsize=(64, 64), interpolation=cv2.INTER_AREA)
    img.resize((1, 1, 64, 64))
    return img

def preprocess(img):
    """Resize input images and convert them to grayscale."""
    if img.shape == (64, 64):
        img.resize((1, 1, 64, 64))
        return img
    
    grayscale = rgb2gray(img)
    processed_img = resize_img(grayscale)
    return processed_img

python

# Replace the following string with your own path/test image
# Make sure your image is square and the dimensions are equal (i.e. 100 * 100 pixels or 28 * 28 pixels)

# Any PNG or JPG image file should work
# Make sure to include the entire path with // instead of /

# e.g. your_test_image = "C:/Users/vinitra.swamy/Pictures/face.png"

your_test_image = "<path to file>"

import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import json
import os

if your_test_image != "<path to file>":
    img = mpimg.imread(your_test_image)
    plt.subplot(1,3,1)
    plt.imshow(img, cmap = plt.cm.Greys)
    print("Old Dimensions: ", img.shape)
    img = preprocess(img)
    print("New Dimensions: ", img.shape)
else:
    img = None

python

if img is None:
    print("Add the path for your image data.")
else:
    input_data = json.dumps({'data': img.tolist()})

    try:
        r_gpu = json.loads(gpu_aks_service.run(input_data))
        gpu_result = r_gpu['result'][0]
        gpu_time_ms = np.round(r_gpu['time_in_sec'][0] * 1000, 2)
        
        if deploy_with_cpu:
            r_cpu = json.loads(cpu_aks_service.run(input_data))
            cpu_result = r_cpu['result'][0]
            cpu_time_ms = np.round(r_cpu['time_in_sec'][0] * 1000, 2)
        else:
            cpu_result, cpu_time_ms = "", ""
            
    except Exception as e:
        print(str(e))

    plt.figure(figsize = (16, 6))
    plt.subplot(1,8,1)
    plt.axhline('')
    plt.axvline('')
    plt.text(x = -10, y = -40, s = "Model prediction: ", fontsize = 14)
    plt.text(x = -10, y = -25, s = "Inference time (GPU, CPU): ", fontsize = 14)
    plt.text(x = 100, y = -40, s = "          "+str(gpu_result)+ "     "+ str(cpu_result), fontsize = 14)
    plt.text(x = 100, y = -25, s = "          "+str(gpu_time_ms) + " ms" + "     " +str(cpu_time_ms)+ " ms", fontsize = 14)
    plt.text(x = -10, y = -10, s = "Model Input image: ", fontsize = 14)
    plt.imshow(img.reshape((64, 64)), cmap = plt.cm.gray)

CPU vs. GPU + TensorRT Performance comparison

python

if deploy_with_cpu:
    x = ['GPU \n(TensorRT)', 'CPU']
    time = [gpu_time_ms, cpu_time_ms]

    x_pos = [i for i, _ in enumerate(x)]

    bar_graph = plt.barh(x_pos, time, color='green')
    bar_graph[1].set_color('grey')
    plt.ylabel("ONNX Runtime Deployment Type")
    plt.xlabel("Inference Time (ms)")
    plt.title("ONNX Runtime CPU vs GPU Performance Comparison")

    plt.yticks(x_pos, x)

    plt.show()

Clean up our workspace

python

# remember to delete your service after you are done using it!

gpu_aks_service.delete()

if deploy_with_cpu:
    cpu_aks_service.delete()

Conclusion

Congratulations!

In this tutorial, you have:

familiarized yourself with ONNX Runtime inference and the pretrained models in the ONNX model zoo
understood a state-of-the-art convolutional neural net image classification model (FER+ in ONNX) and deployed it in the Azure ML cloud
ensured that your deep learning model is working perfectly (in the cloud) on test data, and checked it against some of your own!

Next steps:

If you have not already, check out another interesting ONNX application that lets you set up a state-of-the-art handwritten image classification model (MNIST) on Azure! This tutorial deploys a pre-trained ONNX Computer Vision model for handwritten digit classification in an Azure ML Container Instance.
Contribute to our open source ONNX repository on github and/or add to our ONNX model zoo