Credit Risk Model Serving

Introduction

Model serving is an exciting part of AI/ML. All of our previous work was building to this phase where we can actually serve loan predictions.

So what role does Feast play in model serving? We've already seen that Feast can "materialize" data from the training offline store to the serving online store. This comes in handy because many models need contextual features at inference time.

With this example, we can imagine a scenario something like this:

1. A bank customer submits a loan application on a website.
2. The website backend requests features, supplying the customer's ID as input.
3. The backend retrieves feature data for the ID in question.
4. The backend submits the feature data to the model to obtain a prediction.
5. The backend uses the prediction to make a decision.
6. The response is recorded and made available to the user.

With online requests like this, time and resource usage often matter a lot. Feast facilitates quickly retrieving the correct feature data.

In real-life, some of the contextual feature data points could be requested from the user, while others are retrieved from data sources. While outside the scope of this example, Feast does facilitate retrieving request data, and joining it with feature data. (See Request Source).

In this notebook, we request feature data from the online store for a small batch of users. We then get outcome predictions from our trained model. This notebook is a continuation of the work done in the previous notebooks; it comes as the step after 03_Credit_Risk_Model_Training.ipynb.

Setup

The following code assumes that you have read the example README.md file, and that you have setup an environment where the code can be run. Please make sure you have addressed the prerequisite needs.

# Imports
import os
import joblib
import json
import requests
import warnings
import pandas as pd

from feast import FeatureStore

# ingnore warnings
warnings.filterwarnings(action="ignore")

# Load the model
model = joblib.load("rf_model.pkl")

Query Feast Online Server for Feature Data

Here, we show two different ways to retrieve data from the online feature server. The first is using the Python requests library, and the second is using the Feast Python SDK.

We can use the Python requests library to request feature data from the online feature server (that we deployed in notebook 02_Deploying_the_Feature_Store.ipynb). The request takes the form of an HTTP POST command sent to the server endpoint (url). We request the data we need by supplying the entity and feature information in the data payload. We also need to specify an application/json content type in the request header.

# ID examples
ids = [18, 764, 504, 454, 453, 0, 1, 2, 3, 4, 5, 6, 7, 8]

# Submit get_online_features request to Feast online store server
response = requests.post(
    url="http://localhost:6566/get-online-features",
    headers = {'Content-Type': 'application/json'},
    data=json.dumps({
        "entities": {"ID": ids},
        "features": [
            "data_a:duration",
            "data_a:credit_amount",
            "data_a:installment_commitment",
            "data_a:checking_status",
            "data_b:residence_since",
            "data_b:age",
            "data_b:existing_credits",
            "data_b:num_dependents",
            "data_b:housing"
        ]
    })
)

The response is returned as JSON, with feature values for each of the IDs.

# Show first 1000 characters of response
response.text[:1000]

As the response data comes in JSON format, there is a little formatting required to organize the data into a dataframe with one record per row (and features as columns).

# Inspect the response
resp_data = json.loads(response.text)

# Transform JSON into dataframe
records = pd.DataFrame(
    columns=resp_data["metadata"]["feature_names"], 
    data=[[r["values"][i] for r in resp_data["results"]] for i in range(len(ids))]
)
records.head(3)

Now, let's see how we can do the same with the Feast Python SDK. Note that we instantiate our FeatureStore object with the configuration that we set up in 02_Deploying_the_Feature_Store.ipynb, by pointing to the ./Feature_Store directory.

# Instantiate FeatureStore object
store = FeatureStore(repo_path="./Feature_Store")

# Retrieve features
records = store.get_online_features(
    entity_rows=[{"ID":v} for v in ids],
    features=[
        "data_a:duration",
        "data_a:credit_amount",
        "data_a:installment_commitment",
        "data_a:checking_status",
        "data_b:residence_since",
        "data_b:age",
        "data_b:existing_credits",
        "data_b:num_dependents",
        "data_b:housing"        
    ]
).to_df()

records.head(3)

Get Predictions from the Model

Now we can request predictions from our trained model.

For convenience, we output the predictions along with the implied loan designations. Remember that these are predictions on loan outcomes, given context data from the loan application process. Since we have access to the actual class outcomes, we display those as well to see how the model did.|

# Get predictions from the model
preds = model.predict(records)

# Load labels
labels = pd.read_parquet("Feature_Store/data/labels.parquet")

# Show preds
pd.DataFrame({
    "ID": ids,
    "Prediction": preds,
    "Loan_Designation": ["bad" if i==0.0 else "good" for i in preds],
    "True_Value": labels.loc[ids, "class"]
})

It's important to remember that the model's predictions are like educated guesses based on learned patterns. The model will get some predictions right, and other wrong. With the example records above, it looks like the model did pretty good! An AI/ML team's task is generally to make the model's predictions as useful as possible in helping the organization make decisions (for example, on loan approvals).

In this case, we have a baseline model. While not ready for production, this model has set a low bar by which other models can be measured. Teams can also use a model like this to help with early testing, and with proving out things like pipelines and infrastructure before more sophisticated models are available.

We have used Feast to query the feature data in support of model serving. The next notebook, 05_Credit_Risk_Cleanup.ipynb, cleans up resources created in this and previous notebooks.