Prediction Metrics

When the problem type is classification, creating prediction metrics after training a ML model would result in an interface that provides decile analysis summary for each class on buckets of model confidence scores. Such an analysis helps in testing the model's ability to predict the intended outcome. Ideally, the number of predictions made should increase from top to bottom with most of the predictions made under the last few buckets and the precision for a class should also increase from top to bottom.

Steps to Create Prediction Metrics

Creating the prediction metrics require three values - actual value, predicted probability, and predicted value. So, the first step is to create a batch prediction from the feature group that was used for training:

After setting up the fields as shown in the image above, click on the "Next: Configure Output" button. Select the output type as "Feature Group", provide a name for the feature group table, and click on the "Next: Advanced Options" button.

For this case, there is no need to configure the advanced options, so click on the "Start Batch Prediction" button:

The system will start performing the batch prediction:

Once the batch prediction is complete, you can go to the feature group tab on the left navigation and click on the batch prediction feature group from the list of feature groups. Next, click on the "Materialized Data" option and enter "prediction" into the "Filter Columns By Name" field. You will see the three required values we needed to create our prediction metrics:

The next step is to create the prediction metrics. Click on the "Monitoring" tab and then on the "Batch" sub-tab. Select the "Prediction Metrics" option and finally click on the "Create Prediction Metric" button. Select the appropriate values for the fields as shown below and click on the "Create" button.

Once the prediction metric is processed, click on the prediction metric record from the list of prediction metrics. Finally, click on the "Metrics" button under the Prediction Metrics Versions. The decile analysis summary will be displayed: