Abacus.AI - Evaluating Predictions

Depending upon the data you pointed us to and the inputs/outputs specification you provided, a classification or a regression model is trained under Predictive Modeling use case. Let's walk through the evaluation process for both types of models:

Problem type: Classification

As the name suggests, the classification model takes the input and classifies it as a class label. For example, a binary classifier trained on a set of dog images would classify an image as dog or not dog. With that, let me introduce a quick evaluation recipe for those who might have a company to run and who just want to get a high-level idea of the quality of the model in a few seconds:

Classification

Quick Evaluation Recipe

Compare metric scores with the baseline scores to see if there's an improvement or not
Start with AUC, make sure it's between 0.6 and 0.95 (higher the AUC the better)
Check accuracy, anything above 60% means something useful is being learned
Check precision and recall, one of them should be more than 0.5 for useful results
Finally, go to the prediction dashboard and see if the results generated by the model matched with actual class labels for different classes

If all of the above things look good then the model is good to go and you have trained a world-class deep learning model in just a few minutes with only a few clicks.

Problem type: Regression

In classification we were popping out a class label as the result, now in regression we will predict a continuous outcome variable using a set of input features (columns in your dataset). For example, instead of classifying someone as young or old you might be interested in predicting their age, this is where you would be interested in framing the problem as regression and getting an age value as output. Now that this problem type is clear, let's move to a quick evaluation recipe for those who might have their businesses to run or people who just want to get a high-level idea of the quality of the model in a few seconds:

Regression

Quick Evaluation Recipe

Compare metric scores with the baseline scores to see if there's an improvement or not
Check if WAPE lies between 0 and 2. Lower is better
Check R2, any value between 0.6 to 0.9 is considered good. The higher the value the better
Finally, go to the prediction dashboard and see if the results generated by the model comes close to the actual values. Check about 20-30 data points

If all of the above things look good then the model is good to go and you have trained a world-class deep learning-based regression model in just a few minutes with only a few clicks.