Abacus.AI - Evaluating Predictions

Overview Use Cases Customer Churn Prediction Time Series Anomaly Detection Event Anomaly Detection Cloud Spend Alerts Personalized Promotions Predictive Modeling Real-Time Forecasting Financial Metrics Forecasting Demand Forecasting Feature Group Requirements Training Models Evaluating Predictions Predictions Cumulative Forecasting Text extraction and classification NLP Powered Search Sentiment Analysis Finetuned LLM ChatLLM Language Detection Image Classification & Detection Object Detection Clustering Timeseries Clustering Sales and Revenue Forecasting Predictive Lead Scoring Personalized Search Personalized Recommendations Related Items Model Drift and Monitoring Tensorflow with Vector Matching Custom Python Model Data Ingestion Streaming Feature Store Vector Store AI Agents Named Entity Recognition Guidelines Optimization Connectors Authentication Getting Started with the Python SDK API Documentation Chat Bot API Search How to

Problem type: Forecasting

Forecasting is the process of training models on historical data and then using them to predict future observations. It is known as time-series forecasting because it comes under time series analysis and there's a timestamp associated with every data point for this problem type. It is arguably one of the hardest problem types because of the fact that the future is completely unknown and uncertain and we only have the luxury of the past to estimate the uncertain future. Here's a quick evaluation recipe to quickly understand the quality of the forecasts:

Quick Evaluation Recipe

Compare metric scores with the baseline scores to see if there's an improvement or not

Forecasting

Start with accuracy and point-wise accuracy, if there is a significant increase in both as compared to baseline then the model is learning useful patterns. The percentage increase varies greatly from case to case and is dependent on a variety of factors
Check cov-acc graph (left one on metric dashboard), if it looks dispersed then your cov would be a bit high (0.2 to 0.5 in general) otherwise it should be low (<0.15 in general)
Check the breakdown of items by quartile of accuracy and make sure that accuracies for the item ids in the Top 25% are high and the ones in the Worst 25% are not too low (>40%)

You can click on the option "Subset on Prediction Dashboard" to populate the IDs to the prediction dashboard and verify if the results make sense. The green line (P50) and the actual predictions should be close for the Top 25% Item IDs but must not be overlapping for most of the forecast timestamps. If they are far, you might need to consider adding more data for those item ids. Similarly, check for the Worst 25%

Finally, verify if the P50 graph on the prediction dashboard lies in the middle of P90 and P10 graphs for at least 80% of the timestamps

If all of the above things look good then the model is good to go and you have trained a world-class deep learning-based forecasting model in just a few minutes with only a few clicks.