The research team at Abacus.AI makes fundamental contributions to the field of AI/ML that might have an impact in the short or long run and does applied research that has benefits in practice. The areas we have been focusing on include automated machine learning, meta-learning, data augmentation, deep learning optimization, and fairness/debiasing in deep learning.
The team has an impressive list of publications in top-tier conferences.
Read on for detailed explanations of our active areas of research and product development, which are focused on solving hard problems faced by today's organizations.
Large Language Models and Foundation Models
Code Generation - In recent years, researchers have made remarkable advances in generative language modeling tasks. GPT-3 first showed that large language models perform well on tasks beyond which they were explicitly trained, such as unscrambling words, performing 3-digit arithmetic, and even generating code. This area of research shows no signs of slowing down, with the latest models containing over 500 billion trainable parameters and are trained with over 500 billion tokens.

At Abacus.AI, we believe that the next frontier for language models is in AI-assisted data science. We are developing generative models that can interpret and then execute commands given in natural language (such as, “Train a 10-layer MLP and plot the validation accuracy over time”). To accomplish this, we have curated the first ever “data science” dataset for fine-tuning a language model, consisting of over 1 billion lines of python code. By fine-tuning the model on our own Abacus.AI API’s, the language model allows anyone to easily use Abacus’s platform to perform exploratory data analysis, visualize data, and train powerful machine learning models. Visit our homepage to request a demo.
Large Language Models - In addition to code generation, large language models are incredibly versatile. At Abacus.AI, we use large language models for in-demand industry applications such as named entity recognition (NER), text extraction and classification, search, question answering, and sentiment analysis. We offer these solutions by starting with state-of-the-art large language models, and then performing a custom fine-tuning procedure in order to maximize the accuracy of the desired task.
Time-Series Forecasting - Time-series forecasting is a ubiquitous problem in the industry, used for problems such as forecasting inventory demand and predicting stock prices. While statistical approaches such as ARIMA and Prophet are widely used, deep learning models such as transformers are getting increasingly more popular. However, many high-impact forecasting problems are either low-data, or low signal-to-noise ratio, and many forecastable datasets have standard attributes that carry over to other datasets.

With this intuition in mind, Abacus.AI is developing a radically new approach to time-series forecasting. Inspired by recent innovations in meta-learning and Bayesian inference on tabular data, we are designing the world’s first foundation model for time-series forecasting. To achieve this, we pretrain a transformer on a mix of real-world and synthetic datasets, across a variety of different forecasting tasks, resulting in a state-of-the-art forecasting model that can run inference on a new dataset in less than a second.
Deep recommender systems
Deep Recommender Systems - Recommender systems are used heavily across e-commerce, social media, and entertainment companies such as Amazon, YouTube, and Netflix. While early approaches included k-nearest-neighbor or matrix factorization, deep learning is becoming the dominant paradigm in recommender systems due to the scale and complexity of modern recommender system datasets. At Abacus.AI, we have developed multiple deep learning approaches to recommender systems for use cases such as session-based item recommendation, personalized search query results, and personalized related item recommendations. Our deep learning models are based on state-of-the-art techniques such as two-tower models, which use separate embedding layers for the user history as well as the items, and then makes recommendations based on cosine similarity. We also use recurrent neural networks such as long short-term memory (LSTM) to capture temporal relationships in each user’s history.
Meta-Learning to Ensure Performance - In order to ensure our deep learning models achieve the highest possible performance, we use meta-learning to rapidly find the best algorithm and hyperparameters for new datasets. For more details, see our published research paper below. Our system also compares the best approaches to well-tuned baselines such as P3Alpha, RP3Beta, SLIM-Elastic-Net, and Co-Clustering, to make sure that we achieve the best possible performance.
Vector Matching Engine - The size of the datasets in large retail and e-commerce companies can reach millions of items and billions of total interactions, and deployed models often face millions of requests per day. In order to keep the latency of each request low, Abacus.AI uses our custom-designed, state-of-the-art vector matching engine. Powered by GPUs, our vector matching engine is able to scale up to 10,000+ queries per second, with less than 20ms latency per request.
Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, John Dickerson, Colin White
Workshop at AutoML-Conf 2022
Training with Less Data

While organizations today might have large amounts of data, their datasets tend to be noisy, incomplete and imbalanced. This results in data scientists and engineers spending most of their precious time pre-processing, cleaning, and featurizing the data. These efforts are often insufficient, and deep learning techniques routinely fail on sparse datasets. Organizations are then forced to use classical machine learning techniques that require enormous amounts of manual feature engineering. At Abacus.AI, we are actively pursuing the following research areas that will enable training on less data.

Meta-Learning - Deep learning models typically require training with a large number of training samples. On the other hand, humans learn concepts and skills much more quickly and efficiently. We only need a few examples to tell lions apart from cats. Meta-learning is a sub-field of machine learning that aims to teach models how to learn. We hope to build on the work outlined by Model-Agnostic Meta-Learning (MAML) and first-order Meta-Learning Algorithms. The MAML algorithm provides a good initialization of a model’s parameters to achieve an optimal fast learning on a new task with only a small number of gradient steps while avoiding overfitting that may happen when using a small dataset. Our service uses principles of meta-learning to create robust models even when you have a small number of training examples.
Generative Models for Dataset Augmentation - Dataset augmentation is a technique to synthetically expand a training dataset by applying a wide array of domain-specific transformations. This is a particularly useful tool for small datasets, and it is even shown to be effective on large datasets like Imagenet. While it is a standard tool used in training supervised deep learning models, it requires extensive domain knowledge, and the transformations must be designed and tested carefully. Over the last 2 years, Generative Adversarial Networks (GANs) have been used successfully for dataset augmentation in various domains including computer vision, anomaly detection, and forecasting. The use of GANs makes dataset augmentation possible even with little or no domain-specific knowledge. Fundamentally, GANs learn how to produce data from a dataset that is indistinguishable from the original data. However, there are some practical issues with using GANs, and training a GAN is notoriously difficult. GANs have been a very active area of research, and several new types of GANs including Wasserstein GANs and MMD GANs address some of these issues. Recently, there has also been some work on domain-agnostic GAN implementation for dataset augmentation. At Abacus.AI, we are innovating on the state-of-the-art GAN algorithms that can perform well on noisy and incomplete datasets. We have innovated on Data Augmentation Generative Adversarial Networks to create synthetic datasets that can be combined with original datasets to create more robust models. The demo on our homepage is based on GANs. Check out this blog post to see how it works.
Combining Neural Nets with Logic Rules/Specifications - The cognitive process of human beings indicates that people learn not only from concrete examples (as deep neural nets do) but also from different forms of general knowledge and rich experiences. It’s difficult to encode human intention to guide the models to capture desired patterns. In fact, most enterprise systems today are rule-based. Experts have encoded rules based on tribal knowledge from their domains. ML models that are built to replace these rule-based systems often struggle to beat them on accuracy, especially when there is sparse data. At Abacus.AI, we are working on preserving expert knowledge by developing hybrid systems that combine logic rules with neural nets. While there is some recent research in this area, including a recent paper by DeepMind that lays the groundwork for a general-purpose, constraint-driven AI, it is still nascent. Most research papers don’t address building these hybrid models at scale or incorporating multiple rules into the models. Abacus.AI is working on a service that allows developers and data scientists to specify multiple knowledge rules along with training data to develop accurate models. For example, there may be a rule that ‘dog owners tend to like buying dog toys’ in a recommender system or a constraint that a learned dynamic system must be consistent with physical law. Our publication in this area combines first-order logic constraints with conventional supervised learning.
Transfer Learning - Transfer learning is a machine learning technique that allows us to reuse policies from one domain or dataset on a related domain or dataset. By using transfer learning, we enable organizations to train models in a simulated environment and apply them in the real world. State-of-the-art language and vision modeling techniques typically pre-train on a large dataset, then either use fine-tuning or transfer learning to train a custom model on the target dataset. Abacus.AI packages and extends the state-of-the-art transfer learning techniques that result in the most performant models. As part of our service, we plan to package pre-trained language and vision models. We’ll also make it easy to fine-tune those models or apply transfer learning to adapt them for a custom task.
Naveen Sundar Govindarajulu and Colin White
NeurIPS Workshop on Knowledge Representation to ML, 2019
AI-Assisted ML

Deep learning has seen great success across a wide variety of domains. The best neural architectures are often carefully constructed by seasoned deep learning experts in each domain. For example, years of experimentation have shown how to arrange bidirectional transformers to work well for language tasks and dilated separable convolutions for image tasks. A relatively new sub-field of deep-learning deals with automated machine learning, or as we prefer to call it: AI-assisted machine learning. The fundamental idea is that AI will create a first pass of the deep-learning model given a use-case or a dataset. Developers/data scientists can then either use that model directly or fine-tune. We are conducting cutting-edge research in the main pillars of AI-Assisted ML: hyperparameter optimization (HPO) and neural architecture search (NAS).

Hyperparameter optimization

When developing a deep learning model, there are many knobs and dials to tune that depend on the specific task and dataset at hand. For example, setting the learning rate too high can prevent the algorithm from converging. Setting the learning rate too low can cause the algorithm to get stuck at a local minimum. There are countless other hyperparameters such as the number of epochs, batch size, momentum, regularization, shape, and size of the neural network. These hyperparameters are all dependent on each other and interact in intricate ways, so finding the best hyperparameters for a given dataset is an extremely difficult and highly nonconvex optimization problem.

Randomly testing different sets of hyperparameters may eventually find a decent solution but could take years of computation time. Efficiently tuning deep learning hyperparameters is an active area of research. Five years ago, the best algorithms weren’t much better than random search. Now algorithms are capable of orders of magnitude speedups. At Abacus.AI, we use state-of-the-art HPO while training all our models.

Neural Architecture Search

Neural architecture search (NAS) is a rapidly developing area of research in which the process of choosing the best architecture is automated.

At Abacus.AI, we are using NAS to both fine-tune proven deep network paradigms, and learn novel architectures for new domains. Our goal is to empower data scientists and developers to create custom, production-grade models in days, not months. See this blog post to read about our method, BANANAS, which combines Bayesian optimization with neural predictors to achieve state-of-the-art performance. Since making our code open-source, dozens of developers have forked our repository, and two independent research groups have confirmed that it achieves state-of-the-art performance on NAS-Bench-101. BANANAS has even been cited in survey papers on NAS.

We are also actively conducting fundamental research on the theory of NAS. Recently, we studied local search for NAS - a simple yet effective approach. We showed experimentally that local search gives state-of-the-art performance on smaller benchmark NAS search spaces, but performs worse than random search on extremely large search spaces. Motivated by this stark contrast, we gave a complete theoretical characterization of local search. Our theoretical results confirm that local search performs well on smaller search spaces and when the search space exhibits locality.

Finally, we are conducting formal studies on the building blocks of NAS, including the architecture encoding. In most NAS algorithms, the neural architectures must be passed as input to the algorithm using some encoding. For example, we might encode the neural architectures using an adjacency matrix. Our recent work shows that this encoding can have a substantial impact on the final result of the NAS algorithm. We conduct a set of experiments with eight different encodings with various NAS algorithms. Our results lay out recommendations for the best encodings to use in different settings within NAS.

Colin White, Sam Nolen, Yash Savani
ICML Workshop on AutoML 2020
Colin White, Willie Neiswanger, Sam Nolen, Yash Savani
Selected for spotlight presentation | NeurIPS 2020
Yang Liu, Jeremy Bernstein, Markus Meister, Yisong Yue
ICML 2021
Colin White, Arber Zela, Binxin Ru, Yang Liu, Frank Hutter
Selected as a contributed talk | ICLR Workshop on Neural Architecture Search
Bias and Explainability in Neural Nets

Bias is one of the most important issues in machine learning today. Deep learning models are being deployed in high-stakes scenarios today more than ever, and most of these models are found to exhibit prejudices. For example, the New York Times reported that the majority of facial recognition apps used by law enforcement agencies exhibit bias. They cited a study concluding that facial recognition technology is ten times more likely to falsely identify people of color, women and older people. There has been considerable research in mitigating these biases, with dozens of definitions of bias and algorithms to decrease the level of bias. The majority of fair algorithms are in-processing algorithms, which take as input a training dataset and then train a new, fairer model from scratch. However, this is not always practical. For example, recent neural networks such as XLNet or GPT-3 can take weeks to train and are very expensive. Additionally, for some applications, the full training set may no longer be available due to regulatory or privacy requirements. At Abacus.AI, we are designing new post-hoc methods, which take as input a pretrained model and a smaller validation dataset, and then debias the model through fine-tuning or post-processing. We have designed three new techniques which work for applications with tabular data or structured data. See our blog post for more information.

In addition to bias, we are actively working on explainability in neural networks. Business Analysts and subject matter experts within organizations are often frustrated when dealing with deep learning models. These models can appear to be black boxes that generate predictions which humans can’t explain. Over the last two years, there has been considerable research in explainability in AI. This has resulted in the release of an open-source tool, LIME, which measures the responsiveness of a model’s outputs to perturbations in its inputs. Then there’s SHAP (SHapley Additive exPlanations), a game-theoretic approach to explain the output of any machine learning model. Google has introduced Testing with Concept Activation Vectors (TCAV), a technique that may be used to generate insights and hypotheses. Google Brain’s scientists also explored attribution of predictions to input features in their 2016 paper, Axiomatic attribution for deep neural networks. Our efforts in this area build on these techniques to create a cloud microservice that will explain model predictions and determine if models exhibit bias.

Yash Savani, Colin White, Naveen Sundar Govindarajulu
NeurIPS 2020
Naveen Sundar Govindarajulu and Colin White
NeurIPS Workshop on Robust AI in Financial Services, 2019