Showcase Deconstructed Research
The research team at Abacus.AI makes fundamental contributions to the field of AI/ML that might have an impact in the short or long run and does applied research that has benefits in practice. The areas we have been focusing on include automated machine learning, meta-learning, data augmentation, deep learning optimization, and fairness/debiasing in deep learning.
The team already has an impressive list of publications in top-tier conferences and workshops in just over one year of its conception.
Read on for detailed explanations of our active areas of research and product development, which are focused on solving hard problems faced by today's organizations.
Training with Less Data

While organizations today might have large amounts of data, their datasets tend to be noisy, incomplete and imbalanced. This results in data scientists and engineers spending most of their precious time pre-processing, cleaning, and featurizing the data. These efforts are often insufficient, and deep learning techniques routinely fail on sparse datasets. Organizations are then forced to use classical machine learning techniques that require enormous amounts of manual feature engineering. At Abacus.AI, we are actively pursuing the following research areas that will enable training on less data.

Meta-Learning - Deep learning models typically require training with a large number of training samples. On the other hand, humans learn concepts and skills much more quickly and efficiently. We only need a few examples to tell lions apart from cats. Meta-learning is a sub-field of machine learning that aims to teach models how to learn. We hope to build on the work outlined by Model-Agnostic Meta-Learning (MAML) and first-order Meta-Learning Algorithms. The MAML algorithm provides a good initialization of a model’s parameters to achieve an optimal fast learning on a new task with only a small number of gradient steps while avoiding overfitting that may happen when using a small dataset. Our service uses principles of meta-learning to create robust models even when you have a small number of training examples.
Generative Models for Dataset Augmentation - Dataset augmentation is a technique to synthetically expand a training dataset by applying a wide array of domain-specific transformations. This is a particularly useful tool for small datasets, and it is even shown to be effective on large datasets like Imagenet. While it is a standard tool used in training supervised deep learning models, it requires extensive domain knowledge, and the transformations must be designed and tested carefully. Over the last 2 years, Generative Adversarial Networks (GANs) have been used successfully for dataset augmentation in various domains including computer vision, anomaly detection, and forecasting. The use of GANs makes dataset augmentation possible even with little or no domain-specific knowledge. Fundamentally, GANs learn how to produce data from a dataset that is indistinguishable from the original data. However, there are some practical issues with using GANs, and training a GAN is notoriously difficult. GANs have been a very active area of research, and several new types of GANs including Wasserstein GANs and MMD GANs address some of these issues. Recently, there has also been some work on domain-agnostic GAN implementation for dataset augmentation. At Abacus.AI, we are innovating on the state-of-the-art GAN algorithms that can perform well on noisy and incomplete datasets. We have innovated on Data Augmentation Generative Adversarial Networks to create synthetic datasets that can be combined with original datasets to create more robust models. The demo on our homepage is based on GANs. Check out this blog post to see how it works.
Combining Neural Nets with Logic Rules/Specifications - The cognitive process of human beings indicates that people learn not only from concrete examples (as deep neural nets do) but also from different forms of general knowledge and rich experiences. It’s difficult to encode human intention to guide the models to capture desired patterns. In fact, most enterprise systems today are rule-based. Experts have encoded rules based on tribal knowledge from their domains. ML models that are built to replace these rule-based systems often struggle to beat them on accuracy, especially when there is sparse data. At Abacus.AI, we are working on preserving expert knowledge by developing hybrid systems that combine logic rules with neural nets. While there is some recent research in this area, including a recent paper by DeepMind that lays the groundwork for a general-purpose, constraint-driven AI, it is still nascent. Most research papers don’t address building these hybrid models at scale or incorporating multiple rules into the models. Abacus.AI is working on a service that allows developers and data scientists to specify multiple knowledge rules along with training data to develop accurate models. For example, there may be a rule that ‘dog owners tend to like buying dog toys’ in a recommender system or a constraint that a learned dynamic system must be consistent with physical law. Our publication in this area combines first-order logic constraints with conventional supervised learning.
Transfer Learning - Transfer learning is a machine learning technique that allows us to reuse policies from one domain or dataset on a related domain or dataset. By using transfer learning, we enable organizations to train models in a simulated environment and apply them in the real world. State-of-the-art language and vision modeling techniques typically pre-train on a large dataset, then either use fine-tuning or transfer learning to train a custom model on the target dataset. Abacus.AI packages and extends the state-of-the-art transfer learning techniques that result in the most performant models. As part of our service, we plan to package pre-trained language and vision models. We’ll also make it easy to fine-tune those models or apply transfer learning to adapt them for a custom task.
Differentiable Functions for Combining First-order Constraints with Deep Learning via Weighted Proof Tracing
Naveen Sundar Govindarajulu and Colin White
NeurIPS Workshop on Knowledge Representation to ML, 2019
AI-Assisted ML

Deep learning has seen great success across a wide variety of domains. The best neural architectures are often carefully constructed by seasoned deep learning experts in each domain. For example, years of experimentation have shown how to arrange bidirectional transformers to work well for language tasks and dilated separable convolutions for image tasks. A relatively new sub-field of deep-learning deals with automated machine learning, or as we prefer to call it: AI-assisted machine learning. The fundamental idea is that AI will create a first pass of the deep-learning model given a use-case or a dataset. Developers/data scientists can then either use that model directly or fine-tune. We are conducting cutting-edge research in the main pillars of AI-Assisted ML: hyperparameter optimization (HPO) and neural architecture search (NAS).

Hyperparameter optimization

When developing a deep learning model, there are many knobs and dials to tune that depend on the specific task and dataset at hand. For example, setting the learning rate too high can prevent the algorithm from converging. Setting the learning rate too low can cause the algorithm to get stuck at a local minimum. There are countless other hyperparameters such as the number of epochs, batch size, momentum, regularization, shape, and size of the neural network. These hyperparameters are all dependent on each other and interact in intricate ways, so finding the best hyperparameters for a given dataset is an extremely difficult and highly nonconvex optimization problem.

Randomly testing different sets of hyperparameters may eventually find a decent solution but could take years of computation time. Efficiently tuning deep learning hyperparameters is an active area of research. Five years ago, the best algorithms weren’t much better than random search. Now algorithms are capable of orders of magnitude speedups. At Abacus.AI, we use state-of-the-art HPO while training all our models.

Neural Architecture Search

Neural architecture search (NAS) is a rapidly developing area of research in which the process of choosing the best architecture is automated.

At Abacus.AI, we are using NAS to both fine-tune proven deep network paradigms, and learn novel architectures for new domains. Our goal is to empower data scientists and developers to create custom, production-grade models in days, not months. See this blog post to read about our method, BANANAS, which combines Bayesian optimization with neural predictors to achieve state-of-the-art performance. Since making our code open-source, dozens of developers have forked our repository, and two independent research groups have confirmed that it achieves state-of-the-art performance on NAS-Bench-101. BANANAS has even been cited in survey papers on NAS.

We are also actively conducting fundamental research on the theory of NAS. Recently, we studied local search for NAS - a simple yet effective approach. We showed experimentally that local search gives state-of-the-art performance on smaller benchmark NAS search spaces, but performs worse than random search on extremely large search spaces. Motivated by this stark contrast, we gave a complete theoretical characterization of local search. Our theoretical results confirm that local search performs well on smaller search spaces and when the search space exhibits locality.

Finally, we are conducting formal studies on the building blocks of NAS, including the architecture encoding. In most NAS algorithms, the neural architectures must be passed as input to the algorithm using some encoding. For example, we might encode the neural architectures using an adjacency matrix. Our recent work shows that this encoding can have a substantial impact on the final result of the NAS algorithm. We conduct a set of experiments with eight different encodings with various NAS algorithms. Our results lay out recommendations for the best encodings to use in different settings within NAS.

BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
Colin White, Willie Neiswanger, Yash Savani
AAAI 2021
Local Search is State of the Art for Neural Architecture Search Benchmarks
Colin White, Sam Nolen, Yash Savani
ICML Workshop on AutoML 2020
A Study on Encodings for Neural Architecture Search
Colin White, Willie Neiswanger, Sam Nolen, Yash Savani
Selected for spotlight presentation | NeurIPS 2020
Bias and Explainability in Neural Nets

Bias is one of the most important issues in machine learning today. Deep learning models are being deployed in high-stakes scenarios today more than ever, and most of these models are found to exhibit prejudices. For example, the New York Times reported that the majority of facial recognition apps used by law enforcement agencies exhibit bias. They cited a study concluding that facial recognition technology is ten times more likely to falsely identify people of color, women and older people. There has been considerable research in mitigating these biases, with dozens of definitions of bias and algorithms to decrease the level of bias. The majority of fair algorithms are in-processing algorithms, which take as input a training dataset and then train a new, fairer model from scratch. However, this is not always practical. For example, recent neural networks such as XLNet or GPT-3 can take weeks to train and are very expensive. Additionally, for some applications, the full training set may no longer be available due to regulatory or privacy requirements. At Abacus.AI, we are designing new post-hoc methods, which take as input a pretrained model and a smaller validation dataset, and then debias the model through fine-tuning or post-processing. We have designed three new techniques which work for applications with tabular data or structured data. See our blog post for more information.

In addition to bias, we are actively working on explainability in neural networks. Business Analysts and subject matter experts within organizations are often frustrated when dealing with deep learning models. These models can appear to be black boxes that generate predictions which humans can’t explain. Over the last two years, there has been considerable research in explainability in AI. This has resulted in the release of an open-source tool, LIME, which measures the responsiveness of a model’s outputs to perturbations in its inputs. Then there’s SHAP (SHapley Additive exPlanations), a game-theoretic approach to explain the output of any machine learning model. Google has introduced Testing with Concept Activation Vectors (TCAV), a technique that may be used to generate insights and hypotheses. Google Brain’s scientists also explored attribution of predictions to input features in their 2016 paper, Axiomatic attribution for deep neural networks. Our efforts in this area build on these techniques to create a cloud microservice that will explain model predictions and determine if models exhibit bias.

Intra-Processing Methods for Debiasing Neural Networks
Yash Savani, Colin White, Naveen Sundar Govindarajulu
NeurIPS 2020
DECO: Debiasing through Compositional Optimization of Machine Learning Models
Naveen Sundar Govindarajulu and Colin White
NeurIPS Workshop on Robust AI in Financial Services, 2019