Skip to main content

Creating Pipelines

Overview​

Abacus.AI allows you to create pipelines to orchestrate and automate tasks within your projects.

Use Cases​

Pipelines can be used to automate various workflows, such as:

  • Refreshing data
  • Retraining models
  • Deploying models

Pipelines are extremely flexible and free-form, allowing you to orchestrate virtually any workflow.

Pipeline Components​

Pipelines consist of three main components:

1. Python Functions​

Contains the core logic and functionality of your pipeline operations.

2. Pipeline Steps​

Defines how Python functions are transformed and executed within the pipeline.

3. Pipeline​

The complete workflow created by connecting all pipeline steps together.

Creating a Pipeline​

Step 1: Navigate to Pipelines​

Click on Pipelines after navigating inside a project. This ensures the pipeline is connected to your project.

Pipelines Navigation

Step 2: Select a Template​

Click on Create Pipeline and choose from multiple example templates to get started quickly.

Create Pipeline Templates

Abacus.AI will open a notebook that you can use to create and configure your pipeline.

Managing Your Pipeline​

Once your pipeline is set up, you can manage it directly from the UI:

  • Setup Refresh Schedule: Automate pipeline execution on a schedule
  • Change Input Mappings: Modify input parameters and configurations
  • Edit Pipeline: Opens a notebook for making changes to the pipeline logic
Pipeline Management

Monitoring Pipeline Runs​

At the bottom of the pipeline screen, you can view historical pipeline runs. If you used a logger or print statements in your code, you can view the output in the Logs section.

Pipeline Run History

Handling Pipeline Failures​

The Detail page allows you to reset the pipeline in case of failure. You can selectively reset only the steps that failed, rather than rerunning the entire pipeline.

Pipeline Detail and Reset