Connect your BigQuery to Abacus.AI

To integrate BigQuery with Abacus.AI, you need to set up the connector and provide the necessary permissions.

• Setup Instructions
• How to Use the BigQuery Connector
• Troubleshooting and FAQ

Setup Instructions

1. Gather Information:

   • PROJECT ID: You can find the project ID by following the instructions at Google API Support. The project ID is a unique identifier for your Google Cloud project.
     
     
     
     

   • DATASET ID (OPTIONAL): To get the Dataset ID, browse the dataset in the BigQuery console or follow the instructions at BigQuery Dataset Metadata. The Dataset ID is the identifier for the specific dataset you want to connect to.
     
     
     
     

   • Note: The Dataset ID in the UI is pre-pended with the Project ID, i.e., Project ID: Dataset ID. Capture the Dataset ID by taking the string after the ":".
     

   • Important: If the Dataset ID is not set, tables must be referenced along with the dataset name, e.g., dataset_name.table_name.

2. Access Abacus.AI Connected Services Dashboard:

   • Go to the Abacus.AI Connected Services Dashboard. This is where you manage all your connected services.
   • At the top-right of the page, click "Add New Connector" and select "BigQuery".
     

3. Enter Connector Information:

   • Fill in the information captured in step 1 into the BigQuery connector window. This includes the Project ID and Dataset ID. Click Save
     
     
     
     

4. Set BigQuery Data Viewer Permissions:

   • You will get a pop-up with instructions to add BigQuery Data Viewer permissions to the service account in BigQuery. This step is crucial for granting the necessary permissions for data access.
     
     
     
     

5. Configure Permissions in BigQuery:

   • Set up the permissions in BigQuery by sharing the dataset with the service account as instructed in the step above and hit Verify.
     
     
     
     
     
     

6. Verify Connector Setup:

   • You should see a screen that says your connector has been verified. Your BigQuery connector is now set up and ready to use.
     
     
     
     

Optional: Allowing Write Access to BigQuery

Allowing write access back to BigQuery is always optional. If you choose to enable this, follow the instructions below based on whether you provided a dataset ID or not.

If You Provided a Dataset ID:

1. Navigate to Dataset:

   • Go to your dataset in the BigQuery Console.

2. Share Dataset:

   • Click "Share Dataset".

3. Add Roles:

   • Add the "BigQuery Data Viewer" role to the service account provided in the configuration instructions.
   • To allow Abacus.AI to write back to BigQuery, also add the "BigQuery Data Editor" and "BigQuery User" roles to the service account.

If You Did Not Provide a Dataset ID:

1. Navigate to IAM:

   • Go to the IAM Console for your project.

2. Add Service Account:

   • Click "Add" on the top of the page.

3. Add Roles:

   • Add the service account provided in the configuration instructions as a new principal with roles "BigQuery Data Viewer" and "BigQuery User".
   • To allow Abacus.AI to write back to BigQuery, also add the "BigQuery Data Editor" role to the service account.

How to Use the BigQuery Connector

Once the BigQuery connector is set up, you can fetch data to train models in Abacus.AI.

1. Create a New Project:

   • Create a new project and select the use case, then go to the "Datasets" tab and click "Create Dataset".
     
     
     
     

2. Create New Dataset:

   • Click on "Create New".
     
     
     
     

3. Name the Dataset:

   • Name the dataset, select the data type 'List of documents', and click "Continue".
     
     
     
     

4. Read from External Service:

   • Choose "Read from External Service" and select your BigQuery connector under "Connected Application Connectors".

5. Enter Dataset Details:

   • Enter the details for the BigQuery dataset you want to use.

6. Configure Schema Mapping:

   • After the dataset is uploaded, configure the schema mapping and proceed to train models with the data.
     
     
     
     

Troubleshooting and FAQ for the BigQuery Connector

What if the dataset contains views and the views are querying tables in a different dataset that Abacus.AI is not granted permission to read from?

• There are two suggested solutions:
  

  • Solution 1: Grant the service account access to the specific tables these views reference. Provide the "BigQuery Data Viewer" role to the specific underlying tables for the service account.
    

  • Solution 2: Create an Authorized View (https://cloud.google.com/bigquery/docs/authorized-views) which will grant the view access to the underlying data, allowing Abacus.AI to query from the view without having to grant permission to the table(s) which make up the view.