Skip to main content

Azure Databricks User Connector Setup Guide

This guide describes how to configure a secure connection from Abacus.AI to data hosted on the Azure Databricks platform using the Databricks JDBC driver and Databricks OAuth.

When the connector is created with Import RBAC enabled, every end user signs in to Databricks with their own identity, and all queries run with that user's Databricks permissions.

Prerequisites​

Before you begin, gather the following:

  1. Account admin access to the Databricks account console (https://accounts.azuredatabricks.net)
  • Required to create the OAuth application used for user sign-in (see Step 1).
  1. Databricks personal access token
  • In your Databricks workspace, go to User Settings and generate a personal access token.
  1. JDBC connection details for your Databricks cluster
  • Navigate to Compute, select your cluster, open Advanced options, and then the JDBC tab to view connection info.
  • Note: Do not include https:// in the Database Server URL/IP. Use only the server hostname or address.

Step 1: Create an OAuth app in Databricks​

Abacus.AI authenticates each end user through the Databricks account-level OAuth flow, so you must register Abacus.AI as a custom OAuth application (an "app connection") in your Databricks account console.

  1. Log in to the Databricks account console as an account admin, open Settings, and select the App connections tab.
  2. Click Add connection.
Databricks account console App connections tab with the Add connection button
  1. Application Name: enter a name that identifies the integration, for example Abacus.AI.
  2. Redirect URLs: enter the Abacus.AI callback URL exactly as shown, on its own line:
https://abacus.ai/oauth/callback
  1. Access scopes: using the Add scope dropdown, make sure the application grants the scopes that Abacus.AI requests:
    • all-apis — required. Abacus.AI requests this scope so it can list catalogs, schemas and tables and run queries through the Databricks JDBC driver on behalf of the signed-in user.
    • offline_access — required. Issues the refresh token that keeps a user signed in between queries.
    • openid, email and profile are added by Databricks automatically for the ID token; leave them in place.
  2. Client secret: keep Generate a client secret selected. Abacus.AI connects as a confidential client and needs the secret. Copy the secret as soon as it is displayed — Databricks does not show it again, and it cannot be changed after the connection is created.
  3. Single-use Refresh Tokens: leave Enable single-use refresh tokens (refresh token rotation) unselected. Abacus.AI stores the refresh token issued at sign-in; with rotation enabled that stored token stops working after the first refresh and users are asked to sign in again.
  4. Leave the Access token TTL and Refresh token TTL values at their defaults unless your organization requires different lifetimes.
Databricks Add connection form showing application name, redirect URLs, access scopes and client secret options
  1. Save the connection, then copy the Client ID and Client secret.
  2. Copy your Databricks Account ID from the account console (available in the user menu in the top-right corner of the console).

Note: Databricks can take up to 30 minutes to apply a new or updated OAuth application.

With this app in place, Abacus.AI sends users to https://accounts.azuredatabricks.net/oidc/accounts/<account-id>/v1/authorize to sign in. If your Databricks account is hosted on AWS or GCP, the same app connection is created in the corresponding account console (accounts.cloud.databricks.com or accounts.gcp.databricks.com).

Step 2: Set up the connector in Abacus.AI​

  1. Open the Abacus.AI Connected Services Dashboard: https://abacus.ai/app/profile/connected_services

  2. Click Add New Connector, choose ODBC/JDBC, and select Databricks JDBC driver from the Driver dropdown.

  3. Turn on the Import RBAC toggle. Note: Please enable the Import RBAC toggle; otherwise, users will use the JDBC/ODBC connection and will have access to more data than expected.

  4. Enter the OAuth details of the Databricks app connection created in Step 1:

    • Client ID: the client ID of the Databricks app connection.
    • Client Secret: the client secret generated for that app connection.
    • Account ID: your Databricks account ID.
    • Hosted Platform: Azure. Select AWS or GCP if your Databricks account is hosted there, or Custom for a custom-hosted platform — Custom additionally requires the OAuth authorization and token URLs of your deployment.

  5. Fill in all Databricks configuration details gathered above (server hostname/address, JDBC info, token, etc.) and click Save.

  6. Click Verify and wait for the connector to be validated. If an error appears, follow the message to troubleshoot and correct your configuration.

Build a ChatLLM project on your connector​

Create the project​

  1. From the Abacus.AI logo, go to the Projects page.
  2. Create a new project.
  3. Select ChatLLM – Custom LLM Chat.
  4. Enter a name for your project.
  5. Choose Skip to project dashboard.

Train the model​

  1. In the left toolbar, open Model and click Train Model (top right).
  2. For Structured data source, select External Databases from the dropdown.
  3. Choose the connector you created in the steps above and add the tables to use for testing.
  4. Click Train Model.
  5. Once training completes, open Models and select your model.

Deploy and test​

  1. Click Create a new deployment.
  2. Select Offline Batch + Realtime, then click Next.
  3. Enter an end‑user friendly deployment name and click Deploy.
  4. Return to your model (Model > your model name). When the deployment is Active, click the deployment name.
  5. Under Deployment, open the Prediction Dash.
  6. Click Go to Abacus.AI chat to test your bot.

End‑user sign‑in​

Enter your prompt/question. Each user is asked to sign in to Databricks once, and to approve the scopes requested by the Abacus.AI app connection. Queries then run with that user's own Databricks permissions.

Troubleshooting​

  • Sign-in fails with a redirect URI or invalid client error

    • Confirm the app connection lists https://abacus.ai/oauth/callback exactly, with no trailing slash or extra path.
    • Confirm the Client ID, Client Secret and Account ID in the connector match the Databricks app connection.
    • A newly created or edited app connection can take up to 30 minutes to become active.

  • Sign-in fails with an invalid or insufficient scope error

    • Confirm both all-apis and offline_access are listed under the app connection's access scopes.

  • Users are asked to sign in repeatedly

    • Confirm single-use refresh tokens (refresh token rotation) is disabled on the app connection.
    • Check the refresh token TTL configured for the app connection.

  • Verification fails on connector save

    • Re‑check server hostname/address formatting (no https://).
    • Confirm the personal access token is valid and not expired.
    • Ensure JDBC details (from Compute > Advanced options > JDBC) match the active cluster.

  • Unable to access tables after verification

    • Confirm the user or service principal has the necessary Databricks and data permissions (RBAC).
    • Re‑verify the connector to surface detailed error messages.