Invoking LLMs

Large language models are invoked through the RouteLLM API. RouteLLM exposes an OpenAI-compatible /v1/chat/completions endpoint, so you call it with the standard openai Python package and it allows you to:

• Pass messages to the model and receive generated responses.
• Return structured responses using response_format with a JSON schema
• Pass images
• Stream responses token-by-token
• Invoke the LLM of your choice, or let route-llm pick the best model for the request

Setup

• Install the OpenAI client:

python3 -m pip install openai

• RouteLLM accepts the same Abacus.AI API key that ApiClient uses, so no separate key is needed. Outside the platform, pass the API key from the API Keys Dashboard to ApiClient. Inside the platform (notebooks, AI workflows, prediction operators, pipelines) call ApiClient() with no arguments and the key is picked up automatically.

• Create the RouteLLM client from the ApiClient credentials. The RouteLLM base URL is your organization's own API host, which the SDK resolves for you through client.get_api_endpoint().predict_endpoint. This works for both self-serve and Enterprise Platform organizations. Do not hard-code https://routellm.abacus.ai: it only serves self-serve organizations and rejects enterprise API keys with a 403 "Incorrect hostname" error.

from abacusai import ApiClient
from openai import OpenAI

client = ApiClient('API_KEY')  # Outside the platform: pass your API key. Inside the platform: ApiClient() with no key.

llm = OpenAI(
    base_url=f"{client.get_api_endpoint().predict_endpoint}/v1",  # Your organization's RouteLLM host
    api_key=client.api_key,  # Reuses the Abacus.AI API key
)

Models are referenced by their RouteLLM model ID, for example gpt-4o, claude-sonnet-4-6 or gemini-2.5-flash. Use route-llm to have the best model selected automatically. The full list is in the Supported Models section, or run llm.models.list() for the live list and pricing.

Basic Invocation

response = llm.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You should answer all questions with a single word."},
        {"role": "user", "content": "What is the capital of Greece?"},
    ],
)

# Response:
print(response.choices[0].message.content)

• system message: These are the instructions that the model will follow
• user message: This is the actual message that the model receives from the user
• model: The LLM that will be used to return the response

JSON Response Example

import json

response = llm.chat.completions.create(
    model="gpt-4o",
    messages=[
        # {"role": "system", "content": "OPTIONAL, but good to have"},
        {"role": "user", "content": "In this course, you will learn about car batteries, car doors, and car suspension system"},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "learning_objectives",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "learning_objectives": {
                        "type": "array",
                        "items": {"type": "string"},
                        "description": "A list of learning objectives",
                    }
                },
                "required": ["learning_objectives"],
                "additionalProperties": False,
            },
        },
    },
)
learning_objectives = json.loads(response.choices[0].message.content)
learning_objectives

• response_format: A standard JSON Schema. With "strict": True the model is guaranteed to return exactly this shape, which is what you want when the output is parsed by code. See Structured Output for the full reference.

Sending Images

import base64

with open('test.png', 'rb') as fo:
    encoded_data = base64.b64encode(fo.read()).decode('utf-8')

response = llm.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What can you see in the image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{encoded_data}",
                    },
                },
            ],
        },
    ],
)
print(response.choices[0].message.content)

Streaming

Set stream=True to receive the response as it is generated:

stream = llm.chat.completions.create(
    model="route-llm",
    messages=[{"role": "user", "content": "Explain quantum computing in simple terms."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Using RouteLLM inside the platform

Code that runs inside Abacus.AI (AI workflow nodes, prediction operators, pipelines, notebooks) needs no key handling at all. ApiClient() picks up the credentials of the runtime, and client.api_key passes them straight through to RouteLLM:

from abacusai import ApiClient
from openai import OpenAI

client = ApiClient()  # No API key needed inside the platform
llm = OpenAI(
    base_url=f"{client.get_api_endpoint().predict_endpoint}/v1",  # Your organization's RouteLLM host
    api_key=client.api_key,
)

Remember to add openai to package_requirements when you register an agent or prediction operator so the package is available at runtime. The AI Workflows examples use this pattern.