OpenAI Chat Completions API

The Chat Completions endpoint (/v1/chat/completions) is fully OpenAI-compatible. Point the OpenAI SDK — or any OpenAI-compatible client — at the RouteLLM base URL and use it as a drop-in replacement, including streaming, tool calling, multimodal inputs, and structured outputs.

This is the recommended endpoint for most use cases and works with every text generation model RouteLLM supports.

Base URLs

The base URL depends on your organization type:

• Self-Serve Organizations: https://routellm.abacus.ai/v1
• Enterprise Platform: https://<workspace>.abacus.ai/v1

Replace <workspace> with your specific workspace identifier for enterprise deployments. To find your correct base URL, refer to the RouteLLM API page.

Authentication

All API requests require authentication using a RouteLLM API key. Include it in the request header:

Authorization: Bearer <your_api_key>

You can obtain your API key from the Abacus.AI platform.

Supported Models

The Chat Completions endpoint works with all RouteLLM text generation models. Specify a model ID explicitly or use route-llm to let the system pick the best one. See the full catalog on the RouteLLM API overview, or call GET /v1/models for the live list. Models other than those in the supported list are also supported as a proxy and redirected to their respective clients.

Request Parameters

1. Required Parameters

messages (array, required)

A list of messages comprising the conversation so far. Each message must be an object with the following structure:

• role (string, required): The role of the message sender. Must be one of:

  • user: Messages from the user/end-user
  • assistant: Previous responses from the AI assistant
  • system: System-level instructions that guide the assistant's behavior

• content (string or array, required): The content of the message. Can be:

  • A string for text-only messages
  • An array for multimodal content (text and images)

2. Optional Parameters

model (string, optional)

The ID of the model to use. Can be either a text generation model or an image generation model, depending on the modalities parameter. If omitted, defaults to route-llm.

Note: The model names shown in the Supported Models section use a human-readable format (e.g. flux-2-pro), but the actual model ID accepted by the API may differ (e.g. flux2_pro). Call GET /v1/models to retrieve the exact id string for each model and use that value in your requests.

Text Generation Models: route-llm, gpt-5.4, claude-sonnet-4-6, gemini-3.1-pro, etc.

Image Generation Models: flux-2-pro, flux-kontext, dall-e, ideogram, recraft, imagen, nano-banana-pro, seedream

Examples: route-llm, gpt-5.4, flux-2-pro, seedream

max_tokens (integer, optional)

The maximum number of tokens to generate in the chat completion. The total length of input tokens and generated tokens is limited by the model's context window.

Default: Model-dependent

temperature (number, optional)

What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.

Default: 1.0

Recommended values:

• 0.0-0.3: For factual, deterministic responses
• 0.7-1.0: For creative, varied responses
• 1.0-2.0: For highly creative, diverse outputs

top_p (number, optional)

An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered.

Default: 1.0

Range: 0.0 to 1.0

stream (boolean, optional)

If set to true, partial message deltas will be sent as data-only server-sent events as they become available. The stream will terminate by a data: [DONE] message.

Default: false

stop (string or array, optional)

Up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence.

Example: stop": ["Human:", "AI:"]

presence_penalty (number, optional)

Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.

Default: 0.0

frequency_penalty (number, optional)

Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.

Default: 0.0

response_format (object, optional)

An object specifying the format that the model must output. Two types are supported:

1. JSON Object Mode

"response_format": {
  "type": "json_object"
}

Constrains the model to output valid JSON. You must also instruct the model to produce JSON via a system or user message.

2. JSON Schema Mode

"response_format": {
  "type": "json_schema",
  "json_schema": {
    "name": "your_schema_name",
    "schema": {
      "type": "object",
      "properties": {
        "field_name": { "type": "string" },
        "count": { "type": "integer" }
      },
      "required": ["field_name", "count"],
      "additionalProperties": false
    }
  }
}

JSON Schema mode constrains the model to output JSON that strictly conforms to the provided schema. No system or user message instructing the model to produce JSON is required — the schema itself enforces the format. The json_schema object requires:

Field	Type	Required	Description
name	string	Yes	A name identifier for the schema
schema	object	Yes	The JSON Schema definition
strict	boolean	No	Whether to enforce strict schema adherence (see below)

The inner schema object requires:

Field	Type	Required	Description
type	string	Yes	JSON Schema type (e.g., "object")
properties	object	Yes	Property definitions for the object
required	array	Yes	List of required property names
additionalProperties	boolean	Yes	Whether to allow extra properties beyond those defined

strict mode

The strict field controls how rigidly the model follows the schema:

• strict: true — The schema is treated as a law. The model is guaranteed to produce output that exactly matches the schema. Every field in required will be present, no extra fields are added, and types are enforced precisely.
• strict: false (default) — The schema is treated as a suggestion. The model will try to follow it, but may deviate in edge cases (e.g., omitting optional fields or adding extra context).

Use strict: true whenever your downstream code parses the response programmatically.

Important: When using response_format: { type: "json_object" }, you must instruct the model to produce JSON via a system or user message. This is not required for json_schema mode — the schema enforces the format automatically.

tools (array, optional)

A list of tools the model may call. Each tool is an object with:

• type: Must be "function".
• function: Object with:
  • name (string, required): Name of the function the model can call.
  • description (string, optional): Description of the function for the model.
  • parameters (object, optional): JSON Schema for the function parameters (OpenAI-style).

Example:

"tools": [
  {
    "type": "function",
    "function": {
      "name": "get_current_weather",
      "description": "Get the current weather in a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": { "type": "string", "description": "City and state, e.g. San Francisco, CA" },
          "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
        },
        "required": ["location"]
      }
    }
  }
]

tool_choice (string or object, optional)

Controls whether the model can call tools. Values:

• "none": Do not call any tool (default when tools is omitted).
• "auto": Model may choose to call one or more tools (default when tools is provided).
• {"type": "function", "function": {"name": "get_current_weather"}}: Force the model to call the named function.

Default: "auto" when tools is provided.

abacus_tools (array of strings, optional)

An Abacus.AI extension to the OpenAI schema: a list of names of tools registered in your Abacus.AI organization that the server executes on the model's behalf. Unlike tools, you do not supply a schema and you do not run anything — the server resolves each name to its registered definition, runs the tool when the model calls it, feeds the result back to the model, and returns only the model's final answer.

"abacus_tools": ["lookup_order_status", "search_inventory"]

abacus_tools is additive: it can be combined with your own tools in the same request. See Abacus Tools for the full behavior, limits, and examples.

modalities (array, optional)

Specifies the output type for the request.

• ["text"] — Text generation (default).
• ["image"] — Image generation. See Image Analysis & Generation.
• ["text", "audio"] — Text + audio output (TTS). See Audio Capabilities.

audio (object, optional)

Required when modalities includes "audio". Specifies the voice and output format for audio generation. See Audio Capabilities for full parameter reference, available voices, and examples.

Response Format

Non-Streaming Response

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1677858242,
  "model": "route-llm",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The meaning of life is..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 20,
    "total_tokens": 30
  }
}

Streaming Response

When stream: true, the API returns a stream of server-sent events. Each event is a JSON object:

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1677858242,"model":"route-llm","choices":[{"index":0,"delta":{"role":"assistant","content":"The"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1677858242,"model":"route-llm","choices":[{"index":0,"delta":{"content":" meaning"},"finish_reason":null}]}

data: [DONE]

Response Fields

• id: A unique identifier for the chat completion
• object: The object type, always chat.completion (or chat.completion.chunk for streaming)
• created: The Unix timestamp of when the completion was created
• model: The model used for the completion (may differ from the requested model if using route-llm)
• choices: A list of completion choices
  • index: The index of the choice
  • message: The message object (non-streaming) or delta (streaming)
  • finish_reason: The reason the completion finished (stop, length, content_filter, tool_calls, or null for streaming)
• usage: Token usage statistics (not present in streaming responses until the final chunk)

Tool Calling

The API supports tool (function) calling: the model can request that your application run a function and return the result in a follow-up request. This enables multi-step workflows (e.g. get weather, query a database, run code).

note

Tool calling with tools is stateless. For the tools you define in tools, the server does not execute anything or persist tool-call state. Your application must run the requested functions, send the results back in a follow-up request (with the same tools and full message history), and handle any multi-step flow on the client side.

For tools registered on the Abacus.AI platform, the abacus_tools parameter flips this around: name them and the server runs them for you, returning only the model's final answer.

Request: Defining tools

Pass a tools array with one or more functions. Optionally set tool_choice to "auto" (default), "none", or a specific function to force.

Example request with tools:

{
  "model": "route-llm",
  "messages": [
    {"role": "user", "content": "What's the weather in Boston?"}
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": { "type": "string", "description": "City and state" },
            "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
          },
          "required": ["location"]
        }
      }
    }
  ],
  "tool_choice": "auto"
}

Response: model requests a tool call

When the model decides to call a tool, the completion message includes a tool_calls array and finish_reason is "tool_calls". The message content may be empty or contain reasoning.

Non-Streaming Response

Example response with tool_calls:

{
  "id": "chatcmpl-xyz",
  "object": "chat.completion",
  "created": 1677858242,
  "model": "route-llm",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_abc123",
            "type": "function",
            "function": {
              "name": "get_current_weather",
              "arguments": "{\"location\": \"Boston, MA\", \"unit\": \"fahrenheit\"}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ],
  "usage": { "prompt_tokens": 20, "completion_tokens": 25, "total_tokens": 45 }
}

Streaming Response

When stream: true, tool calls arrive in multiple chunks. Each chunk contains partial data that must be aggregated by tool_call_id before executing the function.

Example streaming response with tool_calls:

data: {"id":"chatcmpl-xyz","object":"chat.completion.chunk","created":1677858242,"model":"route-llm","choices":[{"index":0,"delta":{"role":"assistant","content":null,"tool_calls":[{"index":0,"id":"call_abc123","type":"function","function":{"name":"get_current_weather","arguments":""}}]},"finish_reason":null}]}

data: {"id":"chatcmpl-xyz","object":"chat.completion.chunk","created":1677858242,"model":"route-llm","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"lo"}}]},"finish_reason":null}]}

data: {"id":"chatcmpl-xyz","object":"chat.completion.chunk","created":1677858242,"model":"route-llm","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"cation\": \"Boston, MA\", "}}]},"finish_reason":null}]}

data: {"id":"chatcmpl-xyz","object":"chat.completion.chunk","created":1677858242,"model":"route-llm","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"unit\": \"fahrenheit\"}"}}]},"finish_reason":null}]}

data: {"id":"chatcmpl-xyz","object":"chat.completion.chunk","created":1677858242,"model":"route-llm","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

data: [DONE]

Key points for streaming tool calls:

• The first chunk contains the tool_call_id, function name, and type
• Subsequent chunks contain partial arguments that must be concatenated
• Use the index field to track multiple parallel tool calls
• The final chunk has finish_reason: "tool_calls" indicating the model is done
• Aggregate all argument chunks before parsing the complete JSON

Follow-up: sending tool results

To continue the conversation, send the assistant message (including tool_calls) and add a message with role: "tool" for each tool call, providing the tool_call_id and the result as content.

Example follow-up request:

{
  "model": "route-llm",
  "messages": [
    {"role": "user", "content": "What's the weather in Boston?"},
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [
        {
          "id": "call_abc123",
          "type": "function",
          "function": {
            "name": "get_current_weather",
            "arguments": "{\"location\": \"Boston, MA\", \"unit\": \"fahrenheit\"}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_abc123",
      "content": "{\"temperature\": 72, \"unit\": \"fahrenheit\", \"conditions\": \"Sunny\"}"
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": { "type": "string" },
            "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
          },
          "required": ["location"]
        }
      }
    }
  ]
}

The model will then generate a final reply (e.g. summarizing the weather). Repeat the flow if it returns more tool_calls.

Notes:

• Include the same tools (and optionally tool_choice) in follow-up requests when continuing a tool-calling conversation.
• When streaming, tool call arguments arrive in multiple chunks. Use the index field to match chunks to specific tool calls, and concatenate the arguments strings before parsing as JSON.
• Multiple tool calls can be returned in a single response. In streaming mode, track each tool call by its index and aggregate separately.

Abacus Tools (server-executed)

Tools you define in tools are yours to run. Tools you name in abacus_tools are run by Abacus.AI: pass the names of tools already registered in your organization and the server does the whole loop for you — no schemas to write, no tool results to send back.

For each request that names abacus_tools, the server:

1. Resolves each name to its registered tool definition and merges the generated JSON Schema into the tools the model sees.
2. Runs any of those tools the model calls, in a sandboxed ephemeral Python kernel.
3. Feeds each result back to the model and asks again, repeating until the model answers in words.
4. Returns that answer as an ordinary OpenAI chat completion — no extension fields in the response.

Because an Abacus tool call is never surfaced to a caller that has no way to service it, abacus_tools works with unmodified OpenAI clients.

Example request:

cURL

curl https://routellm.abacus.ai/v1/chat/completions \
  -H "Authorization: Bearer $ABACUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "route-llm",
    "messages": [{"role": "user", "content": "What is the status of order A-1381?"}],
    "abacus_tools": ["lookup_order_status"]
  }'

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_ABACUS_API_KEY",
    base_url="https://routellm.abacus.ai/v1",
)

response = client.chat.completions.create(
    model="route-llm",
    messages=[{"role": "user", "content": "What is the status of order A-1381?"}],
    extra_body={"abacus_tools": ["lookup_order_status"]},
)

# The tool already ran server-side; this is the final answer.
print(response.choices[0].message.content)

abacus_tools is not part of the OpenAI schema, so pass it through extra_body (Python) or body (TypeScript).

The response is a normal completion with finish_reason: "stop":

{
  "id": "chatcmpl-xyz",
  "object": "chat.completion",
  "created": 1677858242,
  "model": "route-llm",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Order A-1381 shipped on March 3 and is out for delivery today."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 512, "completion_tokens": 96, "total_tokens": 608 }
}

usage covers every round of the loop, not just the last one, since each round is billed.

Discovering available tools

Call listRouteLLMTools to list every tool in your organization that the API can address, along with its generated parameter schema:

cURL

curl "https://api.abacus.ai/api/v0/listRouteLLMTools" \
  -X GET \
  -H "apiKey: YOUR_ABACUS_API_KEY"

Python SDK

from abacusai import ApiClient

client = ApiClient("YOUR_ABACUS_API_KEY")
client.list_route_llm_tools()

Example response:

{
  "tools": [
    {
      "name": "lookup_order_status",
      "description": "Looks up the current status of an order by its id.",
      "parameters": {
        "type": "object",
        "properties": {
          "order_id": { "type": "string", "description": "The order to look up" }
        },
        "required": ["order_id"]
      },
      "requires_connectors": [],
      "supported": true
    }
  ]
}

• requires_connectors: user-level connectors that must be authorized (at https://apps.abacus.ai → Connectors) before the tool can run.
• supported: false for a tool the Chat Completions API cannot run — currently, tools that take file attachments, since a stateless request has no uploaded documents to resolve them against. Naming an unsupported tool returns a 400.

Which tools can be named

Python function tools and template tools created under Custom Code → Tools in the Abacus.AI platform are addressable. See Tools for how to create one.

• Tool names are case-sensitive and must match the registered name exactly.
• Connector tools are excluded. Their interactive OAuth flow has no user to prompt behind an API key. A Python function tool that uses a connector is addressable, but it only runs if that connector is already authorized for the calling user — otherwise the failure is reported to the model, which will ask the user to authorize it. Check requires_connectors before naming a tool.
• Naming any other kind of Python function (for example, a feature group function) is rejected as unknown.

Combining abacus_tools with your own tools

You can pass both in one request. Your own tools still come back to you as usual:

• A turn in which the model calls only your tools ends the loop and returns the standard tool_calls response, which you service and send back exactly as described above.
• A mixed turn runs the Abacus tools first; your calls are dropped from that turn's history so the model can re-issue them once the Abacus results are in. You then receive them in a later response.

A name that appears in both tools and abacus_tools is a 400 rather than being silently namespaced.

Streaming

Streaming works with abacus_tools, with two differences:

• Text content streams as it arrives, but tool-call deltas are held until the turn ends — whether a call is one the server answers itself is only known once the model finishes the turn, and a delta already on the wire cannot be recalled. Calls left for you to run are flushed at that point.
• A : abacus-tool-loop SSE comment is emitted every 15 seconds while a tool is running, so a long loop does not look like a dead connection. Standard SSE clients ignore comment lines; a hand-rolled parser should skip lines beginning with :.

Error handling

Tool failures are reported to the model, not to you: a tool that raises, times out, needs an unauthorized connector, or is called with malformed JSON arguments comes back as the tool's result text, and the model decides how to react (usually by retrying differently or explaining the problem in its answer). This keeps you from being billed for a model turn that is then thrown away.

Request-level problems return a 400 before any model call:

• A name not registered in your organization, or registered as a non-addressable type.
• A tool that takes file attachments (supported: false).
• Duplicate names in abacus_tools, or a name shared with tools.
• More than 20 tools in one request.
• abacus_tools combined with n > 1.
• abacus_tools on a request whose credential has no user behind it.

Limits and billing

Limit	Value
Tools per request	20
Model rounds per request	8 (the last round runs with the tools withheld, so hitting the cap still ends in an answer)
Execution time per tool call	300 seconds
Tool result size	200,000 characters (truncated past that, with a note telling the model to narrow the query)
Tool calls per user	1,000 per hour

Each server-side tool call is billed a flat per-call charge under routellm_abacus_tool for the ephemeral kernel, on top of the LLM tokens that every round of the loop already bills. A tool call that fails because a connector is not authorized is not billed.

caution

Two consequences follow from running the loop inside a single stateless request:

• The response doesn't tell you which tools ran. Only the final answer comes back; usage covers every round of the loop, and each tool call is billed to your organization, but neither names a tool. This matters most for tools with side effects.
• A failed request can re-run them. A retry — whether yours or an SDK's automatic one on a timeout, connection error, or 5xx — starts the loop over, after tools may already have run. This matters for tools that are not idempotent.

PDF Support

PDF documents are supported as input for compatible models. Use the file content type with a file object (filename, file_data) for parsing.

Request schema:

{
  "model": "gpt-5.1",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What are the main points in this document?"
        },
        {
          "type": "file",
          "file": {
            "filename": "document.pdf",
            "file_data": "https://bitcoin.org/bitcoin.pdf"
          }
        }
      ]
    }
  ]
}

Notes:

• Use type: "file" with a file object containing filename and file_data
• file_data can be an HTTPS URL to the PDF or base64-encoded content

Image & Audio Capabilities

RouteLLM supports rich multimodal capabilities beyond text. Use the links below to explore each capability in detail.

Images

Analyze images as input (vision) or generate images from text prompts using dedicated generators and multimodal LLMs.

For supported models, request parameters, and code examples → Image Analysis & Generation

Audio

The RouteLLM API supports audio understanding (speech input) and audio generation (text-to-speech) using OpenAI GPT-4o Audio models and Google Gemini TTS models.

For supported models, pricing, audio parameter reference, available voices, and code examples → Audio Capabilities

Error Handling

The API uses standard HTTP status codes to indicate success or failure:

• 200 OK: Request succeeded
• 400 Bad Request: Invalid request (missing parameters, invalid format, etc.)
• 401 Unauthorized: Missing or invalid API key
• 429 Too Many Requests: Rate limit exceeded
• 500 Internal Server Error: Server error

Error Response Format

{
  "error": {
    "message": "The 'messages' parameter is missing, empty, or not a list.",
    "type": "ValidationError",
    "code": "invalid_request_error"
  }
}

Common error scenarios:

• Missing required messages parameter
• Empty messages array
• Missing role or content in message objects
• Invalid role value (must be "user", "assistant", or "system")
• Invalid model name
• Rate limit exceeded

Code Examples

Basic Request

Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="<your base url>",
    api_key="<your_api_key>",
)

response = client.chat.completions.create(
    model="route-llm",
    messages=[
        {"role": "user", "content": "What is the meaning of life?"}
    ]
)

print(response.choices[0].message.content)

TypeScript/JavaScript

import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: '<your base url>',
  apiKey: '<your_api_key>',
});

const completion = await openai.chat.completions.create({
  model: 'route-llm',
  messages: [
    { role: 'user', content: 'What is the meaning of life?' }
  ],
});

console.log(completion.choices[0].message.content);

cURL

curl -X POST "<your base url>/chat/completions" \
  -H "Authorization: Bearer <your_api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "route-llm",
    "messages": [
      {"role": "user", "content": "What is the meaning of life?"}
    ]
  }'

Streaming Request

Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="<your base url>",
    api_key="<your_api_key>",
)

stream = client.chat.completions.create(
    model="route-llm",
    messages=[
        {"role": "user", "content": "Explain quantum computing in simple terms."}
    ],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="", flush=True)

TypeScript/JavaScript

import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: '<your base url>',
  apiKey: '<your_api_key>',
});

const stream = await openai.chat.completions.create({
  model: 'route-llm',
  messages: [
    { role: 'user', content: 'Explain quantum computing in simple terms.' }
  ],
  stream: true,
});

for await (const chunk of stream) {
  if (chunk.choices[0]?.delta?.content) {
    process.stdout.write(chunk.choices[0].delta.content);
  }
}

cURL

curl -X POST "<your base url>/chat/completions" \
  -H "Authorization: Bearer <your_api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "route-llm",
    "messages": [
      {"role": "user", "content": "Explain quantum computing."}
    ],
    "stream": true
  }'

Conversation with History

Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="<your base url>",
    api_key="<your_api_key>",
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "My name is Alice."},
    {"role": "assistant", "content": "Nice to meet you, Alice! How can I help you today?"},
    {"role": "user", "content": "What's my name?"}
]

response = client.chat.completions.create(
    model="route-llm",
    messages=messages,
    temperature=0.7,
    max_tokens=150
)

print(response.choices[0].message.content)

TypeScript/JavaScript

import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: '<your base url>',
  apiKey: '<your_api_key>',
});

const messages = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'My name is Alice.' },
  { role: 'assistant', content: 'Nice to meet you, Alice! How can I help you today?' },
  { role: 'user', content: "What's my name?" }
];

const completion = await openai.chat.completions.create({
  model: 'route-llm',
  messages: messages,
  temperature: 0.7,
  max_tokens: 150,
});

console.log(completion.choices[0].message.content);

cURL

curl -X POST "<your base url>/chat/completions" \
  -H "Authorization: Bearer <your_api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "route-llm",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "My name is Alice."},
      {"role": "assistant", "content": "Nice to meet you, Alice! How can I help you today?"},
      {"role": "user", "content": "What'\''s my name?"}
    ],
    "temperature": 0.7,
    "max_tokens": 150
  }'

JSON Mode

Python SDK

from openai import OpenAI
import json

client = OpenAI(
    base_url="<your base url>",
    api_key="<your_api_key>",
)

response = client.chat.completions.create(
    model="route-llm",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that outputs JSON."
        },
        {
            "role": "user",
            "content": "Return a JSON object with keys 'name', 'age', and 'city'."
        }
    ],
    response_format={"type": "json_object"},
    temperature=0.7
)

content = response.choices[0].message.content
data = json.loads(content)
print(data)

TypeScript/JavaScript

import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: '<your base url>',
  apiKey: '<your_api_key>',
});

const completion = await openai.chat.completions.create({
  model: 'route-llm',
  messages: [
    {
      role: 'system',
      content: 'You are a helpful assistant that outputs JSON.'
    },
    {
      role: 'user',
      content: "Return a JSON object with keys 'name', 'age', and 'city'."
    }
  ],
  response_format: { type: 'json_object' },
  temperature: 0.7,
});

const data = JSON.parse(completion.choices[0].message.content || '{}');
console.log(data);

cURL

curl -X POST "<your base url>/chat/completions" \
  -H "Authorization: Bearer <your_api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "route-llm",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant that outputs JSON."
      },
      {
        "role": "user",
        "content": "Return a JSON object with keys '\''name'\'', '\''age'\'', and '\''city'\''."
      }
    ],
    "response_format": {"type": "json_object"},
    "temperature": 0.7
  }'

Structured Output (JSON Schema)

Python SDK

from openai import OpenAI
import json

client = OpenAI(
    base_url="<your base url>",
    api_key="<your_api_key>",
)

response = client.chat.completions.create(
    model="route-llm",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that outputs JSON."
        },
        {
            "role": "user",
            "content": "Extract the name, age, and city from: 'Alice is 30 years old and lives in Paris.'"
        }
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "person_info",
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "age": {"type": "integer"},
                    "city": {"type": "string"}
                },
                "required": ["name", "age", "city"],
                "additionalProperties": False
            }
        }
    }
)

data = json.loads(response.choices[0].message.content)
print(data)  # {"name": "Alice", "age": 30, "city": "Paris"}

TypeScript/JavaScript

import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: '<your base url>',
  apiKey: '<your_api_key>',
});

const completion = await openai.chat.completions.create({
  model: 'route-llm',
  messages: [
    {
      role: 'system',
      content: 'You are a helpful assistant that outputs JSON.'
    },
    {
      role: 'user',
      content: "Extract the name, age, and city from: 'Alice is 30 years old and lives in Paris.'"
    }
  ],
  response_format: {
    type: 'json_schema',
    json_schema: {
      name: 'person_info',
      schema: {
        type: 'object',
        properties: {
          name: { type: 'string' },
          age: { type: 'integer' },
          city: { type: 'string' }
        },
        required: ['name', 'age', 'city'],
        additionalProperties: false
      }
    }
  }
});

const data = JSON.parse(completion.choices[0].message.content || '{}');
console.log(data); // { name: 'Alice', age: 30, city: 'Paris' }

cURL

curl -X POST "<your base url>/chat/completions" \
  -H "Authorization: Bearer <your_api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "route-llm",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant that outputs JSON."
      },
      {
        "role": "user",
        "content": "Extract the name, age, and city from: '\''Alice is 30 years old and lives in Paris.'\''"
      }
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "person_info",
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"},
            "city": {"type": "string"}
          },
          "required": ["name", "age", "city"],
          "additionalProperties": false
        }
      }
    }
  }'

With Optional Parameters

Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="<your base url>",
    api_key="<your_api_key>",
)

response = client.chat.completions.create(
    model="route-llm",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a haiku about programming."}
    ],
    max_tokens=100,
    temperature=0.8,
    top_p=0.9
)

print(response.choices[0].message.content)

TypeScript/JavaScript

import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: '<your base url>',
  apiKey: '<your_api_key>',
});

const completion = await openai.chat.completions.create({
  model: 'route-llm',
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'Write a haiku about programming.' }
  ],
  max_tokens: 100,
  temperature: 0.8,
  top_p: 0.9,
});

console.log(completion.choices[0].message.content);

cURL

curl -X POST "<your base url>/chat/completions" \
  -H "Authorization: Bearer <your_api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "route-llm",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Write a haiku about programming."}
    ],
    "max_tokens": 100,
    "temperature": 0.8,
    "top_p": 0.9
  }'

Best Practices

1. Use route-llm for most cases: Let the system choose the optimal model automatically
2. Include conversation history: Provide full message history for better context
3. Set appropriate max_tokens: Prevent unnecessarily long responses
4. Use streaming for long responses: Improve user experience with real-time output
5. Handle errors gracefully: Implement retry logic for transient errors