Skip to main content

Responses API Reference

Alongside the OpenAI-compatible /v1/chat/completions endpoint, RouteLLM exposes the OpenAI-native Responses API at /v1/responses. It accepts requests in the OpenAI Responses API format and returns responses in the same format, so existing OpenAI SDK code works with a one-line base-URL change.

Overview​

The Responses endpoint is a thin pass-through to the provider: your request body is forwarded raw and the provider response (including streaming events) is returned verbatim. This preserves every Responses-API feature — reasoning controls, built-in tools, prompt caching, and multi-turn continuity — while billing and credit tracking are handled transparently on the Abacus.AI side.

The Responses API is designed for reasoning and agentic workloads. It represents a turn as a list of typed input and output items (messages, reasoning, tool calls) rather than a flat messages array, which lets reasoning models carry their internal state across tool calls and turns.

Key Features​

  • Drop-in OpenAI Compatibility: Point the official OpenAI SDK at RouteLLM by changing only the base URL and API key, then call client.responses.create(...).
  • Reasoning Controls: Pass reasoning.effort and reasoning.summary to steer reasoning-model behavior.
  • Built-in Tools: Server-side tools such as web_search, code_interpreter, image_generation, and computer_use_preview in addition to your own function tools.
  • Streaming Support: Server-sent event (SSE) streaming in the native Responses event format.
  • Prompt Caching: Automatic prompt-cache reuse to cut cost and latency on repeated context.
  • Multi-turn Continuity: Continue a prior turn with previous_response_id without resending the full history.
note

The Responses API serves reasoning-oriented models only — the GPT-5 family, OpenAI o-series, and xAI Grok 4.2/4.3. To use Claude models, use the Anthropic-native /v1/messages endpoint. For all other models and general use, use the OpenAI-compatible /v1/chat/completions endpoint.

Base URLs​

The Responses endpoint lives at /v1/responses under your RouteLLM base URL. The base URL depends on your organization type:

  • Self-Serve Organizations: https://routellm.abacus.ai/v1
  • Enterprise Platform: https://<workspace>.abacus.ai/v1

Replace <workspace> with your specific workspace identifier for enterprise deployments. To find your correct base URL, refer to the RouteLLM API page.

Authentication​

All requests require a RouteLLM API key. The endpoint accepts the key in either of the two standard headers, so the official OpenAI SDK (which sends Authorization: Bearer) works without modification:

Authorization: Bearer <your_api_key>
x-api-key: <your_api_key>

You can obtain your API key from the Abacus.AI platform.

Supported Models​

Pass a Responses-capable model identifier in the model field. These include the GPT-5 family, the OpenAI o-series, and xAI Grok reasoning models:

ModelModel
gpt-5.5o3
gpt-5.4o3-mini
gpt-5.3-codexo3-pro
gpt-5.1grok-4.2
gpt-5grok-4.3
note

A few models are available only through the Responses API and cannot be used via Chat Completions — for example o3-pro and the GPT-5 Codex variants such as gpt-5.3-codex. Passing a non-Responses model (or a dedicated image model like gpt-image-1) returns a 400 error.

Retrieve the full, up-to-date list of available models programmatically via the GET /v1/models endpoint. The identifiers shown here are examples; call /v1/models for the exact id string to send.

Request Parameters​

The request body follows the OpenAI Responses API schema. The body is forwarded raw to the provider, so any parameter the Responses API supports is accepted. Common parameters:

Required Parameters​

model (string, required)​

A Responses-capable model identifier (see Supported Models).

input (string or array, required)​

The input to the model. Provide a plain string for a simple prompt, or an array of typed input items (message, reasoning, tool calls, etc.) for richer turns. Either input or previous_response_id must be present.

Optional Parameters​

instructions (string)​

High-level, developer-role instructions that guide the model — the Responses-API equivalent of a system prompt.

reasoning (object)​

Reasoning controls for reasoning-capable models, e.g. {"effort": "high", "summary": "auto"}. If you don't set reasoning, a sensible model-specific effort is applied automatically.

tools / tool_choice (array / string or object)​

Tool definitions and the tool-selection strategy. tools may mix your own function tools with built-in server-side tools — see Tool Use. Must be a list of tool-definition objects.

max_output_tokens (integer)​

Upper bound on the number of tokens generated (including reasoning tokens). Unlike the Anthropic Messages endpoint, the Responses API does not require this and RouteLLM does not force a default — omit it to let the model decide.

stream (boolean)​

Set to true to stream the response as SSE events. Defaults to false.

store (boolean)​

Whether the provider should persist the response so it can be referenced later via previous_response_id. See Multi-turn Continuity.

previous_response_id (string)​

The id of a prior response to continue from, so you don't have to resend earlier input. Either input or previous_response_id must be present.

prompt_cache_key / prompt_cache_retention (string)​

Prompt-caching controls. See Prompt Caching. If you omit prompt_cache_key, RouteLLM sets a stable per-organization key automatically for OpenAI and Grok models.

service_tier (string)​

The provider service tier to route the request to. If omitted, a model-appropriate default is applied.

temperature / top_p (number)​

Standard sampling controls, where supported by the model.

Response Format​

A non-streaming request returns a standard Responses API Response object. The generated content lives in the output array as typed items:

{
  "id": "resp_01Abc123",
  "object": "response",
  "created_at": 1730000000,
  "status": "completed",
  "model": "gpt-5.1",
  "output": [
    {
      "type": "message",
      "id": "msg_01Abc123",
      "role": "assistant",
      "content": [
        { "type": "output_text", "text": "Hello! How can I help you today?" }
      ]
    }
  ],
  "usage": {
    "input_tokens": 12,
    "input_tokens_details": { "cached_tokens": 0 },
    "output_tokens": 18,
    "output_tokens_details": { "reasoning_tokens": 0 },
    "total_tokens": 30
  }
}

For reasoning models, the output array may also contain reasoning items in addition to the message item. The OpenAI SDK exposes an output_text convenience property that concatenates the assistant text for you.

Counting Tokens​

Every response includes a usage object with the exact token counts for that request, so you don't need a separate call to measure consumption:

  • input_tokens: tokens in the request input (input items, instructions, tools).
  • input_tokens_details.cached_tokens: input tokens served from the prompt cache (billed at the reduced cache-read rate).
  • output_tokens: tokens generated in the response.
  • output_tokens_details.reasoning_tokens: tokens spent on internal reasoning (a subset of output_tokens).
  • total_tokens: the sum of input and output tokens.

For streaming requests the same figures are reported on the terminal event (response.completed, response.incomplete, or response.failed) — see Streaming.

Streaming​

When stream is true, the response is a text/event-stream of native Responses-API events. Each event is emitted as an SSE event: / data: pair, with incremental text delivered on response.output_text.delta:

event: response.created
data: {"type":"response.created","response":{"id":"resp_01Abc123","status":"in_progress","model":"gpt-5.1"}}

event: response.output_text.delta
data: {"type":"response.output_text.delta","delta":"Hello"}

event: response.output_text.done
data: {"type":"response.output_text.done","text":"Hello! How can I help you today?"}

event: response.completed
data: {"type":"response.completed","response":{"id":"resp_01Abc123","status":"completed","usage":{"input_tokens":12,"output_tokens":18,"total_tokens":30}}}

Token usage is reported on the terminal event (response.completed, or response.incomplete / response.failed if the turn ends early). Reasoning items are preserved verbatim in the stream, which is what enables interleaved thinking across tool calls.

Prompt Caching​

The Responses API caches large, stable prompt prefixes automatically, reducing both cost and latency when you reuse the same instructions, tools, or context across requests. Cached input tokens are billed at a reduced rate and reported under usage.input_tokens_details.cached_tokens.

RouteLLM improves cache reuse for you: when you don't pass a prompt_cache_key, it sets a stable per-organization key automatically for OpenAI and Grok models, so repeated requests from the same organization share cache. You can override this by passing your own prompt_cache_key to group related requests more precisely, and prompt_cache_retention to influence how long the cached prefix is retained.

tip

Keep the stable part of your prompt (instructions, tool definitions, long reference context) at the start of your input and the dynamic, per-turn content at the end. This maximizes the cached prefix and the resulting savings.

Multi-turn Continuity​

Instead of resending the entire conversation on every request, you can continue from a prior response. Send the first request with store: true, then reference the returned response id as previous_response_id on the next request:

{
  "model": "gpt-5.1",
  "previous_response_id": "resp_01Abc123",
  "input": "And summarize that in one sentence."
}

Because the provider retains the prior turn's items — including reasoning items — the model continues with full context and preserved chain-of-thought, without you having to reconstruct the history client-side.

Tool Use​

Provide tools in the tools array. You can mix your own function tools with provider built-in tools, and use tool_choice to control selection.

Function tools — define a function and let the model call it; you execute it and feed the result back as a function-call output item in a follow-up request:

{
  "model": "gpt-5.1",
  "input": "What's the weather in Boston?",
  "tools": [
    {
      "type": "function",
      "name": "get_weather",
      "description": "Get the current weather for a location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": { "type": "string", "description": "City name" }
        },
        "required": ["location"]
      }
    }
  ]
}

Built-in tools — server-side tools the provider runs for you, requested by type alone:

  • web_search — live web search.
  • code_interpreter — sandboxed code execution.
  • image_generation — generate images inline from a Responses-capable model.
  • computer_use_preview — computer-use agent actions.
{
  "model": "gpt-5.1",
  "input": "Create a picture of a red bicycle.",
  "tools": [{ "type": "image_generation" }]
}
note

Image generation is driven by a Responses-capable model with the image_generation tool (as above), not by passing a dedicated image model such as gpt-image-1 — those are not valid Responses models. For direct text-to-image generation, refer to the Image Analysis & Generation section.

Code Examples​

Basic Request​

from openai import OpenAI

client = OpenAI(
    base_url="<your base url>",
    api_key="<your_api_key>",
)

response = client.responses.create(
    model="gpt-5.1",
    input="What is the meaning of life?",
)

print(response.output_text)

Streaming Request​

from openai import OpenAI

client = OpenAI(
    base_url="<your base url>",
    api_key="<your_api_key>",
)

stream = client.responses.create(
    model="gpt-5.1",
    input="Explain quantum computing in simple terms.",
    stream=True,
)

for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)

Reasoning + Tool Use​

from openai import OpenAI

client = OpenAI(
    base_url="<your base url>",
    api_key="<your_api_key>",
)

response = client.responses.create(
    model="gpt-5.1",
    instructions="You are a concise research assistant.",
    input="What were the key announcements at the latest major AI conference?",
    reasoning={"effort": "high"},
    tools=[{"type": "web_search"}],
)

print(response.output_text)

Best Practices​

  1. Use the official OpenAI SDK and override only the base URL and API key, then call client.responses.create(...).
  2. Prefer the Responses API for reasoning and agentic workflows; use Chat Completions for simple, general-purpose calls.
  3. Use previous_response_id with store: true to continue turns instead of resending history — this also preserves reasoning state.
  4. Keep stable content (instructions, tools, reference context) at the start of your input to maximize prompt-cache reuse, and inspect usage.input_tokens_details.cached_tokens to confirm hits.
  5. Prefer streaming for long or interactive responses.