> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Batches
> Submit large offline workloads as JSONL batches through the /v3/router/batches API and download OpenAI-format results.
See [HTTP URLs and authentication](/docs/reference/base-urls.md) and the API reference for complete request and response formats.
Batches are the best way to process a large volume of requests: they run without the throughput limits of real-time requests, at a fraction of the price.
The Batches API runs large offline workloads without a request per line. Upload a JSONL file of requests, create a batch against a supported endpoint, and download the results once the batch reaches a terminal state.
The API is OpenAI-compatible at `/v3/router/batches`. Batch requests, response objects, status values, and result files follow the OpenAI batch format, with **Toqan Router** additions for the endpoint list, listing filters, and project scoping.
> [!NOTE]
>
> Provider batches run against models that declare batch support for the endpoint and are priced at the provider's batch rates. Check batch support for a model on the [Models page](/docs/ai-gateway/using-the-router.md).
## Supported endpoints
| Endpoint | Runs through | Providers |
| - | - | - |
| `/v1/responses` | Provider batch API | OpenAI, Google, Anthropic, AWS Bedrock (Anthropic models) |
| `/v1/chat/completions` | Provider batch API | OpenAI, Google, Anthropic, AWS Bedrock (Anthropic models), Mistral |
| `/v1/embeddings` | Provider batch API | OpenAI, Mistral |
| `/v1/completions` | Provider batch API | OpenAI |
| `/v1/moderations` | Provider batch API | OpenAI, Mistral |
| `/v3/router/decisions` | **Toqan Router** workers | Any model enabled for [Decisions](/docs/ai-gateway/features/decisions.md) |
Provider batches run through the provider's batch service. Decisions batches run on **Toqan Router** workers, which pace the lines and check budgets per line.
The OpenAI SDK type definitions declare `/v1/responses`, `/v1/chat/completions`, `/v1/embeddings`, and `/v1/completions`. Send `/v1/moderations` and `/v3/router/decisions` with the HTTP API.
## Run a batch
Create a batch from the console or the API.
> [tab] AI Studio
>
> 1. Open **Batches** under **AI Gateway**. The list shows one row per batch with its ID, status, model, endpoint, request counts, and creation time.
> 2. Click Batch.
> 3. Choose the JSONL file, pick the endpoint, add metadata when needed, and click **Create batch**. The completion window is `24h` and the field is not editable.
> 4. Follow the request counts, files, and status timeline in the list or the batch detail drawer. Filter by **Status** and **Endpoint**, or search by batch ID, model, or endpoint.
> 5. Download the results once the batch finishes: the drawer footer shows **Download results**, and **Download errors** beside it when the batch produced an error file, or on its own when the batch produced no output file. Cancel a running batch from the row actions or the detail drawer.
> [tab] API & SDK
>
> **1. Write the input file**
>
> Write a JSONL file with one request per line. Each line carries a unique `custom_id`, `"method": "POST"`, a `url` that matches the batch `endpoint`, and the request as `body`.
>
> ```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
> {"custom_id": "ticket-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Summarize this support ticket in one sentence."}]}}
> {"custom_id": "ticket-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Draft a reply to this support ticket."}]}}
> ```
>
> **2. Upload the input file**
>
> Upload the file with the Files API and `purpose` set to `FILE_PURPOSE_BATCH`. The returned file ID is the `input_file_id` for the batch.
>
> ```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
> curl -X POST https://router.toqan.ai/eu/v2/files \
> -H "Authorization: Bearer $TOQAN_API_KEY" \
> -H "Content-Type: application/json" \
> -d "{
> \"filename\": \"requests.jsonl\",
> \"content\": \"$(base64 < requests.jsonl | tr -d '\n')\",
> \"content_type\": \"application/jsonl\",
> \"purpose\": \"FILE_PURPOSE_BATCH\"
> }"
> ```
>
> **3. Create the batch**
>
> Send the file ID, the endpoint the lines target, and a `completion_window` of `24h`. Other values are rejected with `400`.
>
> `output_expires_after` sets a lifetime for both result files when the batch runs through a provider that supports output expiry: `seconds`, counted from `created_at`. An omitted field keeps the provider's default. Batches the gateway executes itself keep their result files available.
>
> ```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
> curl -X POST https://router.toqan.ai/eu/v3/router/batches \
> -H "Authorization: Bearer $TOQAN_API_KEY" \
> -H "Content-Type: application/json" \
> -d '{
> "input_file_id": "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
> "endpoint": "/v1/chat/completions",
> "completion_window": "24h"
> }'
> ```
>
> ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
> import OpenAI from "openai";
>
> const client = new OpenAI({
> apiKey: process.env.TOQAN_API_KEY,
> baseURL: "https://router.toqan.ai/eu/v3/router",
> });
>
> const batch = await client.batches.create({
> input_file_id: "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
> endpoint: "/v1/chat/completions",
> completion_window: "24h",
> });
>
> console.log(batch.id);
> ```
>
> ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
> import os
> from openai import OpenAI
>
> client = OpenAI(
> api_key=os.environ["TOQAN_API_KEY"],
> base_url="https://router.toqan.ai/eu/v3/router",
> )
>
> batch = client.batches.create(
> input_file_id="file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
> endpoint="/v1/chat/completions",
> completion_window="24h",
> )
>
> print(batch.id)
> ```
>
> The creation is validated up front. Every line is checked for a valid `custom_id`, `method`, `url`, and model before the batch is created. An invalid line rejects the file with `400` and names the line.
>
> **4. Poll the batch**
>
> Call `GET /v3/router/batches/{batch_id}` until `status` is `completed`, `failed`, `cancelled`, or `expired`. `request_counts` reports how many lines completed and failed so far.
>
> ```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
> curl https://router.toqan.ai/eu/v3/router/batches/$BATCH_ID \
> -H "Authorization: Bearer $TOQAN_API_KEY"
> ```
>
> ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
> import OpenAI from "openai";
> import { setTimeout as sleep } from "node:timers/promises";
>
> const client = new OpenAI({
> apiKey: process.env.TOQAN_API_KEY,
> baseURL: "https://router.toqan.ai/eu/v3/router",
> });
>
> let batch = await client.batches.retrieve("batch_01K9X2W7Y8Q9M0N1P2R3S4T5V");
>
> while (!["completed", "failed", "cancelled", "expired"].includes(batch.status)) {
> await sleep(5000);
> batch = await client.batches.retrieve(batch.id);
> }
>
> console.log(batch.status, batch.request_counts);
> ```
>
> ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
> import os
> import time
> from openai import OpenAI
>
> client = OpenAI(
> api_key=os.environ["TOQAN_API_KEY"],
> base_url="https://router.toqan.ai/eu/v3/router",
> )
>
> batch = client.batches.retrieve("batch_01K9X2W7Y8Q9M0N1P2R3S4T5V")
>
> while batch.status not in ("completed", "failed", "cancelled", "expired"):
> time.sleep(5)
> batch = client.batches.retrieve(batch.id)
>
> print(batch.status, batch.request_counts)
> ```
>
> **5. Download the results**
>
> Call `GET /v3/router/batches/{batch_id}/results` to download the output file. Send `?file=errors` to download the failed and unexecuted lines instead.
>
> ```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
> curl https://router.toqan.ai/eu/v3/router/batches/$BATCH_ID/results \
> -H "Authorization: Bearer $TOQAN_API_KEY" \
> -o results.jsonl
>
> curl "https://router.toqan.ai/eu/v3/router/batches/$BATCH_ID/results?file=errors" \
> -H "Authorization: Bearer $TOQAN_API_KEY" \
> -o errors.jsonl
> ```
>
> Each download is named after the request: `-results.jsonl` for the output file, `-errors.jsonl` for the error file. Without `file`, the response carries the output file, or the errors file when the batch produced no output, under the `-results` name. A batch with no error file answers `409 output_unavailable`.
>
> Both result files expire according to the batch's `output_expires_after`; download them before it ends.
>
> Downloads are available once the batch reaches a terminal state. Requesting one earlier returns `409 invalid_request`.
## Input file
Each line follows the OpenAI batch input format.
| Field | Type | Description |
| - | - | - |
| `custom_id` | string | Required. Unique identifier for the line, echoed in the results. |
| `method` | string | Required. `POST`. |
| `url` | string | Required. Must match the batch `endpoint`. |
| `body` | object | Required. The request body, including `model`. |
Limits apply to the whole file and to each line.
| Limit | Value |
| - | - |
| File size | 100 MiB |
| Lines per batch | 99,999 |
| Line size | 20 MiB |
| Completion window | `24h` |
All lines in a provider batch must resolve to the same provider, and each model must declare batch support for the endpoint. Lines can use different models only when they share a provider and support the endpoint.
## Batch statuses
| Status | Meaning |
| - | - |
| `validating` | The batch is being checked before work starts. |
| `in_progress` | Lines are running. |
| `finalizing` | All lines finished and the result files are being written. |
| `completed` | The batch finished. Results are available. |
| `failed` | The batch stopped with an error. Details are in `errors`. |
| `expired` | The 24-hour completion window passed before all lines ran. |
| `cancelling` | A cancel request is being applied. |
| `cancelled` | The batch stopped. Lines that had not run appear in the errors file. |
## List and filter batches
`GET /v3/router/batches` returns batches in the caller's projects, newest first.
| Parameter | Type | Description |
| - | - | - |
| `limit` | integer | Batches per page, 1 to 100. Defaults to 20. |
| `after` | string | Return batches created before this batch ID. |
| `before` | string | Return batches created after this batch ID. Cannot be combined with `after`. |
| `status` | string | Comma-separated statuses, for example `completed,failed`. |
| `endpoint` | string | Comma-separated endpoints. |
| `search` | string | Case-insensitive match on batch ID, model, or endpoint. |
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl "https://router.toqan.ai/eu/v3/router/batches?limit=20" \
-H "Authorization: Bearer $TOQAN_API_KEY"
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";
const client = new OpenAI({
apiKey: process.env.TOQAN_API_KEY,
baseURL: "https://router.toqan.ai/eu/v3/router",
});
const page = await client.batches.list({ limit: 20 });
console.log(page.data.length, page.has_more);
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["TOQAN_API_KEY"],
base_url="https://router.toqan.ai/eu/v3/router",
)
page = client.batches.list(limit=20)
print(len(page.data), page.has_more)
```
The OpenAI SDK `list` method sends `limit` and `after`. Send `before`, `status`, `endpoint`, and `search` over the HTTP API:
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl "https://router.toqan.ai/eu/v3/router/batches?limit=20&status=completed&endpoint=/v1/chat/completions&search=gpt-4o" \
-H "Authorization: Bearer $TOQAN_API_KEY"
```
The response uses the OpenAI cursor list envelope:
```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"object": "list",
"data": [
{
"id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
"object": "batch",
"endpoint": "/v1/chat/completions",
"input_file_id": "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
"completion_window": "24h",
"status": "completed",
"output_file_id": "file_01K9X2W7Y8Q9M0N1P2R3S4T5W",
"created_at": 1761000000,
"request_counts": { "total": 2, "completed": 2, "failed": 0 },
"model": "openai/gpt-4o"
}
],
"first_id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
"last_id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
"has_more": false
}
```
## Results
Each result line follows the OpenAI batch output format. `response.body` is the endpoint response for that line. `response.request_id` is the provider's response identifier for a provider batch, and the trace ID of the line for a Decisions batch.
A provider batch records one trace for the whole batch when it reaches a terminal state, so look it up by the batch ID: the trace carries `orq.batch.id`. A Decisions batch traces each line separately, and that trace ID is the line's `request_id`.
```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
{"id": "batch_req_01K9X2W7Y8Q9M0N1P2R3S4T5V", "custom_id": "ticket-1", "response": {"status_code": 200, "request_id": "4f2294c4bd55372a354be496871aa917", "body": {"id": "chatcmpl-01K9X2W7Y8Q9M0N1P2R3S4T5W", "model": "openai/gpt-4o", "choices": [{"index": 0, "message": {"role": "assistant", "content": "The customer was charged twice for the same order."}, "finish_reason": "stop"}]}}, "error": null}
```
Lines that failed carry an `error` object instead of a `response`:
```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
{"id": "batch_req_01K9X2W7Y8Q9M0N1P2R3S4T5W", "custom_id": "ticket-2", "response": null, "error": {"code": "batch_expired", "message": "The line did not run before the completion window expired."}}
```
| Code | Meaning |
| - | - |
| `batch_cancelled` | The batch was cancelled before the line ran. |
| `batch_expired` | The line did not run within the 24-hour completion window. |
Provider batches can also return provider-specific error codes for lines the provider rejected.
Both files download from `GET /v3/router/batches/{batch_id}/results`, as shown in the download step under **Run a batch**: the output file by default, the error file with `?file=errors`.
## Behaviour
* **Validation up front**: every line is checked before the batch is created, including the model and its access checks. An invalid line rejects the whole file with `400` and names the line.
* **Tracing and billing**: a provider batch records one trace with the batch's aggregate usage when it reaches a terminal state, priced at the provider's batch rates. A Decisions batch runs on **Toqan Router** workers, and each line is traced and billed like a single request.
* **Budgets**: Decisions batches check budgets per line. A line that hits a cost or token budget fails with `429` in its result; a requests-per-minute budget delays the line for up to two minutes before failing it.
* **Project scoping**: a batch belongs to the project of its input file. List, retrieve, download, and cancel only reach batches in the caller's projects.
* **Permissions**: the API key needs the `batches` permission. A Decisions batch also needs the `classify` permission.
* **Rate limits**: creating a batch counts as one request against plan rate limits. See [Rate limits and quotas](/docs/ai-gateway/features/rate-limits.md).
* **Cancel**: `POST /v3/router/batches/{batch_id}/cancel` stops the lines that have not run. Calling it on a terminal batch returns the stored batch unchanged.
## Related
> [card] [Decisions](/docs/ai-gateway/features/decisions.md)
>
> Batch classification requests against a Decisions model.
> [card] [Background Responses](/docs/ai-gateway/features/background-responses.md)
>
> Run a single long-running response without batching.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.