> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Load balancing across providers

> Distribute LLM requests across providers using latency-based, weight-based, or round-robin routing to optimize costs, run A/B tests, and ensure redundancy.

**Use Cases**

* Distributing traffic across multiple provider accounts to stay within per-key rate limits.
* A/B testing providers by routing a configurable percentage of traffic to each.
* Spreading traffic across multiple providers to reduce blast radius from a single provider outage.
* Maximizing throughput when one provider's capacity is a bottleneck.
* Minimizing response time by routing to the fastest available model.

***

## Quick Start

Distribute requests across multiple providers using weighted routing.

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://router.toqan.ai/eu/v3/router/responses \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Write a marketing slogan",
    "load_balancer": {
      "type": "weight_based",
      "models": [
        {"model": "openai/gpt-5.4-mini", "weight": 0.7},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.3}
      ]
    }
  }'
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-5.4-mini",
  input: "Write a marketing slogan",
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.7 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.3 },
    ],
  },
});

console.log(response.output_text);
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("TOQAN_API_KEY"),
    base_url="https://router.toqan.ai/eu/v3/router",
)

response = client.responses.create(
    model="openai/gpt-5.4-mini",
    input="Write a marketing slogan",
    extra_body={
        "load_balancer": {
            "type": "weight_based",
            "models": [
                {"model": "openai/gpt-5.4-mini", "weight": 0.7},
                {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.3},
            ],
        }
    },
)

print(response.output_text)
```

```typescript TypeScript (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [{ role: "user", content: "Write a marketing slogan" }],
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.7 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.3 },
    ],
  },
});
```

## Configuration

| Parameter | Type | Required | Description |
| - | - | - | - |
| `load_balancer` | Object | Yes | Load balancer configuration (top-level) |
| `load_balancer.type` | string | Yes | Strategy type (`weight_based`, `round_robin`, or `latency_based`) |
| `load_balancer.models` | Array | Yes | List of models with weights |
| `models[].model` | string | Yes | Model identifier |
| `models[].weight` | number | Yes | Weight assigned to this model (0.001 - 1.0) |

**Weight Calculation:**

* Weights are normalized: `[0.4, 0.8]` → `[33%, 67%]`.
* Higher weight = more traffic.
* Minimum weight: `0.001`.
* Weight is ignored for round robin selection; every model in the list still receives an equal share of traffic.

> [!WARNING]
>
> A matching [Routing Rule](/docs/ai-gateway/configuration/routing-rules.md#load-balancer) with its own target models configured overwrites this `load_balancer` parameter entirely, regardless of what the request sends. The Routing Rule's own strategy and models are used instead; the two configurations are never merged.

> [!NOTE]
>
> `load_balancer` selects one model per request. It does not retry a failed call against another model in the pool. The top-level `model` is used as-is only when `load_balancer` cannot produce a selection, for example an empty `models` list. To fail over when a call errors, pair `load_balancer` with [Fallbacks](/docs/ai-gateway/features/retries.md#fallbacks).

## Latency-based routing

Route each request to the model with the lowest recently observed latency, instead of a fixed traffic split.

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://router.toqan.ai/eu/v3/router/responses \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Write a marketing slogan",
    "load_balancer": {
      "type": "latency_based",
      "models": [
        {"model": "openai/gpt-5.4-mini", "weight": 0.5},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.5}
      ]
    }
  }'
```

**How selection works:**

* Latency is tracked per model as a running average that weights recent calls more heavily, so the selection adapts quickly when a provider speeds up or slows down.
* With fresh data (samples from the last 5 minutes) for every configured model, the lowest-latency model is selected. A model with no data, or stale data, is probed again instead of being written off.
* Ten percent of selections explore the full pool by configured weight to keep latency measurements fresh. Failed calls are penalized so a fast failure does not look like a fast success.
* Configured weights decide exploration and near-ties: with the two fastest models within 0.5 ms of each other, the higher weight wins. With clear data, the lowest latency always wins.

> [accordion] Full selection flow
>
> ```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
> flowchart TD
>     A["New request"] --> B{"10% exploration draw?"}
>     B -- "Yes" --> C["Pick a model by configured weight"]
>     B -- "No" --> D{"Fresh data for every model? (< 5 min old)"}
>
>     D -- "No" --> E["Probe an unknown or stale model by weight"]
>     D -- "Yes" --> F{"Lowest two latencies tie within 0.5 ms?"}
>
>     F -- "Yes" --> G["Higher configured weight wins"]
>     F -- "No" --> H["Lowest latency wins"]
>
>     C --> I["Call the model"]
>     E --> I
>     G --> I
>     H --> I
>
>     I --> J["Record duration, or a penalty on failure, into the latency history"]
> ```

> [card] [See also: Organization-level load balancing](/docs/ai-gateway/configuration/routing-rules.md#load-balancer)
>
> To apply load balancing across your organization without changing request code, use **Routing Rules** to configure Fallback, Latency, Weighted, and Round Robin strategies at the workspace level.

## Weight-based routing

Split traffic across models by percentage weights instead of latency or a fixed rotation.

### Patterns

```typescript Weight-based config patterns theme={"theme":{"light":"github-light","dark":"github-dark"}}
// Equal distribution
load_balancer: {
  type: "weight_based",
  models: [
    { model: "openai/gpt-5.6-sol", weight: 1.0 },
    { model: "anthropic/claude-sonnet-5", weight: 1.0 },
  ],
}

// Cost optimization (cheap model primary)
load_balancer: {
  type: "weight_based",
  models: [
    { model: "openai/gpt-5.4-mini", weight: 0.8 },
    { model: "openai/gpt-5.6-sol", weight: 0.2 },
  ],
}

// A/B testing
load_balancer: {
  type: "weight_based",
  models: [
    { model: "current-model", weight: 0.9 },
    { model: "experimental-model", weight: 0.1 },
  ],
}

// Multi-provider redundancy
load_balancer: {
  type: "weight_based",
  models: [
    { model: "openai/gpt-5.6-sol", weight: 0.5 },
    { model: "anthropic/claude-sonnet-5", weight: 0.3 },
    { model: "azure/gpt-5.6-sol", weight: 0.2 },
  ],
}
```

### Use cases

| Scenario | Approach | Example |
| - | - | - |
| **Cost optimization** | Heavy on cheaper models | 80% GPT-5.4 Mini, 20% GPT-5.6 Sol |
| **Performance testing** | Small traffic to new model | 95% current, 5% experimental |
| **Provider redundancy** | Split across providers | 60% OpenAI, 40% Anthropic |
| **Capacity management** | Distribute during peaks | Even split across models |

## Round Robin routing

Rotate through the configured models evenly, one request at a time, instead of splitting traffic by weight or latency.

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://router.toqan.ai/eu/v3/router/responses \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Write a marketing slogan",
    "load_balancer": {
      "type": "round_robin",
      "models": [
        {"model": "openai/gpt-5.4-mini", "weight": 0.5},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.5}
      ]
    }
  }'
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-5.4-mini",
  input: "Write a marketing slogan",
  load_balancer: {
    type: "round_robin",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.5 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.5 },
    ],
  },
});

console.log(response.output_text);
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("TOQAN_API_KEY"),
    base_url="https://router.toqan.ai/eu/v3/router",
)

response = client.responses.create(
    model="openai/gpt-5.4-mini",
    input="Write a marketing slogan",
    extra_body={
        "load_balancer": {
            "type": "round_robin",
            "models": [
                {"model": "openai/gpt-5.4-mini", "weight": 0.5},
                {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.5},
            ],
        }
    },
)

print(response.output_text)
```

```typescript TypeScript (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [{ role: "user", content: "Write a marketing slogan" }],
  load_balancer: {
    type: "round_robin",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.5 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.5 },
    ],
  },
});
```

**How selection works:**

* Each request selects the next model in the list in turn, wrapping back to the first model after the last one. With two models, requests alternate; with three, the rotation cycles through all three.
* Weights must be included in the request but do not affect selection: round robin is purely rotational, so every model receives an equal share of traffic over time regardless of the weights.
* Round robin only selects a model per request; it does not retry or fail over a failed call.

> [!NOTE]
>
> Round robin selects one model per request but does not fail over when that model errors. To add failover on top of round robin, pair `load_balancer` with [Fallbacks](/docs/ai-gateway/features/retries.md#fallbacks).

## Code examples

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://router.toqan.ai/eu/v3/router/responses \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Write a creative marketing slogan for an eco-friendly coffee brand",
    "load_balancer": {
      "type": "weight_based",
      "models": [
        {"model": "openai/gpt-5.4-mini", "weight": 0.4},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6}
      ]
    }
  }'
```

```bash cURL (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://router.toqan.ai/eu/v3/router/chat/completions \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "messages": [
      {
        "role": "user",
        "content": "Write a creative marketing slogan for an eco-friendly coffee brand"
      }
    ],
    "load_balancer": {
      "type": "weight_based",
      "models": [
        {"model": "openai/gpt-5.4-mini", "weight": 0.4},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6}
      ]
    }
  }'
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-5.4-mini",
  input: "Write a creative marketing slogan for an eco-friendly coffee brand",
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.4 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.6 },
    ],
  },
});

console.log(response.output_text);
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("TOQAN_API_KEY"),
    base_url="https://router.toqan.ai/eu/v3/router",
)

response = client.responses.create(
    model="openai/gpt-5.4-mini",
    input="Write a creative marketing slogan for an eco-friendly coffee brand",
    extra_body={
        "load_balancer": {
            "type": "weight_based",
            "models": [
                {"model": "openai/gpt-5.4-mini", "weight": 0.4},
                {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6},
            ],
        }
    },
)

print(response.output_text)
```

```typescript TypeScript (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [
    {
      role: "user",
      content: "Write a creative marketing slogan for an eco-friendly coffee brand",
    },
  ],
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.4 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.6 },
    ],
  },
});
```

```python Python (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("TOQAN_API_KEY"),
    base_url="https://router.toqan.ai/eu/v3/router",
)

response = client.chat.completions.create(
    model="openai/gpt-5.4-mini",
    messages=[
        {
            "role": "user",
            "content": "Write a creative marketing slogan for an eco-friendly coffee brand",
        }
    ],
    extra_body={
        "load_balancer": {
            "type": "weight_based",
            "models": [
                {"model": "openai/gpt-5.4-mini", "weight": 0.4},
                {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6},
            ],
        }
    },
)
```

## Monitoring

Track these metrics for optimal load balancing:

```typescript Metrics tracking example theme={"theme":{"light":"github-light","dark":"github-dark"}}
// Example monitoring setup
const metrics = {
  requestsByModel: {}, // Count per model
  costsByModel: {}, // Cost per model
  latencyByModel: {}, // Response time per model
  errorsByModel: {}, // Error rate per model
};
```

**Key Metrics:**

* **Traffic distribution**: Actual vs expected percentages.
* **Cost per model**: Monitor spending across providers.
* **Response times**: Compare latency by model.
* **Error rates**: Track failures by provider.

With `latency_based`, response time is already the selection signal. Manual weight tuning for performance is not needed; adjust weights only to influence exploration, cold-start probing, and near-tie behavior.

## Troubleshooting

**Uneven distribution**

* Check if weights are normalized correctly.
* Verify sufficient request volume (min 100 requests for accuracy).
* Monitor over longer time periods.

**Unexpected costs**

* Track actual vs expected cost distribution.
* Monitor for expensive model overuse.
* Set up cost alerts per provider.

**Performance issues**

* Check latency differences between models.
* Monitor for provider-specific slowdowns.
* Adjust weights based on performance data.

**All traffic going to one model with `latency_based`**

* Expected once one model is consistently fastest. Ten percent of requests still explore the rest of the pool to keep their latency data fresh.
* Confirm `load_balancer.type` is set to the intended strategy if an even split was expected instead.

**Selection is slow to adapt after a deploy or restart**

* Expected. Latency history is in-memory: after a restart, every model is treated as unknown until fresh samples are collected.

## Limitations

* **Probabilistic routing**: Short-term traffic may not match exact weights.
* **Minimum volume needed**: Requires sufficient requests for statistical accuracy.
* **Response variations**: Different models may return varying output quality.
* **Cost complexity**: Managing billing across multiple providers.
* **Provider dependencies**: Requires API access to all models.
* **In-memory latency state**: With `latency_based`, latency history is in-memory and does not persist across restarts.

## Advanced weight-based usage

**Environment-specific weights:**

```typescript Environment weight config theme={"theme":{"light":"github-light","dark":"github-dark"}}
const weights = {
  development: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 1.0 }, // Cheap for dev
    ],
  },
  production: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.6-sol", weight: 0.7 }, // Quality primary
      { model: "anthropic/claude-sonnet-5", weight: 0.3 }, // Backup
    ],
  },
};
```

**Dynamic weight adjustment:**

```typescript Dynamic weight calculation theme={"theme":{"light":"github-light","dark":"github-dark"}}
// Adjust weights based on performance
const calculateWeight = (latency: number, cost: number, quality: number) =>
  quality / (latency * cost);

const adjustWeights = (models) => ({
  type: "weight_based",
  models: models.map((model) => ({
    model: model.model,
    weight: calculateWeight(model.latency, model.cost, model.quality),
  })),
});
```

**With other features:**

```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "model": "openai/gpt-5.6-sol",
  "load_balancer": {
    "type": "weight_based",
    "models": [
      { "model": "openai/gpt-5.6-sol", "weight": 0.6 },
      { "model": "anthropic/claude-sonnet-5", "weight": 0.4 }
    ]
  },
  "retry": { "count": 2, "on_codes": [429] },
  "timeout": { "call_timeout": 15000 }
}
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.