Request timeouts

Use Cases

Preventing slow models from blocking user-facing requests indefinitely.
Setting different limits for interactive (short) vs. batch (long) workloads.
Triggering fallback logic when a provider exceeds an acceptable wait time.
Enforcing response-time SLAs on latency-sensitive features.

Quick Start

Set maximum request duration to prevent hanging requests.

curl -X POST https://router.toqan.ai/eu/v3/router/responses \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Summarize AI trends for 2024",
    "timeout": {"call_timeout": 30000}
  }'

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-5.4-mini",
  input: "Summarize AI trends for 2024",
  timeout: {
    call_timeout: 30000,
  },
});

console.log(response.output_text);

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("TOQAN_API_KEY"),
    base_url="https://router.toqan.ai/eu/v3/router",
)

response = client.responses.create(
    model="openai/gpt-5.4-mini",
    input="Summarize AI trends for 2024",
    extra_body={"timeout": {"call_timeout": 30000}},
)

print(response.output_text)

curl -X POST https://router.toqan.ai/eu/v3/router/chat/completions \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "messages": [{ "role": "user", "content": "Summarize AI trends for 2024" }],
    "timeout": {"call_timeout": 30000}
  }'

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [{ role: "user", content: "Summarize AI trends for 2024" }],
  timeout: {
    call_timeout: 30000,
  },
});

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("TOQAN_API_KEY"),
    base_url="https://router.toqan.ai/eu/v3/router",
)

response = client.chat.completions.create(
    model="openai/gpt-5.4-mini",
    messages=[{"role": "user", "content": "Summarize AI trends for 2024"}],
    extra_body={"timeout": {"call_timeout": 30000}},
)

Configuration

Parameter	Type	Required	Description
`call_timeout`	number	No	Maximum time for the model call, in milliseconds. Defaults to `720000` (12 minutes) when `timeout` is omitted.

Timeout applies to:

Model generation time for non-streaming calls. The whole call must finish within call_timeout.
Time to first token for streaming calls. See Timeout with streaming.
Every retry and fallback attempt. Each attempt gets its own call_timeout, except on the streaming Responses API, where the attempts share the request deadline.

call_timeout bounds the model call only. It does not bound Guardrails or Evaluators: each Guardrail reference on the request has its own timeout, defaulting to the Evaluator’s timeout, or 60 seconds if it has none, and configurable up to 10 minutes. See Guardrail timeout.

Timeout error

When call_timeout fires, the gateway cancels the attempt and answers with HTTP 408:

{
  "error": {
    "message": "Request timed out. The operation took longer than the configured timeout.",
    "type": "timeout_error",
    "param": null,
    "code": "timeout_error"
  }
}

408 is in the default retry codes and the default fallback codes, so the caller receives this error only after retries and fallbacks are exhausted.

Recommended Values

Use Case	Timeout (ms)	Reason
Chat applications	`15000` (15s)	User expectation for responses
Real-time features	`5000` (5s)	Immediate feedback required
Batch processing	`60000` (60s)	Complex analysis tasks
Streaming responses	`30000` (30s)	Longer generation time
Development/testing	`10000` (10s)	Fast iteration cycles

Error Handling

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

try {
  const response = await client.responses.create({
    model: "openai/gpt-5.6-sol",
    input: "Explain quantum computing",
    timeout: { call_timeout: 15000 },
  });
  console.log(response.output_text);
} catch (error) {
  if (error instanceof OpenAI.APIConnectionTimeoutError) {
    console.log('Request timed out - try increasing timeout or using faster model');
  }
}

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

try {
  const response = await client.chat.completions.create({
    model: "openai/gpt-5.6-sol",
    messages: [{ role: "user", content: "Explain quantum computing" }],
    timeout: { call_timeout: 15000 }
  });
} catch (error) {
  if (error instanceof OpenAI.APIConnectionTimeoutError) {
    console.log('Request timed out - try increasing timeout or using faster model');
    // Implement fallback behavior
  }
}

Best Practices

Timeout selection:

Set based on user experience requirements.
Consider model complexity and prompt length.
Factor in network latency (add 2-5s buffer).
Test with realistic prompts and data.

Environment-specific timeouts:

const timeouts = {
  development: 10000, // Fast feedback during dev
  staging: 20000, // Realistic testing
  production: 30000, // Conservative for reliability
};

Progressive timeouts:

// Start with short timeout, increase for retries
const attempts = [
  { timeout: 10000, model: "fast-model" },
  { timeout: 20000, model: "standard-model" },
  { timeout: 30000, model: "comprehensive-model" },
];

Fallback Integration

Timeouts work seamlessly with fallbacks:

{
  "timeout": { "call_timeout": 15000 },
  "fallbacks": [
    { "model": "openai/gpt-5.6-sol" },
    { "model": "openai/gpt-5.4-mini" }
  ]
}

Total possible time, chat completions and non-streaming calls: timeout × (1 + retry_count) × (1 + fallback_count)

Primary plus 2 fallbacks with a 15s timeout and the default 2 retries: 15s × 3 attempts × 3 models, up to 135s plus retry backoff delays.
The whole request is also capped by an internal request deadline: 12 minutes for requests that are not streaming or that run an Agent, and 30 minutes for other streaming model requests. With timeout omitted, the 12-minute default already equals the non-streaming deadline.

Troubleshooting

Frequent timeouts

Increase timeout value.
Use faster models (gpt-5.4-mini vs gpt-5.6-sol).
Reduce prompt complexity/length.
Check provider status for slowdowns.

User experience issues

Set timeout based on user expectations.
Show loading states for longer operations.
Implement progressive enhancement.
Consider async processing for long tasks.

Performance optimization

// Monitor timeout patterns
const timeoutMetrics = {
  averageResponseTime: 0,
  timeoutRate: 0,
  responseTimesByModel: {},
  optimalTimeout: 0, // 95th percentile + buffer
};

Advanced Patterns

Dynamic timeout adjustment

const getDynamicTimeout = (promptLength, modelComplexity) => {
  const baseTimeout = 10000;
  const promptFactor = Math.min(promptLength / 1000, 3); // Max 3x for long prompts
  const modelFactor = modelComplexity === "simple" ? 1 : 2;

  return baseTimeout * promptFactor * modelFactor;
};

Timeout with streaming

{
  "stream": true,
  "timeout": {
    "call_timeout": 30000
  }
}

With stream: true, call_timeout bounds the wait for the first token, not the total duration of the stream. The timer stops when the first chunk arrives, so a stream that keeps producing tokens can run longer than call_timeout. After the first token, the gateway cancels the stream only when the provider stays silent for longer than the stream idle timeout, which is 12 minutes. If the timeout fires before the first token, /v3/router/chat/completions returns HTTP 408 with the timeout_error body from Timeout error. On /v3/router/responses, a stream that is already open ends with a response.failed event whose error.code is timeout_error. On the streaming Responses API, attempts share the overall request deadline, so a retry’s first-token budget is capped at the remaining request budget divided by the attempts left.

Circuit breaker pattern

class CircuitBreaker {
  timeout: number;
  failureCount: number;
  failureThreshold: number;
  state: "CLOSED" | "OPEN" | "HALF_OPEN";
  openedAt: number | null;

  constructor(timeout: number, failureThreshold = 5) {
    this.timeout = timeout;
    this.failureCount = 0;
    this.failureThreshold = failureThreshold;
    this.state = "CLOSED";
    this.openedAt = null;
  }

  async call<T>(requestFn: () => Promise<T>): Promise<T> {
    if (this.state === "OPEN") {
      if (this.openedAt !== null && Date.now() - this.openedAt < this.timeout) {
        throw new Error("Circuit breaker is OPEN");
      }
      this.state = "HALF_OPEN";
    }

    try {
      const result = await requestFn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  onSuccess() {
    this.failureCount = 0;
    this.state = "CLOSED";
    this.openedAt = null;
  }

  onFailure() {
    this.failureCount += 1;
    if (this.failureCount >= this.failureThreshold) {
      this.state = "OPEN";
      this.openedAt = Date.now();
    }
  }
}

Limitations

Fixed timeout: call_timeout does not adapt to prompt length or model.
Fallback multiplication: Every fallback attempt gets the same call_timeout, so worst-case latency multiplies by the number of attempts.
Provider variations: Different providers have different baseline response times.
Streaming: call_timeout bounds only the first token; the stream idle timeout bounds the rest.

Monitoring

Key metrics to track:

Timeout rate: % of requests that timeout.
Average response time: Baseline performance.
95th percentile: For setting optimal timeouts.
Timeout impact: User experience degradation.
Model performance: Response times by model.

// Example monitoring
const metrics = {
  totalRequests: 0,
  timeouts: 0,
  responseTimes: [] as number[],
  recommendedTimeout: 0,
};

const calculatePercentile = (arr: number[], p: number): number => {
  const sorted = [...arr].sort((a, b) => a - b);
  return sorted[Math.floor((p / 100) * sorted.length)] ?? 0;
};

const monitorTimeouts = (responseTime: number, wasTimeout: boolean) => {
  metrics.totalRequests++;
  if (wasTimeout) {
    metrics.timeouts++;
  } else {
    metrics.responseTimes.push(responseTime);
  }

  // Calculate optimal timeout (95th percentile + 5s buffer)
  const p95 = calculatePercentile(metrics.responseTimes, 95);
  metrics.recommendedTimeout = p95 + 5000;
};

Toqan Router

Overview

APIs

MCP Portal

Routing

Security & Privacy

Model Capabilities

Server Tools

Common Scenarios

Provider setup

Quick Start

Configuration

Timeout error

Recommended Values

Error Handling

Best Practices

Fallback Integration

Troubleshooting

Advanced Patterns

Dynamic timeout adjustment

Timeout with streaming

Circuit breaker pattern

Limitations

Monitoring

​Quick Start

​Configuration

​Timeout error

​Recommended Values

​Error Handling

​Best Practices

​Fallback Integration

​Troubleshooting

​Advanced Patterns

​Dynamic timeout adjustment

​Timeout with streaming

​Circuit breaker pattern

​Limitations

​Monitoring

Quick Start

Configuration

Timeout error

Recommended Values

Error Handling

Best Practices

Fallback Integration

Troubleshooting

Advanced Patterns

Dynamic timeout adjustment

Timeout with streaming

Circuit breaker pattern

Limitations

Monitoring