> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.

# LlamaIndex framework integration

> Connect LlamaIndex to Toqan Router's AI Gateway for complete observability, built-in reliability, and access to 500+ LLMs across 30+ providers.

> [card] [AI Gateway](#ai-gateway)
>
> Route your LLM calls through the AI Gateway with a single base URL change. Zero vendor lock-in: always run on the best model at the lowest cost for your use case.

> [card] [Observability](#observability)
>
> Instrument your code with OpenTelemetry to capture traces, logs, and metrics for every LLM call, agent step, and tool use.

## AI Gateway

### Overview

LlamaIndex is a powerful framework for building RAG (Retrieval-Augmented Generation) applications with LLMs. By connecting LlamaIndex to Toqan Router's AI Gateway, you transform experimental RAG applications into production-ready systems with enterprise-grade capabilities.

### Key Benefits

Toqan Router's AI Gateway enhances your LlamaIndex applications with:

> [card] Complete Observability
>
> Track document indexing, retrieval performance, and query processing with detailed traces

> [card] Built-in Reliability
>
> Automatic fallbacks, retries, and load balancing for production resilience

> [card] Cost Optimization
>
> Real-time cost tracking and spend management across all your AI operations

> [card] Multi-Provider Access
>
> Access 500+ LLMs and 30+ providers through a single, unified integration

### Prerequisites

Before integrating LlamaIndex with Toqan Router, ensure you have:

* A Toqan Router account and API Key
* Python 3.8 or higher
* LlamaIndex installed in your project

> [!NOTE]
>
> To setup your API key, see API keys & Endpoints.

### Installation

Install LlamaIndex and required dependencies:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install llama-index llama-index-llms-openai-like
```

### Configuration

Configure LlamaIndex to use Toqan Router's AI Gateway with the `OpenAILike` class:

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from llama_index.llms.openai_like import OpenAILike
import os

# Configure OpenAI-compatible LLM with Toqan Router AI Gateway
llm = OpenAILike(
    model="openai/gpt-5.6-sol",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)
```

> **api\_base**: `https://router.toqan.ai/eu/v3/router`

### Basic RAG Example

Here's a complete example of building a RAG application with LlamaIndex through Toqan Router:

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.openai import OpenAIEmbedding
import os

# Configure LLM with Toqan Router AI Gateway
llm = OpenAILike(
    model="openai/gpt-5.6-sol",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)

# Configure embeddings through Toqan Router (required - LlamaIndex defaults to OpenAI)
embed_model = OpenAIEmbedding(
    model="openai/text-embedding-3-small",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
)

# Set as global defaults
Settings.llm = llm
Settings.embed_model = embed_model

# Load documents and create index
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

# Query the index
query_engine = index.as_query_engine()
response = query_engine.query("What is the main topic of these documents?")
print(response)
```

### Model Selection

With Toqan Router, you can use any supported model from 30+ providers:

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from llama_index.llms.openai_like import OpenAILike
import os

# Use Claude
claude_llm = OpenAILike(
    model="anthropic/claude-sonnet-5",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)

# Use Gemini
gemini_llm = OpenAILike(
    model="google-ai/gemini-2.5-flash",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)

# Use any other model
groq_llm = OpenAILike(
    model="groq/qwen/qwen3.8-27b",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)
```

### Streaming Responses

LlamaIndex supports streaming with Toqan Router:

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.openai import OpenAIEmbedding
import os

# Configure LLM
llm = OpenAILike(
    model="openai/gpt-5.6-sol",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)

# Configure embeddings
embed_model = OpenAIEmbedding(
    model="openai/text-embedding-3-small",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
)

Settings.llm = llm
Settings.embed_model = embed_model

# Create index and query engine
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(streaming=True)

# Stream response
streaming_response = query_engine.query("Explain the main concepts")
for text in streaming_response.response_gen:
    print(text, end="", flush=True)
print()
```

## Observability

### Getting Started

Integrate LlamaIndex with Toqan Router's observability to gain comprehensive insights into document indexing, retrieval performance, query processing, and LLM interactions using OpenTelemetry.

### Prerequisites

Before you begin, ensure you have:

* A Toqan Router account and API Key
* LlamaIndex installed in your project
* Python 3.8+
* OpenAI API key (or other LLM provider credentials)

### Install Dependencies

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Core LlamaIndex and OpenInference packages
pip install llama-index openinference-instrumentation-llama-index

# OpenTelemetry packages
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

# LLM providers
pip install openai anthropic

# Optional: For advanced vector stores and embeddings
pip install llama-index-vector-stores-chroma llama-index-embeddings-openai
```

### Configure Toqan Router

Set up your environment variables to connect to Toqan Router's OpenTelemetry collector:

**Unix/Linux/macOS:**

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export OTEL_EXPORTER_OTLP_ENDPOINT="https://router.toqan.ai/eu/v2/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <TOQAN_API_KEY>"
export OTEL_RESOURCE_ATTRIBUTES="service.name=llamaindex-app,service.version=1.0.0"
export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
```

**Windows (PowerShell):**

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
$env:OTEL_EXPORTER_OTLP_ENDPOINT = "https://router.toqan.ai/eu/v2/otel"
$env:OTEL_EXPORTER_OTLP_HEADERS = "Authorization=Bearer <TOQAN_API_KEY>"
$env:OTEL_RESOURCE_ATTRIBUTES = "service.name=llamaindex-app,service.version=1.0.0"
$env:OPENAI_API_KEY = "<YOUR_OPENAI_API_KEY>"
```

**Using .env file:**

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
OTEL_EXPORTER_OTLP_ENDPOINT=https://router.toqan.ai/eu/v2/otel
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <TOQAN_API_KEY>
OTEL_RESOURCE_ATTRIBUTES=service.name=llamaindex-app,service.version=1.0.0
OPENAI_API_KEY=<YOUR_OPENAI_API_KEY>
```

### Integration Example

> [!NOTE]
>
> We'll be using OpenInference as TracerProvider with LlamaIndex

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# Initialize OpenTelemetry
tracer_provider = trace_sdk.TracerProvider()
tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="https://router.toqan.ai/eu/v2/otel/v1/traces",
    headers={"Authorization": "Bearer <TOQAN_API_KEY>"}
)))
trace.set_tracer_provider(tracer_provider)

# Instrument LlamaIndex
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)

# Your LlamaIndex code is automatically traced
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("What is the main topic of these documents?")
```

### View Traces

View your traces in the [AI Studio](/docs/ai-studio/observability/quickstart.md) in the **Traces** tab.

> [!NOTE]
>
> Visit your [AI Studio](https://router.toqan.ai/eu) to view real-time analytics and traces.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.