> ## Documentation Index > Fetch the complete documentation index at: /docs/llms.txt > Use this file to discover all available pages before exploring further. # Databricks integration > Reach 500+ models from inside Databricks by registering the AI Gateway as an external model, and capture traces from agents on Model Serving with OpenTelemetry. **Databricks** Model Serving hosts both external models and agents authored against the MLflow `ResponsesAgent` interface. Register the **AI Gateway** as an external model to reach 500+ models from inside **Databricks**, and instrument agents with OpenTelemetry to capture their LLM calls in **Toqan Router**. > [card] [AI Gateway](#ai-gateway) > > Serve any **Toqan Router** model through a **Databricks** serving endpoint, with cost tracking and fallbacks. > [card] [Observability](#observability) > > Export traces from agents running on Model Serving, with prompts, tokens, and cost. ## Prerequisites * A **Toqan Router** account and API key * A **Databricks** workspace with Model Serving enabled * MLflow 3.11 or later, plus `databricks-sdk` and `databricks-agents` installed locally ## Store the API key Both sections below read the key from a **Databricks** secret. Keep it there rather than in an endpoint configuration, where anyone with view access on the endpoint can read it. Skip `create_scope` if the scope already exists: ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} from databricks.sdk import WorkspaceClient workspace = WorkspaceClient() workspace.secrets.create_scope(scope="toqan") workspace.secrets.put_secret( scope="toqan", key="api_key", string_value="" ) ``` ## AI Gateway **Databricks** external models accept any OpenAI-compatible endpoint through the `custom` provider, so the **AI Gateway** can back a **Databricks** serving endpoint. Set `external_model.name` to the model to serve, and point `custom_provider_url` at the **AI Gateway**: ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} import mlflow.deployments client = mlflow.deployments.get_deploy_client("databricks") client.create_endpoint( config={ "name": "toqan-gateway", "config": { "served_entities": [ { "name": "toqan-gateway", "external_model": { "name": "openai/gpt-5.6-sol", "provider": "custom", "task": "llm/v1/chat", "custom_provider_config": { "custom_provider_url": "https://router.toqan.ai/eu/v3/router/chat/completions", "bearer_token_auth": { "token": "{{secrets/toqan/api_key}}" }, }, }, } ] }, }, ) ``` Query it like any other **Databricks** serving endpoint. Calls appear in **Traces** with model, token usage, and cost: ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} response = client.predict( endpoint="toqan-gateway", inputs={"messages": [{"role": "user", "content": "Hello"}]}, ) print(response["choices"][0]["message"]["content"]) ``` > [!NOTE] > > Any enabled model works. Set `external_model.name` to its **Toqan Router** slug and see [supported models](/docs/ai-gateway/supported-models.md). One endpoint serves one model, so create an endpoint per model. ## Observability Adding OpenTelemetry to an agent already running on Model Serving sends its LLM calls to **Toqan Router**, with full request and response content, token usage, and cost. There are four changes: add an exporter at module level, flush at the end of `predict`, declare three extra packages, and pass the endpoint and key at deploy time. > [!NOTE] > > This covers agents deployed from code with `agents.deploy()`. Agents created in the no-code **Agent Bricks** builder run on an endpoint that **Databricks** manages, which exposes neither environment variables nor model dependencies, so they cannot be instrumented this way. ### Add the exporter to the agent Add this at module level in the agent file, above the agent class. It opens the connection to **Toqan Router** and attaches the instrumentor that records LLM calls. `OpenAIInstrumentor` records calls made through the OpenAI SDK, which is what `WorkspaceClient().serving_endpoints.get_open_ai_client()` returns. Agents built on another client library need the matching OpenInference instrumentor in its place, otherwise the agent deploys cleanly and produces no spans: ```python agent.py theme={"theme":{"light":"github-light","dark":"github-dark"}} import os from openinference.instrumentation.openai import OpenAIInstrumentor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor tracer_provider = TracerProvider() tracer_provider.add_span_processor( BatchSpanProcessor( OTLPSpanExporter( endpoint=os.environ["TOQAN_OTEL_ENDPOINT"], headers={"Authorization": f"Bearer {os.environ['TOQAN_API_KEY']}"}, ) ) ) OpenAIInstrumentor().instrument(tracer_provider=tracer_provider) ``` Then flush at the end of `predict`, before returning. Model Serving keeps the container alive between requests, so the flush is what sends spans promptly rather than leaving them queued: ```python agent.py theme={"theme":{"light":"github-light","dark":"github-dark"}} def predict(self, request: ResponsesAgentRequest) -> ResponsesAgentResponse: response = ... # existing agent logic, unchanged tracer_provider.force_flush(15000) return response ``` ### Declare the dependencies The serving environment installs only the packages declared on the model, so add these to the existing `pip_requirements` on `log_model`. Without them the agent fails to import at request time: ```python deploy.py theme={"theme":{"light":"github-light","dark":"github-dark"}} pip_requirements=[ # existing requirements "opentelemetry-sdk", "opentelemetry-exporter-otlp-proto-http", "openinference-instrumentation-openai", ], ``` ### Pass the endpoint and key Add these to `environment_vars` on `agents.deploy()`. `OTEL_SERVICE_NAME` is optional and sets the name used to filter traces later: ```python deploy.py theme={"theme":{"light":"github-light","dark":"github-dark"}} environment_vars={ "TOQAN_OTEL_ENDPOINT": "https://router.toqan.ai/eu/v2/otel/v1/traces", "TOQAN_API_KEY": "{{secrets/toqan/api_key}}", "OTEL_SERVICE_NAME": "databricks-support-agent", }, ``` > [!NOTE] > > Configuring OTLP through the standard `OTEL_EXPORTER_OTLP_*` variables alone does not export traces from Model Serving. The managed runtime sets up its own MLflow tracing, which takes precedence, and spans reach the **Databricks** experiment rather than **Toqan Router**. Add the exporter to the agent as shown above instead. > [!WARNING] > > A secret reference is substituted only when it is the entire value of a variable. Writing `Authorization=Bearer {{secrets/toqan/api_key}}` sends the braces literally and **Toqan Router** rejects the spans with a 401. Pass the key in its own variable, as above, and build the header in the agent. ### View traces Traces appear in [AI Studio](https://router.toqan.ai/eu) under the **Traces** tab. Each LLM call arrives as a span carrying the request and response messages, prompt and completion tokens, the resolved model and provider, and computed cost. Filter on the value set in `OTEL_SERVICE_NAME` to isolate one agent. ### Complete example This example calls a **Databricks** native endpoint. To route the agent through the **AI Gateway** instead, set `LLM_ENDPOINT` to the endpoint created above. > [accordion] Full agent module and deployment script > > ```python agent.py theme={"theme":{"light":"github-light","dark":"github-dark"}} > import os > > from databricks.sdk import WorkspaceClient > from mlflow.models import set_model > from mlflow.pyfunc import ResponsesAgent > from mlflow.types.responses import ( > ResponsesAgentRequest, > ResponsesAgentResponse, > create_text_output_item, > ) > from openinference.instrumentation.openai import OpenAIInstrumentor > from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter > from opentelemetry.sdk.trace import TracerProvider > from opentelemetry.sdk.trace.export import BatchSpanProcessor > > LLM_ENDPOINT = "databricks-gemini-3-5-flash" > > tracer_provider = TracerProvider() > tracer_provider.add_span_processor( > BatchSpanProcessor( > OTLPSpanExporter( > endpoint=os.environ["TOQAN_OTEL_ENDPOINT"], > headers={"Authorization": f"Bearer {os.environ['TOQAN_API_KEY']}"}, > ) > ) > ) > OpenAIInstrumentor().instrument(tracer_provider=tracer_provider) > > > class SupportAgent(ResponsesAgent): > def predict(self, request: ResponsesAgentRequest) -> ResponsesAgentResponse: > client = WorkspaceClient().serving_endpoints.get_open_ai_client() > completion = client.chat.completions.create( > model=LLM_ENDPOINT, > messages=[{"role": "user", "content": self._question(request)}], > max_tokens=300, > ) > tracer_provider.force_flush(15000) > return ResponsesAgentResponse( > output=[ > create_text_output_item( > text=completion.choices[0].message.content, id="answer" > ) > ] > ) > > def _question(self, request: ResponsesAgentRequest) -> str: > last_message = request.input[-1] > as_dict = ( > last_message > if isinstance(last_message, dict) > else last_message.model_dump() > ) > return as_dict.get("content", "") > > > set_model(SupportAgent()) > ``` > > ```python deploy.py theme={"theme":{"light":"github-light","dark":"github-dark"}} > import mlflow > from databricks import agents > from mlflow.models.resources import DatabricksServingEndpoint > > mlflow.set_registry_uri("databricks-uc") > > with mlflow.start_run(): > logged_agent = mlflow.pyfunc.log_model( > name="agent", > python_model="agent.py", > resources=[ > DatabricksServingEndpoint(endpoint_name="databricks-gemini-3-5-flash") > ], > pip_requirements=[ > "mlflow>=3.11", > "databricks-sdk", > "openai", > "httpx", > "opentelemetry-sdk", > "opentelemetry-exporter-otlp-proto-http", > "openinference-instrumentation-openai", > ], > ) > > registered_agent = mlflow.register_model( > logged_agent.model_uri, "catalog.schema.support_agent" > ) > > agents.deploy( > "catalog.schema.support_agent", > registered_agent.version, > environment_vars={ > "TOQAN_OTEL_ENDPOINT": "https://router.toqan.ai/eu/v2/otel/v1/traces", > "TOQAN_API_KEY": "{{secrets/toqan/api_key}}", > "OTEL_SERVICE_NAME": "databricks-support-agent", > }, > ) > ``` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.