> ## Documentation Index
> Fetch the complete documentation index at: https://docs.portkey.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenTelemetry for LLM Observability

> Leverage OpenTelemetry with Prisma AIRS AI Gateway for comprehensive LLM application observability, combining gateway insights with full-stack telemetry.

[OpenTelemetry (OTel)](https://opentelemetry.io/) is a Cloud Native Computing Foundation (CNCF) open-source framework. It provides a standardized way to collect, process, and export telemetry data (traces, metrics, and logs) from your applications. This is vital for monitoring performance, debugging issues, and understanding complex system behavior.

Many popular AI development tools and SDKs, like the Vercel AI SDK, LlamaIndex, OpenLLMetry, and Logfire, utilize OpenTelemetry for observability. The AI Gateway now embraces OTel, allowing you to send telemetry data from any OTel-compatible source directly into the AI Gateway's observability platform.

## The AI Gateway Advantage: Gateway Intelligence Meets Full-Stack Observability

The AI Gateway's strength lies in its unique combination of an intelligent **LLM Gateway** and a powerful **Observability** backend.

* **Enriched Data from the Gateway:** Your LLM calls routed through the AI Gateway are automatically enriched with deep contextual information—provider configuration, caching status, retry attempts, prompt versions, and more. This data flows seamlessly into AI Gateway Observability.

* **Holistic View with OpenTelemetry:** By adding an OTel endpoint, the AI Gateway now ingests traces and logs from your *entire* application stack, not just the LLM calls. Instrument your frontend, backend services, databases, and any other component with OTel, and send that data to the AI Gateway.

This combination provides an unparalleled, end-to-end view of your LLM application's performance, cost, and behavior. You can correlate application-level events with specific LLM interactions managed by the AI Gateway.

## How OpenTelemetry Data Flows to the AI Gateway

The following diagram illustrates how telemetry data from your instrumented applications and the AI Gateway itself is consolidated within AI Gateway Observability:

```mermaid theme={"system"}
graph LR
    subgraph Your Application Stack
        A[Application Code]
        LIB[OTel Libraries e.g. Logfire]
        A -- Instruments with --> LIB
    end

    PG[AI Gateway]
    PK_OTEL[AI Gateway OTel Endpoint aigw.portkey.ai/v1/otel]
    PK_OBS[AI Gateway Observability Backend]

    LIB -- Direct Export --> PK_OTEL
    A -- LLM Calls via --> PG
    PG -- Rich Telemetry & Logs --> PK_OBS
    PK_OTEL -- Ingests Data --> PK_OBS
```

**Explanation:**

1. Your **Application Code** is instrumented using **OTel Instrumentation Libraries**.
2. This telemetry data (traces, logs) can be sent to the **AI Gateway OTel Backend Endpoint**.
3. Simultaneously, LLM calls made via the **AI Gateway** generate their own rich, structured telemetry.
4. All this data is consolidated in the **AI Gateway Observability Stack**, giving you a unified view.

## Setting Up the AI Gateway as an OpenTelemetry Backend

To send your OpenTelemetry data to the AI Gateway, configure your OTel exporter to point to the AI Gateway's OTLP endpoint and provide your AI Gateway API Key for authentication.

**Key Environment Variables:**

```bash theme={"system"}
# the AI Gateway's OTLP HTTP endpoint for traces and logs
OTEL_EXPORTER_OTLP_ENDPOINT="https://aigw.portkey.ai/v1/otel"
# Your AI Gateway API key (ensure it's a Server Key)
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_PORTKEY_API_KEY"
```

<Note>
  Replace `YOUR_PORTKEY_API_KEY` with your actual AI Gateway API Key found in Strata Cloud Manager.
</Note>

**Signal-Specific Endpoints:**
If your OTel collector or SDK strictly requires signal-specific endpoints:

**For Traces:**
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://aigw.portkey.ai/v1/otel/v1/traces"`

**For Logs:**
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="https://aigw.portkey.ai/v1/otel/v1/logs"`

<Note>
  Remember to include the `OTEL_EXPORTER_OTLP_HEADERS` with your API key for these as well.
</Note>

## Viewing Traces

Once configured, your OpenTelemetry traces appear in Strata Cloud Manager with full visibility for your AI application:

## GenAI Semantic Conventions Support

The AI Gateway automatically enriches OpenTelemetry traces with cost and token metrics following the [GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). When you send traces with GenAI attributes, the AI Gateway automatically:

* **Extracts Token Counts**: Reads `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` from trace attributes
* **Calculates Costs**: Automatically computes costs based on token usage and model pricing
* **Identifies Models & Providers**: Extracts model information from `gen_ai.request.model` or `gen_ai.response.model` and provider from `gen_ai.system`
* **Enriches Analytics**: Makes all trace data available for cost attribution and usage analysis

This means traces sent from OpenTelemetry-instrumented applications automatically get the same cost tracking and analytics as requests made directly through the AI Gateway.

<Note>
  This feature is particularly powerful for applications using frameworks like LangChain, LlamaIndex, or other tools with built-in OpenTelemetry instrumentation that follow GenAI semantic conventions.
</Note>

## W3C Trace Context Headers

When making requests to the AI Gateway, you can use standard W3C trace context headers (`traceparent` and `baggage`) instead of gateway-specific headers. This enables seamless integration with existing OpenTelemetry-instrumented applications.

* **`traceparent`**: The trace ID and span ID are automatically extracted and used for the AI Gateway's tracing
* **`baggage`**: Key-value pairs are parsed and merged into the request metadata

For detailed usage examples and header specifications, see the [W3C Trace Context Support section in the Tracing documentation](/docs/aigw/product/observability/traces#w3c-trace-context-support).

## Why Use OpenTelemetry with the AI Gateway?

The AI Gateway's OTel backend is compatible with any OTel-compliant library. Here are a few popular ones for GenAI and general application observability:

<CardGroup cols={2}>
  <Card title="Language Agnostic" icon="globe">
    Works with any programming language that supports OpenTelemetry - Python, JavaScript, Java, Go, and more
  </Card>

  <Card title="Framework Support" icon="layer-group">
    Compatible with all major LLM frameworks through their OTel instrumentation
  </Card>

  <Card title="Zero Code Changes" icon="wand-magic-sparkles">
    Many libraries offer auto-instrumentation that requires no changes to your application code
  </Card>

  <Card title="Standards-Based" icon="certificate">
    Built on industry-standard protocols ensuring long-term compatibility
  </Card>
</CardGroup>

Navigate to the [Logs page](https://stratacloudmanager.paloaltonetworks.com/) to view your traces, filter by various attributes, and drill down into specific requests.

<Card title="Supported OTel Libraries" icon="gear" href="/docs/aigw/product/observability/opentelemetry/list-of-supported-otel-instrumenters" />

## Getting Started

<Steps>
  <Step title="Get your AI Gateway API key">
    Sign up for [the AI Gateway](https://stratacloudmanager.paloaltonetworks.com/) and grab your API key from the settings page
  </Step>

  <Step title="Choose an instrumentation library">
    Pick from our [supported integrations](#supported-integrations) based on your stack
  </Step>

  <Step title="Configure the endpoint">
    Point your OTel exporter to `https://aigw.portkey.ai/v1/logs/otel` with your API key
  </Step>

  <Step title="Start tracing">
    Run your application and view traces in Strata Cloud Manager
  </Step>
</Steps>

## Next Steps

<CardGroup cols={2}>
  <Card title="Explore Integrations" icon="puzzle-piece" href="/docs/integrations/observability-integrations">
    Browse all available OpenTelemetry integrations
  </Card>

  <Card title="View Traces" icon="chart-line" href="/docs/aigw/product/observability/logs">
    Learn how to analyze traces in the AI Gateway
  </Card>

  <Card title="Auto-Instrumentation" icon="magic" href="/docs/aigw/product/observability/auto-instrumentation">
    Discover the AI Gateway's native auto-instrumentation features
  </Card>
</CardGroup>

## Experimental Features

### Push Logs to an OpenTelemetry compatible endpoint `[Enterprise/Self-Hosted]`

<Warning>
  [OpenTelemetry conventions on GenAI Traces](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/) are still under development and have not been widely adapted, hence the feature is still experimental.
</Warning>

When enabled, AI Gateway pushes logs to an OpenTelemetry-compatible endpoint following the [semconv 1.40.0](https://opentelemetry.io/docs/specs/semconv/) semantic conventions for GenAI. This includes rich attributes such as:

* Operation and provider: `gen_ai.operation.name`, `gen_ai.provider.name`
* Request attributes: `gen_ai.request.model`, `gen_ai.request.max_tokens`, `gen_ai.request.temperature`, `gen_ai.request.top_k`, etc.
* Response attributes: `gen_ai.response.id`, `gen_ai.response.model`, `gen_ai.response.finish_reasons`
* Token usage: `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_creation.input_tokens`, `gen_ai.usage.cache_read.input_tokens`
* Structured messages: `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.system_instructions`
* Tool definitions: `gen_ai.tool.definitions`
* Embeddings support: `gen_ai.request.encoding_formats`, `gen_ai.embeddings.dimension.count`

To push logs to an OpenTelemetry compatible endpoint, you can set the following environment variables in your deployment configuration:

This is an example configuration for LangSmith, but all OpenTelemetry compatible endpoints are supported.

```yaml theme={"system"}
EXPERIMENTAL_GEN_AI_OTEL_TRACES_ENABLED: "true"
EXPERIMENTAL_GEN_AI_OTEL_EXPORTER_OTLP_ENDPOINT: https://api.smith.langchain.com/otel
EXPERIMENTAL_GEN_AI_OTEL_EXPORTER_OTLP_HEADERS: x-api-key=langsmith-api-key
EXPERIMENTAL_GEN_AI_OTEL_RESOURCE_ATTRIBUTES: deployment.environment=production,service.version=1.0
```

**Environment Variables:**

* `EXPERIMENTAL_GEN_AI_OTEL_TRACES_ENABLED`: Set to `"true"` to enable pushing logs to an OpenTelemetry endpoint
* `EXPERIMENTAL_GEN_AI_OTEL_EXPORTER_OTLP_ENDPOINT`: The OpenTelemetry OTLP endpoint URL (e.g., `https://api.smith.langchain.com/otel`)
* `EXPERIMENTAL_GEN_AI_OTEL_EXPORTER_OTLP_HEADERS`: Comma-separated list of headers in the format `key=value` (e.g., `x-api-key=langsmith-api-key`)
* `EXPERIMENTAL_GEN_AI_OTEL_RESOURCE_ATTRIBUTES`: Comma-separated `key=value` pairs added as OTLP resource attributes (e.g., `deployment.environment=production`)

The logs are pushed to the endpoint specified in `EXPERIMENTAL_GEN_AI_OTEL_EXPORTER_OTLP_ENDPOINT` at the `/v1/traces` path.

<Card title="Detailed Documentation" icon="book" href="/docs/aigw/product/enterprise-offering/otel/complete-logs">
  View comprehensive documentation including attribute mapping, span structure, and integration guides.
</Card>


## Related topics

- [Observability (OpenTelemetry)](/docs/aigw/product/observability.md)
- [OpenTelemetry(OTel) Export](/docs/aigw/product/enterprise-offering/otel/otel.md)
- [OpenTelemetry Python SDK](/docs/aigw/integrations/tracing-providers/opentelemetry-python-sdk.md)
- [OpenLIT](/docs/aigw/integrations/tracing-providers/openlit.md)
- [Pydantic AI](/docs/aigw/integrations/agents/pydantic-ai.md)