> ## Documentation Index
> Fetch the complete documentation index at: https://docs.portkey.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# LlamaIndex (Python)

> Add Prisma AIRS AI Gateway's enterprise features to any LlamaIndex app—observability, reliability, caching, and cost control.

LlamaIndex provides a framework for building LLM applications with your data. Add the AI Gateway to get production-grade features: full observability, automatic fallbacks, semantic caching, and cost controls—all without changing your LlamaIndex code.

## Quick Start

Add the AI Gateway to any LlamaIndex app with 3 parameters:

```python theme={"system"}
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="@openai-prod/gpt-4o",        # Provider slug from Model Catalog
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY"           # Your AI Gateway API key
)

response = llm.complete("Tell me a joke")
print(response.text)
```

That's it! You now get:

* ✅ Full observability (costs, latency, logs)
* ✅ Dynamic model selection per request
* ✅ Automatic fallbacks and retries (via configs)
* ✅ Budget controls per team/project

## Why Add the AI Gateway to LlamaIndex?

LlamaIndex handles data indexing and querying. The AI Gateway adds production features:

<CardGroup cols={2}>
  <Card title="Enterprise Observability" icon="chart-line">
    Every request logged with costs, latency, tokens. Team-level analytics and debugging.
  </Card>

  <Card title="Dynamic Model Selection" icon="shuffle">
    Switch models per request. Route simple queries to cheap models, complex to advanced—automatically tracked.
  </Card>

  <Card title="Production Reliability" icon="shield-check">
    Automatic fallbacks, smart retries, load balancing—configured once, works everywhere.
  </Card>

  <Card title="Cost & Access Control" icon="dollar-sign">
    Budget limits per team/project. Rate limiting. Centralized credential management.
  </Card>
</CardGroup>

## Setup

### 1. Install Packages

```bash theme={"system"}
pip install llama-index-llms-openai
```

### 2. Add Provider in Model Catalog

1. Go to [**Model Catalog → Add Provider**](https://stratacloudmanager.paloaltonetworks.com/)
2. Select your provider (OpenAI, Anthropic, Google, etc.)
3. Choose existing credentials or create new by entering your API keys
4. Name your provider (e.g., `openai-prod`)

Your provider slug will be **`@openai-prod`** (or whatever you named it).

<Card title="Complete Model Catalog Guide →" href="/docs/aigw/product/model-catalog">
  Set up budgets, rate limits, and manage credentials
</Card>

### 3. Get AI Gateway API Key

Create your AI Gateway API key at [stratacloudmanager.paloaltonetworks.com](https://stratacloudmanager.paloaltonetworks.com/)

### 4. Use in Your Code

Replace your existing LLM initialization:

```python theme={"system"}
# Before (direct to OpenAI)
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="gpt-4o",
    api_key="OPENAI_API_KEY"
)

# After (via the AI Gateway)
llm = OpenAI(
    model="@openai-prod/gpt-4o",
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY"
)
```

**That's the only change needed!** All your existing LlamaIndex code (indexes, query engines, agents) works exactly the same.

## Switching Between Providers

Just change the model string—everything else stays the same:

```python theme={"system"}
from llama_index.llms.openai import OpenAI

# OpenAI
llm = OpenAI(
    model="@openai-prod/gpt-4o",
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY"
)

# Anthropic
llm = OpenAI(
    model="@anthropic-prod/claude-sonnet-4",
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY"
)

# Google Gemini
llm = OpenAI(
    model="@google-prod/gemini-2.0-flash",
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY"
)
```

<Note>
  The AI Gateway implements OpenAI-compatible APIs for all providers, so you always use `llama_index.llms.openai.OpenAI` regardless of which model you're calling.
</Note>

## Using with LlamaIndex Chat

LlamaIndex's chat interface works seamlessly:

```python theme={"system"}
from llama_index.llms.openai import OpenAI
from llama_index.core.llms import ChatMessage

llm = OpenAI(
    model="@openai-prod/gpt-4o",
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY"
)

messages = [
    ChatMessage(role="system", content="You are a helpful assistant"),
    ChatMessage(role="user", content="What is the capital of France?")
]

response = llm.chat(messages)
print(response.message.content)
```

## Works With All LlamaIndex Features

✅ **Query Engines** - All query types supported
✅ **Chat Engines** - Conversational interfaces
✅ **Agents** - Full agent compatibility
✅ **Streaming** - Token-by-token streaming
✅ **RAG Pipelines** - Retrieval-augmented generation
✅ **Workflows** - Complex LLM workflows

### Streaming

```python theme={"system"}
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="@openai-prod/gpt-4o",
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY"
)

# Stream completions
for chunk in llm.stream_complete("Write a short story"):
    print(chunk.delta, end="", flush=True)

# Stream chat
messages = [ChatMessage(role="user", content="Tell me a joke")]
for chunk in llm.stream_chat(messages):
    print(chunk.delta, end="", flush=True)
```

### Async Support

```python theme={"system"}
import asyncio
from llama_index.llms.openai import OpenAI

async def main():
    llm = OpenAI(
        model="@openai-prod/gpt-4o",
        api_base="https://aigw.portkey.ai/v1",
        api_key="PORTKEY_API_KEY"
    )

    # Async completion
    response = await llm.acomplete("What is 2+2?")
    print(response.text)

    # Async streaming
    async for chunk in await llm.astream_complete("Write a haiku"):
        print(chunk.delta, end="", flush=True)

asyncio.run(main())
```

### RAG with Query Engine

```python theme={"system"}
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI

# Set up LLM with the AI Gateway
llm = OpenAI(
    model="@openai-prod/gpt-4o",
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY"
)

# Load and index documents
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)

# Query with Portkey-enabled LLM
query_engine = index.as_query_engine(llm=llm)
response = query_engine.query("What is the main topic?")
print(response)
```

## Advanced Features via Configs

For production features like fallbacks, caching, and load balancing, use AI Gateway Configs:

```python theme={"system"}
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="gpt-4o",  # Default model
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY",
    default_headers={
        "x-portkey-config": "pc_your_config_id",  # Created in Strata Cloud Manager
    }
)
```

### Example: Fallbacks

```python theme={"system"}
import json
from llama_index.llms.openai import OpenAI

config = {
    "strategy": {"mode": "fallback"},
    "targets": [
        {"override_params": {"model": "@openai-prod/gpt-4o"}},
        {"override_params": {"model": "@anthropic-prod/claude-sonnet-4"}}
    ]
}

llm = OpenAI(
    model="gpt-4o",
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY",
    default_headers={"x-portkey-config": json.dumps(config)}
)

# Automatically falls back to Anthropic if OpenAI fails
response = llm.complete("Hello!")
```

### Example: Load Balancing

```python theme={"system"}
import json
config = {
    "strategy": {"mode": "loadbalance"},
    "targets": [
        {"override_params": {"model": "@openai-prod/gpt-4o"}, "weight": 0.5},
        {"override_params": {"model": "@anthropic-prod/claude-sonnet-4"}, "weight": 0.5}
    ]
}

llm = OpenAI(
    model="gpt-4o",
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY",
    default_headers={"x-portkey-config": json.dumps(config)}
)

# Requests distributed 50/50 between OpenAI and Anthropic
response = llm.complete("Hello!")
```

### Example: Caching

```python theme={"system"}
import json
config = {
    "cache": {
        "mode": "semantic",  # or "simple" for exact matches
        "max_age": 3600      # Cache for 1 hour
    },
    "override_params": {"model": "@openai-prod/gpt-4o"}
}

llm = OpenAI(
    model="gpt-4o",
    api_base="https://aigw.portkey.ai/v1",
    api_key="PORTKEY_API_KEY",
    default_headers={"x-portkey-config": json.dumps(config)}
)

# Responses cached for similar queries
response = llm.complete("What is machine learning?")
```

<Card title="Learn About Configs →" href="/docs/aigw/product/ai-gateway/configs">
  Set up fallbacks, retries, caching, load balancing, and more
</Card>

## Observability

The AI Gateway automatically logs all requests. Add custom metadata for better analytics:

Filter and analyze logs by metadata in Strata Cloud Manager.

<Card title="Observability Guide →" href="/docs/aigw/product/observability">
  Track costs, performance, and debug issues
</Card>

## Prompt Management

Use prompts from the AI Gateway's Prompt Library:

## Migration from Direct OpenAI

Already using LlamaIndex with OpenAI? Just update 3 parameters:

```python theme={"system"}
# Before
from llama_index.llms.openai import OpenAI
import os

llm = OpenAI(
    model="gpt-4o",
    api_key=os.getenv("OPENAI_API_KEY"),
    temperature=0.7
)

# After (add 2 parameters, change 1)
llm = OpenAI(
    model="@openai-prod/gpt-4o",          # Add provider slug
    api_base="https://aigw.portkey.ai/v1",     # Add this
    api_key="PORTKEY_API_KEY",             # Change to AI Gateway API key
    temperature=0.7                         # Keep existing params
)
```

**Benefits:**

* Zero code changes to your existing LlamaIndex logic
* Instant observability for all requests
* Production-grade reliability features
* Cost controls and budgets

## Next Steps

<CardGroup cols={2}>
  <Card title="Model Catalog" icon="database" href="/docs/aigw/product/model-catalog">
    Set up providers, budgets, and access control
  </Card>

  <Card title="Configs" icon="gear" href="/docs/aigw/product/ai-gateway/configs">
    Configure fallbacks, caching, and routing
  </Card>

  <Card title="Observability" icon="chart-line" href="/docs/aigw/product/observability">
    Track costs, performance, and usage
  </Card>

  <Card title="Guardrails" icon="shield" href="/docs/aigw/product/guardrails">
    Add PII detection and content filtering
  </Card>
</CardGroup>


## Related topics

- [Overview](/docs/aigw/integrations/libraries.md)
- [Llama Agents by Llamaindex](/docs/aigw/integrations/agents/llama-agents.md)
- [Tracing](/docs/aigw/product/observability/traces.md)
- [Introduction](/docs/aigw/api-reference/inference-api/introduction.md)
- [MLflow Tracing](/docs/aigw/integrations/tracing-providers/ml-flow.md)