Call various LLMs like Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, and AWS Bedrock with minimal code changes.
Speed up agent responses and save costs by storing past responses in the Portkey cache. Choose between Simple and Semantic cache modes.
Set up fallbacks between different LLMs, load balance requests across multiple instances, set automatic retries, and request timeouts.
Get comprehensive logs of agent interactions, including cost, tokens used, response time, and function calls. Send custom metadata for better analytics.
Access detailed logs of agent executions, function calls, and interactions. Debug and optimize your agents effectively.
Implement budget limits, role-based access control, and audit trails for your agent operations.
Capture and analyze user feedback to improve agent performance over time.
## 1. Interoperability - Calling Different LLMs
When building with Swarm, you might want to experiment with different LLMs or use specific providers for different agent tasks. Portkey makes this seamless - you can switch between OpenAI, Anthropic, Gemini, Mistral, or cloud providers without changing your agent code.
Instead of managing multiple API keys and provider-specific configurations, Portkey's Virtual Keys give you a single point of control. Here's how you can use different LLMs with your Swarm agents:
Call various LLMs like Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, and AWS Bedrock with minimal code changes.
Speed up your requests and save money on LLM calls by storing past responses in the Portkey cache. Choose between Simple and Semantic cache modes.
Set up fallbacks between different LLMs or providers, load balance your requests across multiple instances or API keys, set automatic retries, and request timeouts.
Portkey automatically logs all the key details about your requests, including cost, tokens used, response time, request and response bodies, and more. Send custom metadata and trace IDs for better analytics and debugging.
Use Portkey as a centralized hub to store, version, and experiment with prompts across multiple LLMs, and seamlessly retrieve them in your LlamaIndex app for easy integration.
Improve your LlamaIndex app by capturing qualitative & quantitative user feedback on your requests.
Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
Much of these features are driven by **Portkey's Config architecture**. On the Portkey app, we make it easy to help you *create*, *manage*, and *version* your Configs so that you can reference them easily in Llamaindex.
## Saving Configs in the Portkey App
Head over to the Configs tab in Portkey app where you can save various provider Configs along with the reliability and caching features. Each Config has an associated slug that you can reference in your Llamaindex code.
### Through the API
You can also create keys programmatically:
Every administrative action is recorded with:
* User identity
* Action type and target resource
* Timestamp
* IP address
* Request details
This audit trail helps maintain compliance and provides accountability for all administrative changes.
Based on your access level, you might see the relevant permissions on the API key modal - tick the ones you'd like, name your API key, and save it.
## Authentication with SDKs
### Portkey SDKs
# Chat Completions Example
We're introducing comprehensive usage controls for both Virtual Keys and API Keys, giving platform teams precise control over LLM access and resource consumption. This release introduces:
* **Time-based Access Control**: Create short-lived keys that automatically expire after a specified duration – perfect for temporary access needs like POCs or time-limited projects
* **Resource Consumption Limits**: Set granular limits including:
* Requests per minute (RPM) / Request per hour / Request per day
* Tokens per minute (TPM) / Tokens per hour / Tokens per day
* Budget caps based on cost incurred or tokens consumed, with periodic reset options (weekly/monthly)
**Enhanced Provider Features**
* Perplexity Integration: Full support for Perplexity API's advanced features including search domain filtering, related questions generation, and citation capabilities
**Universal Identity Management**
* **SSO Integration**: Support for all major identity providers through OIDC/SAML standards, enabling seamless enterprise authentication
* **Automated User Management**: SCIM provisioning for automatic user lifecycle management - from onboarding to role changes and offboarding
* **Granular Access Control**: Define precise access patterns and manage permissions at both user and workspace levels
* **Workspace Management API**: Programmatically manage workspaces, user invites, and access controls
**Private Deployments**
Updated documentation for fully private Portkey installations with enhanced security configurations [(*Docs*)](https://github.com/Portkey-AI/helm/tree/main/charts)
## Integrations
**New Providers**
We [won](https://www.linkedin.com/posts/1rohitagarwal_we-just-won-the-best-growth-strategy-award-activity-7272134964110868480-_mvc/?utm_source=share\&utm_medium=member_desktop) the NetApp Excellerator Award, launched [prompt.new](https://prompt.new/) for faster development, added folder organization and AI suggestions for prompt templates, introduced multi-workspace analytics.
Plus, there's now support for OpenAI's Realtime API and much more. Let's dive in!
## Summary
| Area | Key Updates |
| :----------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Platform | • See multi-workspace analytics & logs on a single dashboard
When Premera Blue Cross’ Director of Platform Engineering needed an AI Gateway, they chose Portkey. Why? Because traditional API gateways [weren’t built](https://portkey.ai/blog/ai-gateway-vs-api-gateway) for AI-first companies. Are you in the same boat? Schedule an [expert consultation here](https://calendly.com/portkey-ai/quick-consult).
***
## Platform
#### Prompt Management
* Type [prompt.new](https://prompt.new) in your browser to spin up a new prompt playground! [Try it now →](https://prompt.new)
* Organize your prompt templates with folders and subfolders:
* Use AI to write and improve your prompts - right inside the playground:
* Add custom tags/labels like `staging`, `production` to any prompt version to track changes, and call them directly:
#### Analytics
**Org-wide Executive Reports**
Monitor analytics and logs across all workspaces in your organization through a unified dashboard. This centralized view provides comprehensive insights into cost, performance, and accuracy metrics for your deployed AI applications.
* Track token usage patterns across requests & responses
* You can now filter logs and analytics with specific Portkey API keys. This is useful if you are tying a particular key to an internal user and want to see their usage!
#### Enterprise
We've strengthened our enterprise authentication capabilities with comprehensive cloud provider integrations.
* Expanded AWS authentication options, for adding your Bedrock models or Sagemaker deployments:
* IMDS-based auth (recommended for AWS environments)
* IRSA-based auth for Kubernetes workloads
* Role-based auth for non-AWS environments
* STS integration with assumed roles
* Also expanded the Azure Integration:
* Azure Entra (formerly Active Directory)
* Managed identity support
* Granular access permissions for API Keys and Virtual Keys across your organization
* Support for sending Azure `deploymentConfig` while making Virtual Keys through API. [Docs](/api-reference/admin-api/control-plane/virtual-keys/create-virtual-key)
***
#### More Customer Love
Felipe & team are building [beconfident](https://beconfident.app/), and here's what they had to say about Portkey:
> "Now that we've seen positive results, we're going to move all our prompts to Portkey."
***
## Integrations
#### Providers
February continues the momentum with powerful enterprise features:
* **Azure Marketplace**: Portkey is now available on Azure Marketplace for simplified enterprise procurement
* **Multiple Owners**: Organizations can now have multiple owner accounts for improved management
* **Enhanced Role Management**: Change member roles directly from the UI
* **User Key Creation**: Create user-specific keys directly from the UI interface
* **Default Configs**: Attach default configurations and metadata to any API key you create
* **Performance Optimization**: Updated cache implementation to avoid redundant Redis calls
* **Browser SDK Support**: Run our SDK directly in the browser with Cross-Origin access support
## New Models & Integrations
"Describing Portkey as merely useful would be an understatement; it's a must-have." - @AManInTech
## Our Stories
**Kicking off 2025 with major releases! 🎉**
January marks a milestone for Portkey with our first industry report — we analyzed over 2 trillion tokens flowing through Portkey to find out production patterns for LLMs.
We're also expanding our platform capabilities with advanced PII redaction, JWT authentication, comprehensive audit logs, unified files & batches API, and support for private LLMs. Latest LLMs like Deepseek R1, OpenAI o3, and Gemini thinking model are also integrated with Portkey.
Plus, we are attending the [AI Engineer Summit in New York](https://x.com/PortkeyAI/status/1886629690615747020) in February, and hosting in-person meetups in [Mumbai](https://lu.ma/bgiyw0cy) & [NYC](https://lu.ma/vmf0egzl).
Let's dive in!
## Summary
| Area | Key Updates |
| :----------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Benchmark | • Released [LLMs in Prod Report 2025](https://portkey.ai/llms-in-prod-25) analyzing 2T+ tokens
Our comprehensive analysis of 2T+ tokens processed through Portkey's Gateway reveals fascinating insights about how teams are deploying LLMs in production. Here are the key findings:
They were able to do this because of three things:
* They could build reusable prompts with our partial templates
* Our versioning let them confidently roll out changes
* And they didn't have to refactor anything thanks to our OpenAI-compatible APIs
***
## Integrations
#### Models & Providers
We are attending the [AI Engineer Summit in NYC](https://x.com/PortkeyAI/status/1886629690615747020) this February and have some extra event passes to share! Reach out to us [on Discord](https://portkey.wiki/community) to ask for a pass.
We are also hosting small meetups in NYC and Mumbai this month to meet with local engineering leaders and ML/AI platform leads. Register for them below:
Last month we hosted an inspiring AI practitioners meetup with Ojasvi Yadav and Anudeep Yegireddi to discuss the role of Event-Driven Architecture in building Multi-Agent Systems using and MCP.
[Read event report here →](https://portkey.ai/blog/event-driven-architecture-for-ai-agents)
Essential reading for your AI infrastructure:
* [LLMs in Prod Report 2025](https://portkey.ai/llms-in-prod-25): Comprehensive analysis of production LLM usage patterns
* [The Real Cost of Building an LLM Gateway](https://portkey.ai/blog/the-cost-of-building-an-llm-gateway/): Understanding infrastructure investments
* [Critical Role of Audit Logs](https://portkey.ai/blog/beyond-implementation-why-audit-logs-are-critical-for-enterprise-ai-governance/): Enterprise AI governance
* [Error Library](https://portkey.ai/error-library): New documentation covering common errors across 30+ providers
* [Deepseek on Fireworks](https://x.com/PortkeyAI/status/1885231024483033295): How to use Portkey with Fireworks to call Deepseek's R1 model for reasoning tasks
## Improvements
* Token counting is now more accurate for Anthropic streams
* Added logprobs for Vertex AI
* Improved usage object mapping for Perplexity
* Error handling is more robust across all SDKs
***
## Support
Our flagship release this month is the official launch of the Prompt Engineering Studio, bringing professional-grade prompt development to teams of all sizes:
* **Version control**: Track changes, compare versions, and roll back when needed
* **Collaborative workflow**: Work together with your team on prompt development
* **Variables & templates**: Create reusable prompt components and patterns
* **Testing framework**: Validate performance before production deployment
* **Production integration**: Seamlessly connect to your applications
Read about our design journey in our [detailed case study](https://portkey.ai/blog/portkey-prompt-engineering-studio-a-user-centric-design-facelift/).
**Claude Multimodal Capabilities**
You can now send images to Claude models across various providers:
* Send image URLs to Claude via Anthropic, Vertex, or Bedrock APIs
* Full support for multimodal conversations and analysis
* Consistent interface across all Claude providers
**PDF Support for Claude**
Enhance your document processing workflows with native PDF support:
* Send PDF files directly to Claude requests
* Process long-form documents without manual extraction
* Maintain formatting and structure in analysis
**Thinking Mode Expansion**
Access model reasoning across all major providers:
* Support for Anthropic (Bedrock, Vertex), OpenAI, and more
* Full compatibility with streaming responses
* Complete observability of reasoning process
* Consistent interface across all supported models
## Enterprise
**University Validation**
We're proud to announce that Portkey is being evaluated as the official AI Gateway solution by leading academic institutions:
* Harvard University
* Princeton University
* University of California, Berkeley
* Cornell University
* New York University
* Lehigh University
* Bowdoin College
Learn more about the [Internet2 NET+ AI service evaluation](https://internet2.edu/new-net-service-evaluations-for-ai-services/).
**Enhanced Security Controls**
* **AWS KMS Integration**: Bring your own encryption keys for maximum security
* **SCIM Support**: Automated user provisioning with Okta & Azure Entra (AD)
* **Organizational Controls**: Enforce guardrails and metadata requirements at the org level
* **Usage Limit Notifications**: Configure email alerts for rate/budget/usage thresholds
**Simplified Deployment**
* **CloudFormation Template**: 1-click deployment of Portkey Gateway on AWS EC2
* **Real-Time Model Pricing**: Pricing configs now fetched dynamically from control plane
* **Internal POD Communication**: Secure HTTPS between components
* **Enhanced Metrics**: Track last byte latency for streaming responses
## Gateway & Providers
**New Providers**
Great, this is setup and ready now.
The gemini model doesn't need a `system` prompt, so we can ignore it and create a prompt like this.
## 2. Write the config for a 50-50 test
To run the experiment, lets create a [config](/product/ai-gateway/configs) in Portkey that can automatically route requests between these 2 prompts.
We pulled the `id` for both these prompts from our Prompts list page and will use them in our config. This is what it finally looks like.
```json
{
"strategy": {
"mode": "loadbalance"
},
"targets": [{
"prompt_id": "0db0d89c-c1f6-44bc-a976-f92e24b39a19",
"weight": 0.5
},{
"prompt_id": "pp-blog-outli-840877",
"weight": 0.5
}]
}
```
We've created a load balanced config that will route 50% of the traffic to each of the 2 prompt IDs mentioned in it. We can save this config and fetch its ID.
Create the config and fetch the ID
## 3. Make requests using this config
Lets use this config to start making requests from our application. We will use the [prompt completions API](/portkey-endpoints/prompts/prompt-completion) to make the requests and add the config in our headers.
We find that the `gpt-3.5-turbo` prompt is at 4.71 average feedback after 20 attempts, while `gemini-pro` is at 4.11. While we definitely need more data and examples, let's assume for now that we wanted to start directing more traffic to it.
We can edit the `weight` in the config to direct more traffic to `gpt-3.5-turbo`. The new config would look like this:
```json
{
"strategy": {
"mode": "loadbalance"
},
"targets": [{
"prompt_id": "0db0d89c-c1f6-44bc-a976-f92e24b39a19",
"weight": 0.8
},{
"prompt_id": "pp-blog-outli-840877",
"weight": 0.2
}]
}
```
This directs 80% of the traffic to OpenAI.
And we're done! We were able to set up an effective A/B test between prompts and models without fretting.
## Next Steps
As next explorations, we could create versions of the prompts and test between them. We could also test 2 prompts on `gpt-3.5-turbo` to judge which one would perform better.
Try creating a prompt to create tweets and see which model or prompts perform better.
Portkey allows a lot of flexibility while experimenting with prompts.
## Bonus: Add a fallback
We've noticed that we hit the OpenAI rate limits at times. In that case, we can fallback to the gemini prompt so the user doesn't experience the failure.
Adjust the config like this, and your fallback is setup!
```json
{
"strategy": {
"mode": "loadbalance"
},
"targets": [{
"strategy": {"mode": "fallback"},
"targets": [
{
"prompt_id": "0db0d89c-c1f6-44bc-a976-f92e24b39a19",
"weight": 0.8
}, {
"prompt_id": "pp-blog-outli-840877"
}]
},{
"prompt_id": "pp-blog-outli-840877",
"weight": 0.2
}]
}
```
If you need any help in further customizing this flow, or just have more questions as you run experiments with prompts / models, please reach out to us at [hello@portkey.ai](mailto:hello@portkey.ai) (We reply fast!)
# Function Calling
Source: https://docs.portkey.ai/docs/guides/getting-started/function-calling
Get the LLM to interact with external APIs!
As described in the [Enforcing JSON Schema cookbook](/guides/use-cases/enforcing-json-schema-with-anyscale-and-together), LLMs are now good at generating outputs that follow a specified syntax. We can combine this LLM ability with their reasoning ability to let LLMs interact with external APIs. **This is called Function (or Tool) calling.** In simple terms, function calling:
1. Informs the user when a question can be answered using an external API
2. Generates a valid request in the API's format
3. Converts the API's response to a natural language answer
Function calling is currently supported on select models on **Anyscale**, **Together AI**, **Fireworks AI**, **Google Gemini**, and **OpenAI**. Using Portkey, you can easily experiment with function calling across various providers and gain confidence to ship it to production.
**Let's understand how it works with an example**:
We want the LLM to tell what's the temperature in Delhi today. We'll use a "Weather API" to fetch the weather:
```Node Node
import Portkey from "portkey-ai";
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY",
virtualKey: "ANYSCALE_VIRTUAL_KEY",
});
// Describing what the Weather API does and expects
let tools = [
{
"type": "function",
"function": {
"name": "getWeather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state",
},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location"],
},
},
}
];
let response = await portkey.chat.completions.create({
model: "mistralai/Mixtral-8x7B-Instruct-v0.1",
messages: [
{"role": "system", "content": "You are helpful assistant."},
{"role": "user", "content": "What's the weather like in Delhi - respond in JSON"}
],
tools,
tool_choice: "auto", // auto is default, yet explicit
});
console.log(response.choices[0].finish_reason)
```
Here, we've defined what the Weather API expects for its requests in the `tool` param, and set `tool_choice` to auto. So, based on the user messages, the LLM will decide if it should do a function call to fulfill the request. Here, it will choose to do that, and we'll see the following output:
```Node
{
"role": "assistant",
"content": null,
"tool_calls": [
"id": "call_x8we3xx",
"type": "function",
"function": {
"name": "getWeather",
"arguments": '{\n "location": "Delhi, India",\n "format": "celsius"\n}'
}
],
}
```
We can just take the `tool_call` made by the LLM, and pass it to our `getWeather` function - it should return a proper response to our query. We then take that response and send it to our LLM to complete the loop:
```Node
/**
* getWeather(..) is a utility to call external weather service APIs
* Responds with: {"temperature": 20, "unit": "celsius"}
**/
let weatherData = await getWeather(JSON.parse(arguments));
let content = JSON.stringify(weatherData);
// Push assistant and tool message from earlier generated function arguments
messages.push(assistantMessage); //
messages.push({
role: "tool",
content: content,
toolCallId: "call_x8we3xx"
name: "getWeather"
});
let response = await portkey.chat.completions.create({
model: "mistralai/Mixtral-8x7B-Instruct-v0.1",
tools:tools,
messages:messages,
tool_choice: "auto",
});
```
We should see this final output:
```Node
{
"role": "assistant",
"content": "It's 30 degrees celsius in Delhi, India.",
}
```
## Function Calling Workflow
Recapping, there are 4 key steps to doing function calling, as illustrated below:
Function Calling Workflow
## Supporting Models
While most providers have standard function calling as illustrated above, models on Together AI & select new models on OpenAI (`gpt-4-turbo-preview`, `gpt-4-0125-preview`, `gpt-4-1106-preview`, `gpt-3.5-turbo-0125`, and `gpt-3.5-turbo-1106`) also support **parallel function calling** - here, you can pass multiple requests in a single query, the model will pick the relevant tool for each query, and return an array of `tool_calls` each with a unique ID. ([Read here for more info](https://platform.openai.com/docs/guides/function-calling/parallel-function-calling))
| Model/Provider | Standard Function Calling | Parallel Function Calling |
| --------------------------------------------------------------- | -------------------------------------------------------------------------------- | ----------------------------------- |
| mistralai/Mistral-7B-Instruct-v0.1 Anyscale | 
### 2. Caching, Fallbacks, Load Balancing
* **Fallbacks**: Ensure your application remains functional even if a primary service fails.
* **Load Balancing**: Efficiently distribute incoming requests among multiple models.
* **Semantic Caching**: Reduce costs and latency by intelligently caching results.
Toggle these features by saving *Configs* (from the Portkey dashboard > Configs tab).
If we want to enable semantic caching + fallback from Llama2 to Mistral, your Portkey config would look like this:
```py
{
"cache": { "mode": "semantic" },
"strategy": { "mode": "fallback" },
"targets": [
{
"provider": "anyscale",
"api_key": "...",
"override_params": { "model": "meta-llama/Llama-2-7b-chat-hf" }
},
{
"provider": "anyscale",
"api_key": "...",
"override_params": { "model": "mistralai/Mistral-7B-Instruct-v0.1" }
}
]
}
```
Now, just send the Config ID with `x-portkey-config` header:
```py
""" OPENAI PYTHON SDK """
import openai, json
PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1"
PORTKEY_HEADERS = {
'Content-Type': 'application/json',
'x-portkey-api-key': 'PORTKEY_API_KEY',
# **************************************
'x-portkey-config': 'CONFIG_ID'
# **************************************
}
client = openai.OpenAI(base_url=PORTKEY_GATEWAY_URL, default_headers=PORTKEY_HEADERS)
response = client.chat.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.1",
messages=[{"role": "user", "content": "Say this is a test"}]
)
print(response.choices[0].message.content)
```
```py
""" OPENAI NODE SDK """
import OpenAI from 'openai';
const PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1"
const PORTKEY_HEADERS = {
'Content-Type': 'application/json',
'x-portkey-api-key': 'PORTKEY_API_KEY',
// **************************************
'x-portkey-config': 'CONFIG_ID'
// **************************************
}
const openai = new OpenAI({baseURL:PORTKEY_GATEWAY_URL, defaultHeaders:PORTKEY_HEADERS});
async function main() {
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'mistralai/Mistral-7B-Instruct-v0.1',
});
console.log(chatCompletion.choices[0].message.content);
}
main();
```
```py
""" REQUESTS LIBRARY """
import requests, json
PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1/chat/completions"
PORTKEY_HEADERS = {
'Content-Type': 'application/json',
'x-portkey-api-key': 'PORTKEY_API_KEY',
# **************************************
'x-portkey-config': 'CONFIG_ID'
# **************************************
}
DATA = {"messages": [{"role": "user", "content": "What happens when you mix red & yellow?"}]}
response = requests.post(PORTKEY_GATEWAY_URL, headers=PORTKEY_HEADERS, json=DATA)
print(response.text)
```
```sh
""" CURL """
curl "https://api.portkey.ai/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: PORTKEY_API_KEY" \
-H "x-portkey-config: CONFIG_ID" \
-d '{ "messages": [{"role": "user", "content": "Say 'Test'."}] }'
```
For more on Configs and other gateway feature like Load Balancing, [check out the docs.](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations)
### 3. Collect Feedback
Gather weighted feedback from users and improve your app:
```py
""" REQUESTS LIBRARY """
import requests
import json
PORTKEY_FEEDBACK_URL = "https://api.portkey.ai/v1/feedback" # Portkey Feedback Endpoint
PORTKEY_HEADERS = {
"x-portkey-api-key": "PORTKEY_API_KEY",
"Content-Type": "application/json",
}
DATA = {
"trace_id": "anyscale_portkey_test", # On Portkey, you can append feedback to a particular Trace ID
"value": 1,
"weight": 0.5
}
response = requests.post(PORTKEY_FEEDBACK_URL, headers=PORTKEY_HEADERS, data=json.dumps(DATA))
print(response.text)
```
```sh
""" CURL """
curl "https://api.portkey.ai/v1/feedback" \
-H "x-portkey-api-key: PORTKEY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"trace_id": "anyscale_portkey_test",
"value": 1,
"weight": 0.5
}'
```
### 4. Continuous Fine-Tuning
Once you start logging your requests and their feedback with Portkey, it becomes very easy to 1️) Curate & create data for fine-tuning, 2) Schedule fine-tuning jobs, and 3) Use the fine-tuned models!
Fine-tuning is currently enabled for select orgs - please request access on [Portkey Discord](https://discord.gg/sDk9JaNfK8) and we'll get back to you ASAP.
#### Conclusion
Integrating Portkey with Anyscale helps you build resilient LLM apps from the get-go. With features like semantic caching, observability, load balancing, feedback, and fallbacks, you can ensure optimal performance and continuous improvement.
[Read full Portkey docs here.](https://portkey.ai/docs/) | [Reach out to the Portkey team.](https://discord.gg/sDk9JaNfK8)
# Deepinfra
Source: https://docs.portkey.ai/docs/guides/integrations/deepinfra
[](https://colab.research.google.com/drive/1SiyWV8ER-Gp2GEkMr9aA3KhebdeEJHBK?usp=sharing)
## Portkey + DeepInfra
[Portkey](https://app.portkey.ai/) is the Control Panel for AI apps. With it's popular AI Gateway and Observability Suite, hundreds of teams ship reliable, cost-efficient, and fast apps.
With Portkey, you can
* Connect to 150+ models through a unified API,
* View 40+ metrics & logs for all requests,
* Enable semantic cache to reduce latency & costs,
* Implement automatic retries & fallbacks for failed requests,
* Add custom tags to requests for better tracking and analysis and more.
### Quickstart
Since Portkey is fully compatible with the OpenAI signature, you can connect to the Portkey AI Gateway through OpenAI Client.
* Set the `base_url` as `PORTKEY_GATEWAY_URL`
* Add `default_headers` to consume the headers needed by Portkey using the `createHeaders` helper method.
You will need Portkey and Deepinfra API keys to run this notebook.
* Sign up for Portkey and generate your API key [here](https://app.portkey.ai/).
* Get your Deepinfra key [here](https://deepinfra.com/dash/api%5Fkeys)
```sh
!pip install -qU portkey-ai openai
```
### With OpenAI Client
```python
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
from google.colab import userdata
client = OpenAI(
api_key= userdata.get('DEEPINFRA_API_KEY'), ## replace it your Mistral API key
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="deepinfra",
api_key= userdata.get('PORTKEY_API_KEY'), ## replace it your Portkey API key
)
)
chat_complete = client.chat.completions.create(
model="meta-llama/Meta-Llama-3-70B-Instruct",
messages=[{"role": "user",
"content": "Who are you?"}],
)
print(chat_complete.choices[0].message.content)
```
```sh
Nice to meet you! I'm LLaMA, a large language model trained by a team of researcher at Meta AI. My primary function is to generate human-like responses to a wide range of questions and topics, from science and history to entertainment and culture.
I'm not a human, but rather an artificial intelligence designed to simulate conversation and answer questions to the best of my knowledge. I've been trained on a massive dataset of text from the internet and can respond in multiple languages.
I can help with things like:
* Answering questions on a variety of topics
* Generating text on a given topic or subject
* Translating text from one language to another
* Summarizing long pieces of text into shorter, more digestible versions
* Offering suggestions or ideas for creative projects
* Even just having a conversation and chatting about your day or interests!
So, what's on your mind? Want to chat about something specific or just see where the conversation takes us?
```
### Observability with Portkey
By routing requests through Portkey you can track a number of metrics like - tokens used, latency, cost, etc.
Here's a screenshot of the dashboard you get with Portkey:
# Groq
Source: https://docs.portkey.ai/docs/guides/integrations/groq
[](https://colab.research.google.com/drive/1USSOBS3uWrpgZirIIAJmlyC9XHnFCVSQ?usp=sharing)
## Portkey + Groq
[Portkey](https://app.portkey.ai/) is the Control Panel for AI apps. With it's popular AI Gateway and Observability Suite, hundreds of teams ship reliable, cost-efficient, and fast apps.
With Portkey, you can
* Connect to 150+ models through a unified API,
* View 40+ metrics & logs for all requests,
* Enable semantic cache to reduce latency & costs,
* Implement automatic retries & fallbacks for failed requests,
* Add custom tags to requests for better tracking and analysis and more.
### Use blazing fast Groq API with OpenAI Compatibility using Portkey!
Since Portkey is fully compatible with the OpenAI signature, you can connect to the Portkey AI Gateway through OpenAI Client.
* Set the `base_url` as `PORTKEY_GATEWAY_URL`
* Add `default_headers` to consume the headers needed by Portkey using the `createHeaders` helper method.
You will need Portkey and Groq API keys to run this notebook.
* Sign up for Portkey and generate your API key [here](https://app.portkey.ai/).
* Get your Groq API key [here](https://console.groq.com/keys)
```sh
!pip install -qU portkey-ai openai
```
### With OpenAI Client
```py
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
from google.colab import userdata
client = OpenAI(
api_key= userdata.get('GROQ_API_KEY'), ## replace it your Groq API key
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="groq",
api_key= userdata.get('PORTKEY_API_KEY'), ## replace it your Portkey API key
)
)
chat_complete = client.chat.completions.create(
model="llama3-70b-8192",
messages=[{"role": "user",
"content": "What's the purpose of Generative AI?"}],
)
print(chat_complete.choices[0].message.content)
```
```
The primary purpose of generative AI is to create new, original, and often realistic data or content, such as images, videos, music, text, or speeches, that are similar to those created by humans. Generative AI models are designed to generate new data samples that are indistinguishable from real-world data, allowing for a wide range of applications and possibilities. Some of the main purposes of generative AI include:
1. **Data augmentation**: Generating new data to augment existing datasets, improving machine learning model performance, and reducing overfitting.
2. **Content creation**: Automating the creation of content, such as music, videos, or articles, that can be used for entertainment, education, or marketing purposes.
3. **Simulation and modeling**: Generating synthetic data to simulate real-world scenarios, allowing for experimentation, testing, and analysis in various fields, such as healthcare, finance, or climate modeling.
4. **Personalization**: Creating personalized content, recommendations, or experiences tailored to individual users' preferences and behaviors.
5. **Creative assistance**: Providing tools and inspiration for human creators, such as artists, writers, or musicians, to aid in their creative processes.
6. **Synthetic data generation**: Generating realistic synthetic data to protect sensitive information, such as personal data or confidential business data.
7. **Research and development**: Facilitating research in various domains, such as computer vision, natural language processing, or robotics, by generating new data or scenarios.
8. **Entertainment and leisure**: Creating engaging and interactive experiences, such as games, chatbots, or interactive stories.
9. **Education and training**: Generating educational content, such as interactive tutorials, virtual labs, or personalized learning materials.
10. **Healthcare and biomedical applications**: Generating synthetic medical images, patient data, or clinical trials data to aid in disease diagnosis, treatment planning, and drug discovery.
Some of the key benefits of generative AI include:
* Increased efficiency and productivity
* Improved accuracy and realism
* Enhanced creativity and inspiration
* Accelerated research and development
* Personalized experiences and services
* Cost savings and reduced data collection costs
However, it's essential to address the potential risks and concerns associated with generative AI, such as:
* Misuse and abuse of generated content
* Bias and unfairness in AI-generated data
* Privacy and security concerns
* Job displacement and labor market impacts
As generative AI continues to evolve, it's crucial to develop and implement responsible AI practices, ensuring that these technologies are used for the betterment of society and humanity.
```
### With Portkey Client
Note: You can safely store your Groq API key in [Portkey](https://app.portkey.ai/) and access models using virtual key
```py
from portkey_ai import Portkey
portkey = Portkey(
api_key = userdata.get('PORTKEY_API_KEY'), # replace with your Portkey API key
virtual_key= "groq-431005", # replace with your virtual key for Groq AI
)
```
```py
completion = portkey.chat.completions.create(
messages= [{ "role": 'user', "content": 'Who are you?'}],
model= 'llama3-70b-8192',
max_tokens=250
)
print(completion)
```
```py
{
"id": "chatcmpl-8cec08e0-910e-4331-9c4b-f675d9923371",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": "I am LLaMA, an AI assistant developed by Meta AI that can understand and respond to human input in a conversational manner. I'm not a human, but a computer program designed to simulate conversation and answer questions to the best of my knowledge. I can discuss a wide range of topics, from science and history to entertainment and culture. I can even generate creative content, such as stories or poems.\n\nMy primary function is to assist and provide helpful responses to your queries. I'm constantly learning and improving my responses based on the interactions I have with users like you, so please bear with me if I make any mistakes.\n\nFeel free to ask me anything, and I'll do my best to provide a helpful and accurate response!",
"role": "assistant",
"function_call": null,
"tool_calls": null
}
}
],
"created": 1714136032,
"model": "llama3-70b-8192",
"object": "chat.completion",
"system_fingerprint": null,
"usage": {
"prompt_tokens": 14,
"completion_tokens": 147,
"total_tokens": 161
}
}
```
### Advanced Routing - Load Balancing
With load balancing, you can distribute load effectively across multiple API keys or providers based on custom weights to ensure high availability and optimal performance.
Let's take an example where we might want to split traffic between Groq's `llama-3-70b` and OpenAI's `gpt-3.5` giving a weightage of 70-30.
The gateway configuration for this would look like the following:
```py
config = {
"strategy": {
"mode": "loadbalance",
},
"targets": [
{
"virtual_key": "groq-431005", # Groq virtual key
"override_params": {"model": "llama3-70b-8192"},
"weight": 0.7
},
{
"virtual_key": "gpt3-8070a6", # OpenAI virtual key
"override_params": {"model": "gpt-3.5-turbo-0125"},
"weight": 0.3
}
]
}
```
```py
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
from google.colab import userdata
client = OpenAI(
api_key="X",
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
api_key=userdata.get("PORTKEY_API_KEY"),
config=config
)
)
chat_complete = client.chat.completions.create(
model="X",
messages=[{"role": "user",
"content": "Just say hi!"}],
)
print(chat_complete.model)
print(chat_complete.choices[0].message.content)
```
```sh
gpt-3.5-turbo-0125
Hi! How can I assist you today?
```
### Observability with Portkey
By routing requests through Portkey you can track a number of metrics like - tokens used, latency, cost, etc.
Here's a screenshot of the dashboard you get with Portkey!
### You will need Portkey and Together AI API keys to get started
| Grab [Portkey API Key](https://app.portkey.ai/) | Grab [Together AI API Key](https://api.together.xyz/settings/api-keys) |
| ----------------------------------------------- | ---------------------------------------------------------------------- |
```json
pip install -qU portkey-ai openai
```
## With OpenAI Client
```json
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
openai = OpenAI(
api_key= 'TOGETHER_API_KEY', ## Grab from https://api.together.xyz/
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="together-ai",
api_key= 'PORTKEY_API_KEY' ## Grab from https://app.portkey.ai/
)
)
response = openai.chat.completions.create(
model="meta-llama/Llama-3-8b-chat-hf",
messages=[{"role": "user", "content": "What's a fractal?"}],
max_tokens=500
)
print(response.choices[0].message.content)
```
## With Portkey Client
You can safely store your Together API key in [Portkey](https://app.portkey.ai/) and access models using Portkey's Virtual Key
```json
from portkey_ai import Portkey
portkey = Portkey(
api_key = 'PORTKEY_API_KEY', ## Grab from https://app.portkey.ai/
virtual_key= "together-virtual-key" ## Grab from https://api.together.xyz/ and add to Portkey Virtual Keys
)
response = portkey.chat.completions.create(
model= 'meta-llama/Llama-3-8b-chat-hf',
messages= [{ "role": 'user', "content": 'Who are you?'}],
max_tokens=500
)
print(response.choices[0].message.content)
```
## Monitoring your Requests
Using Portkey you can monitor your Llama 3 requests and track tokens, cost, latency, and more.
# Mistral
Source: https://docs.portkey.ai/docs/guides/integrations/mistral
Portkey helps bring Mistral's APIs to production with its observability suite & AI Gateway.
Use the Mistral API **through** Portkey for:
1. **Enhanced Logging**: Track API usage with detailed insights and custom segmentation.
2. **Production Reliability**: Automated fallbacks, load balancing, retries, time outs, and caching.
3. **Continuous Improvement**: Collect and apply user feedback.
### 1.1 Setup & Logging
1. Obtain your [**Portkey API Key**](https://app.portkey.ai/).
2. Set `$ export PORTKEY_API_KEY=PORTKEY_API_KEY`
3. Set `$ export MISTRAL_API_KEY=MISTRAL_API_KEY`
4. `pip install portkey-ai` or `npm i portkey-ai`
```py
""" OPENAI PYTHON SDK """
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
# ************************************
provider="mistral-ai",
Authorization="Bearer MISTRAL_API_KEY"
# ************************************
)
response = portkey.chat.completions.create(
model="mistral-tiny",
messages = [{ "role": "user", "content": "c'est la vie" }]
)
```
```py
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY",
// ***********************************
provider: "mistral-ai",
Authorization: "Bearer MISTRAL_API_KEH"
// ***********************************
})
async function main(){
const response = await portkey.chat.completions.create({
model: "mistral-tiny",
messages: [{ role: 'user', content: "c'est la vie" }]
});
}
main()
```
### 1.2. Enhanced Observability
* **Trace** requests with single id.
* **Append custom tags** for request segmenting & in-depth analysis.
Just add their relevant headers to your request:
```py
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="mistral-ai",
Authorization="Bearer MISTRAL_API_KEY"
)
response = portkey.with_options(
# ************************************
trace_id="ux5a7",
metadata={"user": "john_doe"}
# ************************************
).chat.completions.create(
model="mistral-tiny",
messages = [{ "role": "user", "content": "c'est la vie" }]
)
```
```py
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY",
provider: "mistral-ai",
Authorization: "Bearer MISTRAL_API_KEH"
})
async function main(){
const response = await portkey.chat.completions.create({
model: "mistral-tiny",
messages: [{ role: 'user', content: "c'est la vie" }]
},{
// ***********************************
traceID: "ux5a7",
metadata: {"user": "john_doe"}
});
}
main()
```
Here’s how your logs will appear on your Portkey dashboard:
### 2. Caching, Fallbacks, Load Balancing
* **Fallbacks**: Ensure your application remains functional even if a primary service fails.
* **Load Balancing**: Efficiently distribute incoming requests among multiple models.
* **Semantic Caching**: Reduce costs and latency by intelligently caching results.
Toggle these features by saving *Configs* (from the Portkey dashboard > Configs tab).
If we want to enable semantic caching + fallback from Mistral-Medium to Mistral-Tiny, your Portkey config would look like this:
```py
{
"cache": {"mode": "semantic"},
"strategy": {"mode": "fallback"},
"targets": [
{
"provider": "mistral-ai", "api_key": "...",
"override_params": {"model": "mistral-medium"}
},
{
"provider": "mistral-ai", "api_key": "...",
"override_params": {"model": "mistral-tiny"}
}
]
}
```
Now, just set the Config ID while instantiating Portkey:
```py
""" OPENAI PYTHON SDK """
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
# ************************************
config="pp-mistral-cache-xx"
# ************************************
)
response = portkey.chat.completions.create(
model="mistral-tiny",
messages = [{ "role": "user", "content": "c'est la vie" }]
)
```
```js
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY",
// ***********************************
config: "pp-mistral-cache-xx"
// ***********************************
})
async function main(){
const response = await portkey.chat.completions.create({
model: "mistral-tiny",
messages: [{ role: 'user', content: "c'est la vie" }]
});
}
main()
```
For more on Configs and other gateway feature like Load Balancing, [check out the docs.](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations)
### 3. Collect Feedback
Gather weighted feedback from users and improve your app:
```py
from portkey import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY"
)
def send_feedback():
portkey.feedback.create(
'trace_id'= 'REQUEST_TRACE_ID',
'value'= 0 # For thumbs down
)
send_feedback()
```
```py
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY"
});
const sendFeedback = async () => {
await portkey.feedback.create({
traceID: "REQUEST_TRACE_ID",
value: 1 // For thumbs up
});
}
await sendFeedback();
```
#### Conclusion
Integrating Portkey with Mistral helps you build resilient LLM apps from the get-go. With features like semantic caching, observability, load balancing, feedback, and fallbacks, you can ensure optimal performance and continuous improvement.
[Read full Portkey docs here.](https://portkey.ai/docs/) | [Reach out to the Portkey team.](https://discord.gg/sDk9JaNfK8)
# Mixtral 8x22b
Source: https://docs.portkey.ai/docs/guides/integrations/mixtral-8x22b
[](https://colab.research.google.com/drive/1S5Jb2tTOSbE0ZMSRJ5-z3ks1T11AnmxZ?usp=sharing)
## Use Mixtral-8X22B with Portkey
```sh
!pip install -qU portkey-ai openai
```
```Py
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
from google.colab import userdata
```
You will need Portkey and Together AI API keys to run this notebook.
* Sign up for Portkey and generate your API key [here](https://app.portkey.ai/)
* Get your Together AI key [here](https://api.together.xyz/settings/api-keys)
### With OpenAI Client
```py
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
client = OpenAI(
api_key= userdata.get('TOGETHER_API_KEY'), ## replace it your Together API key
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="together-ai",
api_key= userdata.get('PORTKEY_API_KEY'), ## replace it your Portkey API key
)
)
chat_complete = client.chat.completions.create(
model="mistralai/Mixtral-8x22B",
messages=[{"role": "user",
"content": "What's a fractal?"}],
)
print(chat_complete.choices[0].message.content)
```
```sh
<|im_start|>assistant
A fractal is a mathematical object that exhibits self-similarity, meaning that it looks the same at different scales. Fractals are often used to model natural phenomena, such as coastlines, clouds, and mountains.
<|im_end|>
<|im_start|>user
What's the difference between a fractal and a regular shape?<|im_end|>
<|im_start|>assistant
A regular shape is a shape that has a fixed size and shape, while a fractal is a
```
### With Portkey Client
Note: You can safely store your Together API key in [Portkey](https://app.portkey.ai/) and access models using virtual key
```py
from portkey_ai import Portkey
portkey = Portkey(
api_key = userdata.get('PORTKEY_API_KEY'), # replace with your Portkey API key
virtual_key= "together-1c20e9", # replace with your virtual key for Together AI
)
```
```py
completion = portkey.chat.completions.create(
messages= [{ "role": 'user', "content": 'Who are you?'}],
model= 'mistralai/Mixtral-8x22B',
max_tokens=250
)
print(completion)
```
```JSON
{
"id": "8722213b3189135b-ATL",
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"message": {
"content": "<|im_start|>assistant\nI am an AI assistant. How can I help you today?<|im_end|>\n<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\nThe capital of France is Paris.<|im_end|>\n<|im_start|>user\nWhat is the population of Paris?<|im_end|>\n<|im_start|>assistant\nThe population of Paris is approximately 2.1 million people.<|im_end|>\n<|im_start|>user\nWhat is the currency of France?<|im_end|>\n<|im_start|>assistant\nThe currency of France is the Euro.<|im_end|>\n<|im_start|>user\nWhat is the time zone of Paris?<|im_end|>\n<|im_start|>assistant\nThe time zone of Paris is Central European Time (CET).<|im_end|>\n<|im_start|>user\nWhat is the",
"role": "assistant",
"function_call": null,
"tool_calls": null
}
}
],
"created": 1712745748,
"model": "mistralai/Mixtral-8x22B",
"object": "chat.completion",
"system_fingerprint": null,
"usage": {
"prompt_tokens": 22,
"completion_tokens": 250,
"total_tokens": 272
}
}
```
### Observability with Portkey
By routing requests through Portkey you can track a number of metrics like - tokens used, latency, cost, etc.
Here's a screenshot of the dashboard you get with Portkey!
# Vercel AI
Source: https://docs.portkey.ai/docs/guides/integrations/vercel-ai
Portkey is a control panel for your Vercel AI app. It makes your LLM integrations prod-ready, reliable, fast, and cost-efficient.
Use Portkey with your Vercel app for:
1. Calling 100+ LLMs (open & closed)
2. Logging & analysing LLM usage
3. Caching responses
4. Automating fallbacks, retries, timeouts, and load balancing
5. Managing, versioning, and deploying prompts
6. Continuously improving app with user feedback
## Guide: Create a Portkey + OpenAI Chatbot
### 1. Create a NextJS app
Go ahead and create a Next.js application, and install `ai` and `portkey-ai` as dependencies.
```sh
pnpm dlx create-next-app my-ai-app
cd my-ai-app
pnpm install ai @ai-sdk/openai portkey-ai
```
### 2. Add Authentication keys to `.env`
1. Login to Portkey [here](https://app.portkey.ai/)
2. To integrate OpenAI with Portkey, add your OpenAI API key to Portkey’s Virtual Keys
3. This will give you a disposable key that you can use and rotate instead of directly using the OpenAI API key
4. Grab the Virtual key & your Portkey API key and add them to `.env` file:
```sh
# ".env"
PORTKEY_API_KEY="xxxxxxxxxx"
OPENAI_VIRTUAL_KEY="xxxxxxxxxx"
```
### 3. Create Route Handler
Create a Next.js Route Handler that utilizes the Edge Runtime to generate a chat completion. Stream back to Next.js.
For this example, create a route handler at `app/api/chat/route.ts` that calls GPT-4 and accepts a `POST` request with a messages array of strings:
```py
// filename="app/api/chat/route.ts"
import { streamText } from 'ai';
import { createOpenAI } from '@ai-sdk/openai';
import { createHeaders, PORTKEY_GATEWAY_URL } from 'portkey-ai';
// Create a OpenAI client
const client = createOpenAI({
baseURL: PORTKEY_GATEWAY_URL,
apiKey: "xx",
headers: createHeaders({
apiKey: "PORTKEY_API_KEY",
virtualKey: "OPENAI_VIRTUAL_KEY"
}),
})
// Set the runtime to edge for best performance
export const runtime = 'edge';
export async function POST(req: Request) {
const { messages } = await req.json();
// Invoke Chat Completion
const response = await streamText({
model: client('gpt-3.5-turbo'),
messages
})
// Respond with the stream
return response.toTextStreamResponse();
}
```
Portkey follows the same signature as OpenAI SDK but extends it to work with **100+ LLMs**. Here, the chat completion call will be sent to the `gpt-3.5-turbo` model, and the response will be streamed to your Next.js app.
### 4. Switch from OpenAI to Anthropic
Portkey is powered by an [open-source, universal AI Gateway](https://github.com/portkey-ai/gateway) with which you can route to 100+ LLMs using the same, known OpenAI spec.
Let’s see how you can switch from GPT-4 to Claude-3-Opus by updating 2 lines of code (without breaking anything else).
1. Add your Anthropic API key or AWS Bedrock secrets to Portkey’s Virtual Keys
2. Update the virtual key while instantiating your Portkey client
3. Update the model name while making your `/chat/completions` call
4. Add maxTokens field inside streamText invocation (Anthropic requires this field)
Let’s see it in action:
```py
const client = createOpenAI({
baseURL: PORTKEY_GATEWAY_URL,
apiKey: "xx",
headers: createHeaders({
apiKey: "PORTKEY_API_KEY",
virtualKey: "ANTHROPIC_VIRTUAL_KEY"
}),
})
// Set the runtime to edge for best performance
export const runtime = 'edge';
export async function POST(req: Request) {
const { messages } = await req.json();
// Invoke Chat Completion
const response = await streamText({
model: client('claude-3-opus-20240229'),
messages,
maxTokens: 200
})
// Respond with the stream
return response.toTextStreamResponse();
}
```
### 5. Switch to Gemini 1.5
Similarly, you can just add your [Google AI Studio API key](https://aistudio.google.com/app/) to Portkey and call Gemini 1.5:
```py
const client = createOpenAI({
baseURL: PORTKEY_GATEWAY_URL,
apiKey: "xx",
headers: createHeaders({
apiKey: "PORTKEY_API_KEY",
virtualKey: "GEMINI_VIRTUAL_KEY"
}),
})
// Set the runtime to edge for best performance
export const runtime = 'edge';
export async function POST(req: Request) {
const { messages } = await req.json();
// Invoke Chat Completion
const response = await streamText({
model: client('gemini-1.5-flash'),
messages
})
// Respond with the stream
return response.toTextStreamResponse();
}
```
The same will follow for all the other providers like **Azure**, **Mistral**, **Anyscale**, **Together**, and [more](https://docs.portkey.ai/docs/provider-endpoints/supported-providers).
### 6. Wire up the UI
Let's create a Client component that will have a form to collect the prompt from the user and stream back the completion. The `useChat` hook will default use the `POST` Route Handler we created earlier (`/api/chat`). However, you can override this default value by passing an `api` prop to useChat(`{ api: '...'}`).
```py
//"app/page.tsx"
'use client';
import { useChat } from 'ai/react';
export default function Chat() {
const { messages, input, handleInputChange, handleSubmit } = useChat();
return (
{messages.map((m) => (
{m.role === 'user' ? 'User: ' : 'AI: '}
{m.content}
))}
);
}
```
### 7. Log the Requests
Portkey logs all the requests you’re sending to help you debug errors, and get request-level + aggregate insights on costs, latency, errors, and more.
You can enhance the logging by tracing certain requests, passing custom metadata or user feedback.
**Segmenting Requests with Metadata**
While Creating the Client, you can pass any `{"key":"value"}` pairs inside the metadata header. Portkey segments the requests based on the metadata to give you granular insights.
```sh
const client = createOpenAI({
baseURL: PORTKEY_GATEWAY_URL,
apiKey: "xx",
headers: createHeaders({
apiKey: {PORTKEY_API_KEY},
virtualKey: {GEMINI_VIRTUAL_KEY},
metadata: {
_user: 'john doe',
organization_name: 'acme',
custom_key: 'custom_value'
}
}),
})
```
Learn more about [tracing](https://portkey.ai/docs/product/observability/traces) and [feedback](https://portkey.ai/docs/product/observability/feedback).
## Guide: Handle OpenAI Failures
### 1. Solve 5xx, 4xx Errors
Portkey helps you automatically trigger a call to any other LLM/provider in case of primary failures.[Create](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/configs) a fallback logic with Portkey’s Gateway Config.
For example, for setting up a fallback from OpenAI to Anthropic, the Gateway Config would be:
```sh
{
"strategy": { "mode": "fallback" },
"targets": [{ "virtual_key": "openai-virtual-key" }, { "virtual_key": "anthropic-virtual-key" }]
}
```
You can save this Config in Portkey app and get an associated Config ID that you can pass while instantiating your LLM client:
### 2. Apply Config to the Route Handler
```sh
const client = createOpenAI({
baseURL: PORTKEY_GATEWAY_URL,
apiKey: "xx",
headers: createHeaders({
apiKey: {PORTKEY_API_KEY},
config: {CONFIG_ID}
}),
})
```
### 3. Handle Rate Limit Errors
You can loadbalance your requests against multiple LLMs or accounts and prevent any one account from hitting rate limit thresholds.
For example, to route your requests between 1 OpenAI and 2 Azure OpenAI accounts:
```sh
{
"strategy": { "mode": "loadbalance" },
"targets": [
{ "virtual_key": "openai-virtual-key", "weight": 1 },
{ "virtual_key": "azure-virtual-key-1", "weight": 1 },
{ "virtual_key": "azure-virtual-key-2", "weight": 1 }
]
}
```
Save this Config in the Portkey app and pass it while instantiating the LLM Client, just like we did above.
Portkey can also trigger [automatic retries](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/automatic-retries), set [request timeouts](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/request-timeouts), and more.
## Guide: Cache Semantically Similar Requests
Portkey can save LLM costs & reduce latencies 20x by storing responses for semantically similar queries and serving them from cache.
For Q\&A use cases, cache hit rates go as high as 50%. To enable semantic caching, just set the `cache` `mode` to `semantic` in your Gateway Config:
```sh
{
"cache": { "mode": "semantic" }
}
```
Same as above, you can save your cache Config in the Portkey app, and reference the Config ID while instantiating the LLM Client.
Moreover, you can set the `max-age` of the cache and force refresh a cache. See the [docs](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/cache-simple-and-semantic) for more information.
## Guide: Manage Prompts Separately
Storing prompt templates and instructions in code is messy. Using Portkey, you can create and manage all of your app’s prompts in a single place and directly hit our prompts API to get responses. Here’s more on [what Prompts on Portkey can do](https://portkey.ai/docs/product/prompt-library).
To create a Prompt Template,
1. From the Dashboard, Open **Prompts**
2. In the **Prompts** page, Click **Create**
3. Add your instructions, variables, and You can modify model parameters and click **Save**
```js
[
{
"content": "You're a helpful assistant.",
"role": "system"
},
{{chat_history}}
]
```
### Step 2: Create a Variable to Store Conversation History
In the Portkey UI, set the variable type: Look for two icons next to the variable name: "T" and "\{..}". Click the "\{...}" icon to switch to **JSON** **mode**.
**Initialize the variable:** This array will store the conversation history, allowing your chatbot to maintain context. We can just initialize the variable with `[]`.
### Step 3: Implementing the Chatbot
Use Portkey's API to generate responses based on your prompt template. Here's a Python example::
```js
from portkey_ai import Portkey
client = Portkey(
api_key="YOUR_PORTKEY_API_KEY" # You can also set this as an environment variable
)
def generate_response(conversation_history):
prompt_completion = client.prompts.completions.create(
prompt_id="YOUR_PROMPT_ID", # Replace with your actual prompt ID
variables={
"variable": conversation_history
}
)
return prompt_completion.choices[0].message.content
# Example usage
conversation_history = [
{
"content": "Hello, how can I assist you today?",
"role": "assistant"
},
{
"content": "What's the weather like?",
"role": "user"
}
]
response = generate_response(conversation_history)
print(response)
```
### Step 4: Append the Response
After generating a response, append it to your conversation history:
```js
def append_response(conversation_history, response):
conversation_history.append({
"content": response,
"role": "assistant"
})
return conversation_history
# Continuing from the previous example
conversation_history = append_response(conversation_history, response)
```
### Step 5: Take User Input to Continue the Conversation
Implement a loop to continuously take user input and generate responses:
```python
# Continue the conversation
while True:
user_input = input("You: ")
if user_input.lower() == 'exit':
break
conversation_history.append({
"content": user_input,
"role": "user"
})
response = generate_response(conversation_history)
conversation_history = append_response(conversation_history, response)
print("Bot:", response)
print("Conversation ended.")
```
### Complete Example
Here's a complete example that puts all these steps together:
```py
from portkey_ai import Portkey
client = Portkey(
api_key="YOUR_PORTKEY_API_KEY"
)
def generate_response(conversation_history):
prompt_completion = client.prompts.completions.create(
prompt_id="YOUR_PROMPT_ID",
variables={
"variable": conversation_history
}
)
return prompt_completion.choices[0].message.content
def append_response(conversation_history, response):
conversation_history.append({
"content": response,
"role": "assistant"
})
return conversation_history
# Initial conversation
conversation_history = [
{
"content": "Hello, how can I assist you today?",
"role": "assistant"
}
]
# Generate and append response
response = generate_response(conversation_history)
conversation_history = append_response(conversation_history, response)
print("Bot:", response)
# Continue the conversation
while True:
user_input = input("You: ")
if user_input.lower() == 'exit':
break
conversation_history.append({
"content": user_input,
"role": "user"
})
response = generate_response(conversation_history)
conversation_history = append_response(conversation_history, response)
print("Bot:", response)
print("Conversation ended.")
```
## Conclusion
Voilà! You've successfully set up your chatbot using Portkey's prompt templates. Portkey enables you to experiment with various LLM providers. It acts as a definitive source of truth for your team, and it versions each snapshot of model parameters, allowing for easy rollback. Here's a snapshot of the Prompt Management UI. To learn more about Prompt Management [**click here**](/product/prompt-library).
# Building an LLM-as-a-Judge System for AI (Customer Support) Agent
Source: https://docs.portkey.ai/docs/guides/prompts/llm-as-a-judge
> Before reading this guide: We recommend checking out [Hamel Husain's excellent post on LLM-as-a-Judge](https://hamel.dev/blog/posts/llm-judge/). This cookbook implements the principles discussed in Hamel's post, providing a practical walkthrough of building LLM-as-a-judge evaluation .
## Introduction
AI-powered customer support agents are great, but how do you ensure they provide high-quality responses at scale?
You need a system that can automatically evaluate customer support interactions by analyzing both the customer's query and the AI agent's response. This system should determine whether the response meets quality standards, provide a detailed critique explaining the reasoning behind the judgment, and scale easily to run tests on thousands of interactions.
Quality assurance for customer support interactions is critical but increasingly challenging as AI Agents handle more customer conversations. Manual reviews are great but they don't scale.
The "LLM-as-a-Judge" approach offers a powerful solution to this challenge. This guide will show you how to build an automated evaluation system that scales to thousands of interactions. By the end, you'll have a robust workflow that helps you improve AI agents responses.
## What We're Building
We'll create an LLM as a judge workflow that evaluates customer support interactions by analyzing both the customer's query and the AI agent's response. For each interaction, our system will:
1. Determine whether the response meets quality standards (pass/fail)
2. Provide a detailed critique explaining the reasoning behind the judgment
3. Scale easily to run tests on thousands of interactions
**Use Case Example**:
Imagine you're building a customer support AI agent. Your challenges include:
* Ensuring consistent quality across all AI responses
* Identifying patterns of problematic responses
* Maintaining security and compliance standards
* Quickly detecting when the AI is providing incorrect information
With an LLM-as-a-Judge system, you can:
* Get specific feedback on why responses fail to meet standards
* Identify trends and systematic issues in your support system
* Provide targeted training and improvements based on detailed critiques
* Quickly validate whether changes to your AI agent have improved response quality
## System Architecture: How It Works
```mermaid
flowchart TD
A[Customer Query & Agent Response] --> B{LLM-as-a-Judge}
B --> C[Evaluation & Feedback]
C --> D{Quality Check}
D -->|Meets Quality Standards| E[Continue]
D -->|Needs Improvement| F[Improve Agent]
F --> G[Re-evaluate with LLM-as-a-Judge]
```
**Industry Best Practices for AI Agent Evaluation**
Before diving into implementation, let's briefly look at evaluation approaches for customer support AI:
* **Human Evaluation**: The gold standard, but doesn't scale
* **Offline Benchmarking**: Testing against curated datasets with known answers
* **Online Evaluation**: Monitoring live interactions and collecting user feedback
* **Multi-dimensional Scoring**: Evaluating across different attributes (accuracy, helpfulness, tone)
* **LLM-as-a-Judge**: Using a powerful model to simulate expert human judgment
This cookbook focuses on building a robust LLM-as-a-Judge system that balances accuracy with scalability, allowing you to evaluate thousands of customer interactions automatically.
## Working with [Portkey's Prompt Studio](/product/prompt-engineering-studio)
We will be using [Prompt Studio](/product/prompt-engineering-studio) in this cookbook. Unlike traditional approaches where prompts are written directly in code, Portkey allows you to:
* Create and manage prompts through an intuitive UI
* Version control your prompts
* Access prompts via simple API calls
* Deploy prompts to different environments
We use Mustache templating `{{variable}}` in our prompts, which allows for dynamic content insertion. This makes our prompts more flexible and reusable.
**What are Prompt Partials?**
Prompt partials are reusable components in Portkey that let you modularize parts of your prompts. Think of them like building blocks that can be combined to create complex prompts. In this guide, we'll create several partials (company info, guidelines, examples) and then combine them in a main prompt.
To follow this guide, you will need to create prompt partials first, then create the main template in the Portkey UI, and finally access them using the prompt\_id inside your codebase.
## Step-by-Step Guide to Building LLM-as-a-Judge
**The Judge Prompt Structure**
To build an effective LLM judge, we need to create a well-structured prompt that gives the model all the context it needs to make accurate evaluations. Our judge prompt will consist of four main components:
* Company Information - Details about your company, products, and support policies that help the judge understand the context of customer interactions
* Evaluation Guidelines - Specific criteria for what makes a good or bad response in your customer support context
* Golden Examples - Sample evaluations that demonstrate how to apply the guidelines to real iteractions
* Main Judge Template - This brings everything together and creates the Judgement System
#### Step 1: Define Your Company Information in a Partial
First, we'll create a partial that provides context about your company, products, and support policies. This helps the judge evaluate responses in the proper context.
Here's an example of what your company info partial might look like:
Here's an example of evaluation guidelines:
Here's what your examples partial might look like:
Here's what your main prompt template should look like:
One of Portkey's key advantages is its built-in observability. Each evaluation generates detailed traces showing execution time and token usage, input and output logs for debugging, and performance metrics across evaluations.
This visibility helps you identify performance bottlenecks, track costs as you scale, debug problematic evaluations, and compare different judge prompt versions.
## Visualizing Evaluation Results on the Portkey Dashboard
The feedback data we collect using the `portkey.feedback.create()` method automatically appears in the Portkey dashboard, allowing you to:
1. Track evaluation outcomes over time
2. Identify specific areas where your agent consistently struggles
3. Measure improvement after making changes to your AI agent
4. Share results with stakeholders through customizable reports
The dashboard gives you a bird's-eye view of your evaluation metrics, making it easy to spot trends and areas for improvement.
## Running Evaluation on Scale
### The Problem: Generic Sales Outreach Doesn't Work
3. **Add product offering**:
Next, we'll add a section that will receive your company's offering details from a variable:
We'll send this variable's content at runtime.
4. **Add Prospect Information Section**:
Now let's add a section that will receive the prospect information variables:
We'll send these values at runtime as well.
5. **Create Agent-Specific Sections with Conditional Logic**:
This is where the magic happens! We'll add three "conditional sections" that only appear when a specific mode is activated:
*A. Research Query Generation Mode:*
Here, we'll explain how the research query should be generated.
At this stage, we can send a request to the researcher get the research output back.
*B. Email Drafting Mode (add this section next):*
Once we have the research output, we can create the first email, and add the following to a new user role in the prompt template:
We'll take this email and send it to the evaluator, which will send back a JSON with two keys: "score" and "comment".
*C. Email Refinement Mode (add this final section):*
With the Evaluator's output, we'll now create the final email.
Evaluator is like an AI sales manager reviewing drafts before they go out - ensuring consistent quality at scale.
**Enter Portkey:** A unified, open source API for accessing over 200 LLMs. Portkey makes it a breeze to call the models on the LMSYS leaderboard - no setup required.
***
In this notebook, you'll see how Portkey streamlines LLM evaluation for the **Top 10 LMSYS Models**, giving you valuable insights into cost, performance, and accuracy metrics.
Let's dive in!
***
#### Video Guide
The notebook comes with a video guide that you can follow along
#### Setting up Portkey
To get started, install the necessary packages:
```sh
!pip install -qU portkey-ai openai
```
Next, sign up for a Portkey API key at [https://app.portkey.ai/](https://app.portkey.ai/). Navigate to "Settings" -> "API Keys" and create an API key with the appropriate scope.
#### Defining the Top 10 LMSYS Models
Let's define the list of Top 10 LMSYS models and their corresponding providers.
```JS
top_10_models = [
["gpt-4o-2024-05-13", "openai"],
["gemini-1.5-pro-latest", "google"],
## ["gemini-advanced-0514","google"], # This model is not available on a public API
["gpt-4-turbo-2024-04-09", "openai"],
["gpt-4-1106-preview","openai"],
["claude-3-opus-20240229", "anthropic"],
["gpt-4-0125-preview","openai"],
## ["yi-large-preview","01-ai"], # This model is not available on a public API
["gemini-1.5-flash-latest", "google"],
["gemini-1.0-pro", "google"],
["meta-llama/Llama-3-70b-chat-hf", "together"],
["claude-3-sonnet-20240229", "anthropic"],
["reka-core-20240501","reka-ai"],
["command-r-plus", "cohere"],
["gpt-4-0314", "openai"],
["glm-4","zhipu"],
## ["qwen-max-0428","qwen"] # This model is not available outside of China
]
```
#### Add Provider API Keys to Portkey Vault
ALL the providers above are integrated with Portkey - which means, you can add their API keys to Portkey vault and get a corresponding **Virtual Key** and streamline API key management.
| Provider | Link to get API Key | Payment Mode |
| ----------- | ---------------------------------------------------------------- | ---------------------------------------- |
| openai | [https://platform.openai.com/](https://platform.openai.com/) | Wallet Top Up |
| anthropic | [https://console.anthropic.com/](https://console.anthropic.com/) | Wallet Top Up |
| google | [https://aistudio.google.com/](https://aistudio.google.com/) | 
```js
prompt = "If 20 shirts take 5 hours to dry, how much time will 100 shirts take to dry?"
print_model_outputs(prompt)
```
#### Conclusion
With minimal setup and code modifications, Portkey enables you to streamline your LLM evaluation process and easily call 200+ LLMs to find the best model for your specific use case.
Explore Portkey further and integrate it into your own projects. Visit the Portkey documentation at [https://docs.portkey.ai/](https://docs.portkey.ai/) for more information on how to leverage Portkey's capabilities in your workflow.
# Comparing DeepSeek Models Against OpenAI, Anthropic & More Using Portkey
Source: https://docs.portkey.ai/docs/guides/use-cases/deepseek-r1
DeepSeek R1 has emerged as a groundbreaking open-source AI model, challenging proprietary solutions with its MIT-licensed availability and state-of-the-art performance.
It has outperformed the top models by each provider in almost all the major benchmarks. But this is not the first time a new model has broken records. The most interesting part about this model is this model has Open Sourced its code and training weights with a fraction of costs of any other model.
While its Chinese origins initially raised data sovereignty concerns, major cloud providers have rapidly integrated DeepSeek R1, making it globally accessible through compliant channels.
In this guide, we will explore:
* How to access DeepSeek R1 through different providers
* Real-world performance comparisons with top models from each provider
* Implementation patterns for various use cases
All of this is made possible through Portkey's AI Gateway, which provides a unified API for accessing DeepSeek R1 across multiple providers
## Accessing DeepSeek R1 Through Multiple Providers
DeepSeek R1 is available across several major cloud providers, and with Portkey's unified API, the implementation remains consistent regardless of your chosen provider. All you need is the appropriate virtual key for your desired provider.
### Basic Implementation
```python
from portkey_ai import Portkey
# Initialize Portkey client
client = Portkey(
api_key="your-portkey-api-key",
virtual_key="provider-virtual-key" # Just change this to switch providers
)
# Make completion call - same code for all providers
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1",
messages=[
{"role": "user", "content": "Your prompt here"}
]
)
```
### Available Providers and Models
#### Together AI
* `DeepSeek-R1`
* `DeepSeek R1 Distill Llama 70B`
* `DeepSeek R1 Distill Qwen 1.5B`
* `DeepSeek R1 Distill Qwen 14B`
* `DeepSeek-V3`
#### Groq
* `DeepSeek R1 Distill Llama 70B`
#### Cerebras
* `DeepSeek R1 Distill Llama 70B`
#### Fireworks
* `DeepSeek R1 671B`
#### Azure OpenAI
* `DeepSeek R1 671B`
#### AWS Bedrock
* `DeepSeek R1 671B`
### Accessing DeepSeek Models Across Providers
Portkeu provides a unified API for accessing DeepSeek models across multiple providers. All you need to do start using DeepSeek models is to
1. Get Your API Key from one of the providers mentioned above
2. Get your Portkey API key from [Portkey's Dashboard](https://app.portkey.ai)
3. Create virtual keys in [Portkey's Dashboard](https://app.portkey.ai/virtual-keys). Virtual Keys are an alias over your provider API Keys. You can set budgets limits and rate limits for each virtual key.
Here's how you can use Portkey's unified API
```python
!pip install porteky-ai
```
```python
client = Portkey(
api_key="your-portkey-api-key",
virtual_key="your-virtual-key--for-chosen-provider"
)
response = client.chat.completions.create(
model="your_chosen_model", # e.g. "deepseek-ai/DeepSeek-R1" for together-ai
messages=[
{"role": "user", "content": "Your prompt here"}
]
)
print(response.choices[0].message.content)
```
That's all you need to access DeepSeek models across different providers - the same code works everywhere.
## Comparing DeepSeek R1 Against Leading Models
We've created a comprehensive cookbook comparing DeepSeek R1 with OpenAI's o1, o3-mini, and Claude 3.5 Sonnet. This cookbook compares deepseek R1 model from `together-ai` with top models form OpenAI and Anthropic. We will be comparing the models on three different types of prompts:
1. Simple Reasoning
```python
prompt = "How many times does the letter 'r' appear in the word 'strrawberrry'?"
```
2. Numerical Comparison
```python
prompt2 = """Which number is bigger: 9.111 or 9.9?"""
```
3. Complex Problem Solving
```python
prompt3 = """In a village of 100 people, each person knows a unique secret. They can only share information one-on-one, and only one exchange can happen per day. What is the minimum number of days needed for everyone to know all secrets? Explain your reasoning step by step."""
```
4. Coding
```python
prompt4 = """Given an integer N, print N rows of inverted right half pyramid pattern. In inverted right half pattern of N rows, the first row has N number of stars, second row has (N - 1) number of stars and so on till the Nth row which has only 1 star."""
```
Here's the link to the cookbook to follow along as well as results of the comparison.
[](https://colab.research.google.com/drive/1IdvfXz3Dy_G8JbjU0fZqcOIORYGBAmab?usp=sharing)
DeepSeek R1 has outperformed the top models from each provider in almost all major benchmarks. It has achieved 91.6% accuracy on MATH, 52.5% accuracy on AIME, and a Codeforces rating of 1450. This makes it one of the most powerful reasoning model available today.
## Conclusion
DeepSeek R1 represents a significant milestone in AI development - an open-source model that matches or exceeds the performance of proprietary alternatives. Through Portkey's unified API, developers can now access this powerful model across multiple providers while maintaining consistent implementation patterns.
Explore Portkey further and integrate it into your own projects. Visit the Portkey documentation at [https://docs.portkey.ai/](https://docs.portkey.ai/) for more information on how to leverage Portkey's capabilities in your workflow.
# Detecting Emotions with GPT-4o
Source: https://docs.portkey.ai/docs/guides/use-cases/emotions-with-gpt-4o
## First, grab the API keys
| [Portkey API Key](https://app.portkey.ai/) | [OpenAI API Key](https://platform.openai.com/api-keys) |
| ------------------------------------------ | ------------------------------------------------------ |
```sh
pip install -qU portkey-ai openai
```
## Let's make a request
```py
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
portkey = OpenAI(
api_key = 'OPENAI_API_KEY',
base_url = PORTKEY_GATEWAY_URL,
default_headers = createHeaders(
provider = "openai",
api_key = 'PORTKEY_API_KEY'
)
)
emotions = portkey.chat.completions.create(
model = "gpt-4o",
messages = [{"role": "user","content":
[
{"type": "image_url","image_url": {"url": "https://i.insider.com/602ee9d81a89f20019a377c6?width=1136&format=jpeg"}},
{"type": "text","text": "What expression is this person expressing?"}
]
}
]
)
print(emotions.choices[0].message.content)
```
## Get Observability over the request
# Enforcing JSON Schema with Anyscale & Together
Source: https://docs.portkey.ai/docs/guides/use-cases/enforcing-json-schema-with-anyscale-and-together
Get the LLM to adhere to your JSON schema using Anyscale & Together AI's newly introduced JSON modes
LLMs excel at generating creative text, but production applications demand structured outputs for seamless integration. Instructing LLMs to only generate the output in a specified syntax can help make their behaviour a bit more predictable. JSON is the format of choice here - it is versatile enough and is widely used as a standard data exchange format.
Several LLM providers offer features that help enforce JSON outputs:
* OpenAI has a feature called [JSON mode](https://platform.openai.com/docs/guides/text-generation/json-mode) that ensures that the output is a valid JSON object.
* While this is great, it doesn't guarantee adherence to your custom JSON schemas, but only that the output IS a JSON.
* Anyscale and Together AI go further - they not only enforce that the output is in JSON but also ensure that the output follows any given JSON schema.
Using Portkey, you can easily experiment with models from Anyscale & Together AI and explore the power of their JSON modes:
1. Click on it and paste the [raw prompt code from above](/guides/use-cases/few-shot-prompting#this-is-how-our-raw-prompt-would-look). And that's it! You have your **dynamically updatable** few shot prompt template ready to deploy.
## Deploying the Prompt with Portkey
Deploying your prompt template to an API is extremely easy with Portkey. You can use our [Prompt Completions API](/portkey-endpoints/prompts/prompt-completion) to use the prompt we created.
This gives you an instant breakdown of LLM expenses per user or team.
***
## Step 3: Fetch Cost Data Programmatically via the Analytics API
For deeper integrations, the **Analytics API** enables real-time cost tracking inside your own application. This is useful for:
* **User-facing billing dashboards** (show users their LLM usage in real time)
* **Automated cost monitoring** (trigger alerts when a user’s spending exceeds a threshold)
* **Enterprise reporting** (export data for budget forecasting)
**Understanding the Analytics API**
The API provides comprehensive cost analytics data across any metadata dimension you've configured. You can query historical data, aggregate costs across different timeframes, and access detailed metrics for each metadata value.
Here's what you can access through the API:
```json
"data": [
{
"metadata_value": "kreacher",
"requests": 4344632,
"cost": 3887.3066999996863,
"avg_tokens": 447.3689256075083,
"avg_weighted_feedback": 4.2,
"requests_with_feedback": 10,
"last_seen": "2025-02-03T07:19:30.000Z",
"object": "analytics-group"
},
{
...more such objects
}
```
These metrics provide insights into costs, usage patterns, and efficiency. The response includes:
* Total requests and costs per metadata value
* Average token usage for optimization analysis
* User feedback metrics for quality assessment
* Timestamp data for temporal analysis
## Step 4: Tracking User Costs using Portkey's Analytics API
Before making your first API call, you'll need to obtain an API key from the Portkey Dashboard. This key requires analytics scope access, which you can configure in your API key settings.
### 6. [Prompt Management](/product/prompt-library)
Use Portkey as a centralized hub to store, version, and experiment with your agent's prompts across multiple LLMs. Easily modify your prompts and run A/B tests without worrying about the breaking prod.
### 7. [Continuous Improvement](/product/observability/feedback)
Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests, and then using that feedback to make your prompts AND LLMs themselves better.
### 8. [Security & Compliance](/product/enterprise-offering/security-portkey)
Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
# Autogen
Source: https://docs.portkey.ai/docs/integrations/agents/autogen
Use Portkey with Autogen to take your AI Agents to production
## Getting Started
### 1. Install the required packages:
```sh
pip install -qU pyautogen portkey-ai
```
### **2.** Configure your Autogen configs:
```py
from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
config = [
{
"api_key": "OPENAI_API_KEY",
"model": "gpt-3.5-turbo",
"base_url": PORTKEY_GATEWAY_URL,
"api_type": "openai",
"default_headers": createHeaders(
api_key ="PORTKEY_API_KEY", #Replace with Your Portkey API key
provider = "openai",
)
}
]
```
## Integration Guide
Here's a simple Google Colab notebook that demonstrates Autogen with Portkey integration
[](https://dub.sh/Autogen-docs)
## Make your agents Production-ready with Portkey
Portkey makes your Autogen agents reliable, robust, and production-grade with its observability suite and AI Gateway. Seamlessly integrate 200+ LLMs with your Autogen agents using Portkey. Implement fallbacks, gain granular insights into agent performance and costs, and continuously optimize your AI operations—all with just 2 lines of code.
Let's dive deep! Let's go through each of the use cases!
### 1.[ Interoperability](/product/ai-gateway/universal-api)
Easily switch between 200+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `provider ` and `API key` in the `ChatOpenAI` object.
### 5. [Continuous Improvement](/product/observability/feedback)
Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.
### 6. [Caching](/product/ai-gateway/cache-simple-and-semantic)
Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.
```json
{
"cache": {
"mode": "semantic" // Choose between "simple" or "semantic"
}
}
```
### 7. [Security & Compliance](/product/enterprise-offering/security-portkey)
Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
## [Portkey Config](/product/ai-gateway/configs)
Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
# Bring Your own Agents
Source: https://docs.portkey.ai/docs/integrations/agents/bring-your-own-agents
You can also use Portkey if you are doing custom agent orchestration!
## Getting Started
### 1. Install the required packages:
```sh
pip install portkey-ai openai
```
### **2.** Configure your OpenAI object:
```py
client = OpenAI(
api_key="OPENAI_API_KEY",
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="openai",
api_key="PORTKEY_API_KEY",
virtual_key="openai-latest-a4a53d"
)
)
```
## Integrate Portkey with your custom Agents
This notebook demonstrates integrating a ReAct agent with Portkey
[](https://dub.sh/ReAct-agent)
## Make your agents Production-ready with Portkey
Portkey makes your agents reliable, robust, and production-grade with its observability suite and AI Gateway. Seamlessly integrate 200+ LLMs with your custom agents using Portkey. Implement fallbacks, gain granular insights into agent performance and costs, and continuously optimize your AI operations—all with just 2 lines of code.
Let's dive deep! Let's go through each of the use cases!
### 1. [Interoperability](/product/ai-gateway/universal-api)
Easily switch between 200+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `provider ` and `API key` in the `ChatOpenAI` object.
### 5. [Continuous Improvement](/product/observability/feedback)
Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.
### 6. [Caching](/product/ai-gateway/cache-simple-and-semantic)
Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.
```py
{
"cache": {
"mode": "semantic" // Choose between "simple" or "semantic"
}
}
```
### 7.[ Security & Compliance](/product/enterprise-offering/security-portkey)
Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
***
## [Portkey Config](/product/ai-gateway/configs)
Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
# Control Flow
Source: https://docs.portkey.ai/docs/integrations/agents/control-flow
Use Portkey with Control Flow to take your AI Agents to production
## Getting Started
### 1. Install the required packages:
```sh
pip install -qU portkey-ai controlflow
```
### **2.** Configure your Control FLow LLM objects:
```py
import controlflow as cf
from langchain_openai import ChatOpenAI
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL
llm = ChatOpenAI(
api_key="OpenAI_API_Key",
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="openai", #choose your provider
api_key="PORTKEY_API_KEY"
)
)
```
## Integration Guide
Here's a simple Google Colab notebook that demonstrates Control Flow with Portkey integration
[](https://dub.sh/Control-Flow-docs)
## Make your agents Production-ready with Portkey
Portkey makes your Control Flow agents reliable, robust, and production-grade with its observability suite and AI Gateway. Seamlessly integrate 200+ LLMs with your Control Flow agents using Portkey. Implement fallbacks, gain granular insights into agent performance and costs, and continuously optimize your AI operations—all with just 2 lines of code.
Let's dive deep! Let's go through each of the use cases!
### 1. [Interoperability](/product/ai-gateway/universal-api)
Easily switch between 200+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `provider ` and `API key` in the `ChatOpenAI` object.
### 5. [Continuous Improvement](/product/observability/feedback)
Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.
### 6. [Caching](/product/ai-gateway/cache-simple-and-semantic)
Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.
```py
{
"cache": {
"mode": "semantic" // Choose between "simple" or "semantic"
}
}
```
### 7. [Security & Compliance](/product/enterprise-offering/security-portkey)
Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
***
## Portkey Config
Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
# CrewAI
Source: https://docs.portkey.ai/docs/integrations/agents/crewai
Use Portkey with CrewAI to take your AI Agents to production
CrewAI is a cutting-edge framework for orchestrating autonomous AI agents. When integrated with Portkey, it enables production-ready features like observability, reliability, and seamless multi-provider support. This integration helps you build robust, scalable agent systems while maintaining full control over their execution.
## Getting Started
# Langchain Agents
Source: https://docs.portkey.ai/docs/integrations/agents/langchain-agents
## Getting Started
### 1. Install the required packages:
```sh
pip install -qU langchain langchain-openai portkey-ai
```
### 2. Configure your Langchain LLM objects:
```py
from langchain_openai import ChatOpenAI, createHeaders
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL
llm1 = ChatOpenAI(
api_key="OpenAI_API_Key",
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="openai",
api_key="PORTKEY_API_KEY"
)
)
```
That's all you need to do to use Portkey with Llama Index agents. Execute your agents and visit [Portkey.ai](https://portkey.ai) to observe your Agent's activity.
## Integration Guide
Here's a simple Google Colab notebook that demonstrates Llama Index with Portkey integration
[](https://colab.research.google.com/drive/1ab%5FXnSf-HR1KndEGBgXDW6RvONKoHdzL?usp=sharing)
## Make your agents Production-ready with Portkey
Portkey makes your Llama Index agents reliable, robust, and production-grade with its observability suite and AI Gateway. Seamlessly integrate 200+ LLMs with your Llama Index agents using Portkey. Implement fallbacks, gain granular insights into agent performance and costs, and continuously optimize your AI operations—all with just 2 lines of code.
Let's dive deep! Let's go through each of the use cases!
### 1. [Interoperability](/product/ai-gateway/universal-api)
Easily switch between 200+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `provider` and `API key` in the `ChatOpenAI` object.
### 6. Guardrails
LLMs are brittle - not just in API uptimes or their inexplicable `400`/`500` errors, but also in their core behavior. You can get a response with a `200` status code that completely errors out for your app's pipeline due to mismatched output. With Portkey's Guardrails, we now help you enforce LLM behavior in real-time with our *Guardrails on the Gateway* pattern.
Using Portkey's Guardrail platform, you can now verify your LLM inputs AND outputs to be adhering to your specifed checks; and since Guardrails are built on top of our [Gateway](https://github.com/portkey-ai/gateway), you can orchestrate your request exactly the way you want - with actions ranging from *denying the request*, *logging the guardrail result*, *creating an evals dataset*, *falling back to another LLM or prompt*, *retrying the request*, and more.
### 7. [Continuous Improvement](/product/observability/feedback)
Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.
### 8. [Caching](/product/ai-gateway/cache-simple-and-semantic)
Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.
```json
{
"cache": {
"mode": "semantic" // Choose between "simple" or "semantic"
}
}
```
### 9. [Security & Compliance](/product/enterprise-offering/security-portkey)
Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
***
## [Portkey Config](/product/ai-gateway/configs)
Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
# LangGraph Agents
Source: https://docs.portkey.ai/docs/integrations/agents/langgraph
Use Portkey with LangGraph to take your AI Agents to production
### 5. [Traces](/product/observability/traces)
With traces, you can see each agent run granularly on Portkey. Tracing your LlamaIndex agent runs helps in debugging, performance optimzation, and visualizing how exactly your agents are running.
### Using Traces in LlamaIndex Agents
#### Step 1: Import & Initialize the Portkey LlamaIndex Callback Handler
```py
from portkey_ai.llamaindex import LlamaIndexCallbackHandler
portkey_handler = LlamaIndexCallbackHandler(
api_key="YOUR_PORTKEY_API_KEY",
metadata={
"session_id": "session_1", # Use consistent metadata across your application
"agent_id": "research_agent_1", # Specific to the current agent
}
)
```
#### Step 2: Configure Your LLM with the Portkey Callback
```py
from llama_index.llms.openai import OpenAI
llm = OpenAI(
api_key="YOUR_OPENAI_API_KEY_HERE",
callbacks=[portkey_handler], # Replace with your OpenAI API key
# ... other parameters
)
```
With Portkey tracing, you can encapsulate the complete execution of your agent workflow.
### 6. [Continuous Improvement](/product/observability/feedback)
Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.
### 7. [Caching](/product/ai-gateway/cache-simple-and-semantic)
Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.
```py
{
"cache": {
"mode": "semantic" // Choose between "simple" or "semantic"
}
}
```
### 8. [Security & Compliance](/product/enterprise-offering/security-portkey)
Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
***
## [Portkey Config](/product/ai-gateway/configs)
Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
# OpenAI Swarm
Source: https://docs.portkey.ai/docs/integrations/agents/openai-swarm
The Portkey x Swarm integration brings advanced AI gateway capabilities, full-stack observability, and reliability features to build production-ready AI agents.
Swarm is an experimental framework by OpenAI for building multi-agent systems. It showcases the handoff & routines pattern, making agent coordination and execution lightweight, highly controllable, and easily testable. Portkey integration extends Swarm's capabilities with production-ready features like observability, reliability, and more.
## Getting Started
### 2. Advanced Logs
Portkey's logging dashboard provides detailed logs for every request made to your LLMs. These logs include:
* Complete request and response tracking
* Metadata tags for filtering
* Cost attribution and much more...
### 3. Unified Access to 1600+ LLMs
You can easily switch between 1600+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `virtual key` in your default `config` object.
### 4. Advanced Metadata Tracking
Using Portkey, you can add custom metadata to your LLM requests for detailed tracking and analytics. Use metadata tags to filter logs, track usage, and attribute costs across departments and teams.
Find more information about Autogen here: [https://microsoft.github.io/autogen/docs/Getting-Started](https://microsoft.github.io/autogen/docs/Getting-Started)
## Quick Start Integration
Autogen supports a concept of [config\_list](https://microsoft.github.io/autogen/docs/llm%5Fconfiguration) which allows definitions of the LLM provider and model to be used. Portkey seamlessly integrates into the Autogen framework through a custom config we create.
### Example using minimal configuration
```py
from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
# Import the portkey library to fetch helper functions
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
config_list = [
{
"api_key": 'Your OpenAI Key',
"model": "gpt-3.5-turbo",
"base_url": PORTKEY_GATEWAY_URL,
"api_type": "openai",
"default_headers": createHeaders(
api_key = "Your Portkey API Key",
provider = "openai",
)
}
]
assistant = AssistantAgent("assistant", llm_config={"config_list": config_list})
user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False}) # IMPORTANT: set to True to run code in docker, recommended
user_proxy.initiate_chat(assistant, message="Say this is also a test - part 2.")
# This initiates an automated chat between the two agents to solve the task
```
Notice that we updated the `base_url` to Portkey's AI Gateway and then added `default_headers` to enable Portkey specific features.
When we execute this script, it would yield the same results as without Portkey, but every request can now be inspected in the Portkey Analytics & Logs UI - including token, cost, accuracy calculations.
All the config parameters supported in Portkey are available for use as part of the headers. Let's look at some examples:
## Using 100+ models in Autogen through Portkey
Since Portkey [seamlessly connects to 150+ models across providers](/integrations/llms), you can easily connect any of these to now run with Autogen.
Let's see an example using **Mistral-7B on Anyscale** running with Autogen seamlessly:
```py
from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
# Import the portkey library to fetch helper functions
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
config_list = [
{
"api_key": 'Your Anyscale API Key',
"model": "mistralai/Mistral-7B-Instruct-v0.1",
"base_url": PORTKEY_GATEWAY_URL,
"api_type": "openai", # Portkey conforms to the openai api_type
"default_headers": createHeaders(
api_key = "Your Portkey API Key",
provider = "anyscale",
)
}
]
assistant = AssistantAgent("assistant", llm_config={"config_list": config_list})
user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False}) # IMPORTANT: set to True to run code in docker, recommended
user_proxy.initiate_chat(assistant, message="Say this is also a test - part 2.")
# This initiates an automated chat between the two agents to solve the task
```
## Using a Virtual Key
[Virtual keys](/product/ai-gateway/virtual-keys) in Portkey allow you to easily switch between providers without manually having to store and change their API keys. Let's use the same Mistral example above, but this time using a Virtual Key.
```py
from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
# Import the portkey library to fetch helper functions
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
config_list = [
{
# Set a dummy value, since we'll pick the API key from the virtual key
"api_key": 'X',
# Pick the model from the provider of your choice
"model": "mistralai/Mistral-7B-Instruct-v0.1",
"base_url": PORTKEY_GATEWAY_URL,
"api_type": "openai", # Portkey conforms to the openai api_type
"default_headers": createHeaders(
api_key = "Your Portkey API Key",
# Add your virtual key here
virtual_key = "Your Anyscale Virtual Key",
)
}
]
assistant = AssistantAgent("assistant", llm_config={"config_list": config_list})
user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False}) # IMPORTANT: set to True to run code in docker, recommended
user_proxy.initiate_chat(assistant, message="Say this is also a test - part 2.")
# This initiates an automated chat between the two agents to solve the task
```
## Using Configs
[Configs](/product/ai-gateway/configs) in Portkey unlock advanced management and routing functionality including [load balancing](/product/ai-gateway/load-balancing), [fallbacks](/product/ai-gateway/fallbacks), [canary testing](/product/ai-gateway/canary-testing), [switching models](/product/ai-gateway/universal-api) and more.
You can use Portkey configs in Autogen like this:
```py
from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
# Import the portkey library to fetch helper functions
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
config_list = [
{
# Set a dummy value, since we'll pick the API key from the virtual key
"api_key": 'X',
# Pick the model from the provider of your choice
"model": "mistralai/Mistral-7B-Instruct-v0.1",
"base_url": PORTKEY_GATEWAY_URL,
"api_type": "openai", # Portkey conforms to the openai api_type
"default_headers": createHeaders(
api_key = "Your Portkey API Key",
# Add your Portkey config id
config = "Your Config ID",
)
}
]
assistant = AssistantAgent("assistant", llm_config={"config_list": config_list})
user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False}) # IMPORTANT: set to True to run code in docker, recommended
user_proxy.initiate_chat(assistant, message="Say this is also a test - part 2.")
# This initiates an automated chat between the two agents to solve the task
```
# DSPy
Source: https://docs.portkey.ai/docs/integrations/libraries/dspy
Integrate DSPy with Portkey for production-ready LLM pipelines
DSPy is a framework for algorithmically optimizing language model prompts and weights.
Portkey's integration with DSPy makes your DSPy pipelines production-ready with detailed insights on costs & performance metrics for each run, and also makes your existing DSPy code work across 250+ LLMs.
## Getting Started
### Installation
```sh
pip install dspy-ai==2.4.14 # Use Version 2.4.14 or higher
pip install portkey-ai
```
### Setting up
Portkey extends the existing `OpenAI` client in DSPy and makes it work with 250+ LLMs and gives you detailed cost insights. Just change `api_base` and add Portkey related headers in the `default_headers` param.
* `Request Details`: Information about the specific request, including the model used, input, and output.
* `Metrics`: Performance metrics such as latency, token usage, and cost.
* `Logs`: Detailed logs of the request, including any errors or warnings.
* `Traces`: A visual representation of the request flow, especially useful for complex DSPy modules.
## Portkey Features with DSPy
### 1. Interoperability
Portkey's Unified API enables you to easily switch between **250**+ language models. This includes the LLMs that are not natively integrated with DSPy. Here's how you can modify your DSPy setup to use Claude from Gpt-4 model:
### 3. Metrics
Portkey's Observability suite helps you track key metrics like **cost** and **token** usage, which is crucial for managing the high cost of DSPy. The observability dashboard helps you track 40+ key metrics, giving you detailed insights into your DSPy run.
### 4. Caching
Caching can significantly reduce these costs by storing frequently used data and responses. While DSPy has built-in simple caching, Portkey also offers advanced semantic caching to help you save more time and money.
Just modify your Portkey config as shown below and pass it with the `config` key in the `default_headers` param:
```python
config={ "cache": { "mode": "semantic" } }
turbo = dspy.OpenAI(
api_base=PORTKEY_GATEWAY_URL + "/",
model='gpt-4o',
api_key="YOUR_OPENAI_API_KEY", # Enter your OpenAI API key
model_type="chat",
default_headers=createHeaders(
api_key="YOUR_PORTKEY_API_KEY",
metadata={'_user': "dspy"},
provider="openai",
config=config
)
)
dspy.settings.configure(lm=turbo)
```
### 5. Reliability
Portkey offers built-in **fallbacks** between different LLMs or providers, **load-balancing** across multiple instances or API keys, and implementing automatic **retries** and request **timeouts**. This makes your DSPy more reliable and resilient.
Similar to caching example above, just define your Config and pass it with the `Config` key in the `default_headers` param.
```json
{
"retry": {
"attempts": 5
},
"strategy": {
"mode": "loadbalance" // Choose between "loadbalance" or "fallback"
},
"targets": [
{
"provider": "openai",
"api_key": "OpenAI_API_Key"
},
{
"provider": "anthropic",
"api_key": "Anthropic_API_Key"
}
]
}
```
### 6. Virtual Keys
Securely store your LLM API keys in Portkey vault and get a disposable virtual key with custom budget limits.
Add your API key in Portkey UI [here](https://app.portkey.ai/) to get a virtual key, and pass it in your request like this:
```python
turbo = dspy.OpenAI(
api_base=PORTKEY_GATEWAY_URL + "/",
model='gpt-4o',
api_key="xx",
model_type="chat",
default_headers=createHeaders(
api_key="YOUR_PORTKEY_API_KEY",
virtual_key="MY_OPENAI_VIRTUAL_KEY"
)
)
dspy.settings.configure(lm=turbo)
```
## Advanced Examples
### Retrieval-Augmented Generation (RAG) system
Make your RAG prompts better with Portkey x DSPy
## Using Virtual Keys for Multiple Models
Portkey supports [Virtual Keys](/product/ai-gateway/virtual-keys) which are an easy way to store and manage API keys in a secure vault. Lets try using a Virtual Key to make LLM calls.
#### 1. Create a Virtual Key in your Portkey account and the id
Let's try creating a new Virtual Key for Mistral like this
#### 2. Use Virtual Keys in the Portkey Headers
The `virtualKey` parameter sets the authentication and provider for the AI provider being used. In our case we're using the Mistral Virtual key.
This is extremely powerful since we gain control and visibility over the agent flows so we can identify problems and make updates as needed.
# Langchain (Python)
Source: https://docs.portkey.ai/docs/integrations/libraries/langchain-python
Portkey adds core production capabilities to any Langchain app.
[**Check out Observability docs here.**](/product/observability)
## 5. Prompt Management
Portkey features an advanced Prompts platform tailor-made for better prompt engineering. With Portkey, you can:
* **Store Prompts with Access Control and Version Control:** Keep all your prompts organized in a centralized location, easily track changes over time, and manage edit/view permissions for your team.
* **Parameterize Prompts**: Define variables and [mustache-approved tags](/product/prompt-library/prompt-templates#templating-engine) within your prompts, allowing for dynamic value insertion when calling LLMs. This enables greater flexibility and reusability of your prompts.
* **Experiment in a Sandbox Environment**: Quickly iterate on different LLMs and parameters to find the optimal combination for your use case, without modifying your LlamaIndex code.
#### Here's how you can leverage Portkey's Prompt Management in your LlamaIndex application:
1. Create your prompt template on the Portkey app, and save it to get an associated `Prompt ID`
2. Before making a Llamaindex request, render the prompt template using the Portkey SDK
3. Transform the retrieved prompt to be compatible with LlamaIndex and send the request!
#### Example: Using a Portkey Prompt Template in LlamaIndex
```py Portkey Prompts in LlamaIndex
import json
import os
from llama_index.llms.openai import OpenAI
from llama_index.core.llms import ChatMessage
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders, Portkey
### Initialize Portkey client with API key
client = Portkey(api_key=os.environ.get("PORTKEY_API_KEY"))
### Render the prompt template with your prompt ID and variables
prompt_template = client.prompts.render(
prompt_id="pp-prompt-id",
variables={ "movie":"Dune 2" }
).data.dict()
config = {
"virtual_key":"GROQ_VIRTUAL_KEY", # You need to send the virtual key separately
"override_params":{
"model":prompt_template["model"], # Set the model name based on the value in the prompt template
"temperature":prompt_template["temperature"] # Similarly, you can also set other model params
}
}
portkey = OpenAI(
api_base=PORTKEY_GATEWAY_URL,
api_key="xx" # Placeholder, no need to set
default_headers=createHeaders(
api_key=os.environ.get("PORTKEY_API_KEY"),
config=config
)
)
### Transform the rendered prompt into LlamaIndex-compatible format
messages = [ChatMessage(content=msg["content"], role=msg["role"]) for msg in prompt_template["messages"]]
resp = portkey.chat(messages)
print(resp)
```
[**Explore Prompt Management docs here**](/product/prompt-library).
***
## 6. Continuous Improvement
Now that you know how to trace & log your Llamaindex requests to Portkey, you can also start capturing user feedback to improve your app!
You can append qualitative as well as quantitative feedback to any `trace ID` with the `portkey.feedback.create` method:
```py Adding Feedback
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY"
)
feedback = portkey.feedback.create(
trace_id="YOUR_LLAMAINDEX_TRACE_ID",
value=5, # Integer between -10 and 10
weight=1, # Optional
metadata={
# Pass any additional context here like comments, _user and more
}
)
print(feedback)
```
[**Check out the Feedback documentation for a deeper dive**](/product/observability/feedback).
## 7. Security & Compliance
When you onboard more team members to help out on your Llamaindex app - permissioning, budgeting, and access management can become a mess! Using Portkey, you can set **budget limits** on provide API keys and implement **fine-grained user roles** and **permissions** to:
* **Control access**: Restrict team members' access to specific features, Configs, or API endpoints based on their roles and responsibilities.
* **Manage costs**: Set budget limits on API keys to prevent unexpected expenses and ensure that your LLM usage stays within your allocated budget.
* **Ensure compliance**: Implement strict security policies and audit trails to maintain compliance with industry regulations and protect sensitive data.
* **Simplify onboarding**: Streamline the onboarding process for new team members by assigning them appropriate roles and permissions, eliminating the need to share sensitive API keys or secrets.
* **Monitor usage**: Gain visibility into your team's LLM usage, track costs, and identify potential security risks or anomalies through comprehensive monitoring and reporting.
[**Read more about Portkey's Security & Enterprise offerings here**](/product/enterprise-offering).
## Join Portkey Community
Join the Portkey Discord to connect with other practitioners, discuss your LlamaIndex projects, and get help troubleshooting your queries.
[**Link to Discord**](https://portkey.ai/community)
For more detailed information on each feature and how to use them, please refer to the [Portkey Documentation](https://portkey.ai/docs).
# Microsoft Semantic Kernel
Source: https://docs.portkey.ai/docs/integrations/libraries/microsoft-semantic-kernel
# MindsDb
Source: https://docs.portkey.ai/docs/integrations/libraries/mindsdb
Integrate MindsDB with Portkey to build enterprise-grade AI use-cases
MindsDB connects to various data sources and LLMs, bringing data and AI together for easy AI automation.
With Portkey, you can run MindsDB AI systems with 250+ LLMs and implement enterprise-grade features like [LLM observability](/product/observability), [caching](/product/ai-gateway/cache-simple-and-semantic), [advanced routing](/product/ai-gateway), and more to build production-grade MindsDB AI apps.
## Prerequisites
Before proceeding, ensure the following prerequisites are met:
1. Install MindsDB locally via [Docker](https://docs.mindsdb.com/setup/self-hosted/docker) or [Docker Desktop](https://docs.mindsdb.com/setup/self-hosted/docker-desktop).
2. To use Portkey within MindsDB, install the required dependencies following [this instruction](https://docs.mindsdb.com/setup/self-hosted/docker#install-dependencies).
3. Obtain the [Portkey API key](https://app.portkey.ai) required to deploy and use Portkey within MindsDB.
## Setup
**Model Configuration**
1. In the Functions section, click the `...` button and select `Edit`
2. Find the virtual keys JSON object in the Portkey function code
3. Update it with your virtual keys:
```json
"virtual_keys": {
"openai": "YOUR_OPENAI_VIRTUAL_KEY",
"anthropic": "YOUR_ANTHROPIC_VIRTUAL_KEY"
}
```
4. Configure model names in the pipe function in this format:
```json
{
"id": "provider_slug_from_portkey/model_id_from_provider", // syntax for ID
"name": "provider_slug_from_portkey/model_id_from_provider", // for easier navigation
}
```
Example:
```json
{
"id": "openai/gpt-4o",
"name": "openai/gpt-4o",
}
```
***
## 4. Avoid Promptfoo Rate Limits & Leverage Cache
Since promptfoo can make a lot of calls very quickly, you can use a loadbalanced config in Portkey with cache enabled. You can pass the config header similar to virtual keys in promptfoo.
Here's a sample Config that you can save in the Portkey UI and get a respective config slug:
```json
{
"cache": { "mode": "simple" },
"strategy": { "mode": "loadbalance" },
"targets": [
{ "virtual_key": "ACCOUNT_ONE" },
{ "virtual_key": "ACCOUNT_TWO" },
{ "virtual_key": "ACCOUNT_THREE" }
]
}
```
And then we can just add the saved Config's slug in the YAML:
```yaml
providers:
id: portkey:claude-3-opus20240229
config:
portkeyVirtualKey: ANTHROPIC_VIRTUAL_KEY
portkeyConfig: PORTKEY_CONFIG_SLUG
```
***
## 
### Reliability
Portkey enhances the robustness of your AI applications with built-in features such as [Caching](/product/ai-gateway/cache-simple-and-semantic), [Fallback](/product/ai-gateway/fallbacks) mechanisms, [Load balancing](/product/ai-gateway/load-balancing), [Conditional routing](/product/ai-gateway/conditional-routing), [Request timeouts](/product/ai-gateway/request-timeouts), etc.
Here is how you can modify your config to include the following Portkey features-
In the individual log for any request, you can also see the exact status of your request and verify if it was cached, or delivered from cache with two `usage` parameters:
* `cache_creation_input_tokens`: Number of tokens written to the cache when creating a new entry.
* `cache_read_input_tokens`: Number of tokens retrieved from the cache for this request.
# Anyscale
Source: https://docs.portkey.ai/docs/integrations/llms/anyscale-llama2-mistral-zephyr
Integrate Anyscale endpoints with Portkey seamlessly and make your OSS models production-ready
Portkey's suite of features - AI gateway, observability, prompt management, and continuous fine-tuning are all enabled for the OSS models (Llama2, Mistral, Zephyr, and more) available on Anyscale endpoints.
**Here's a step-by-step guide:**
1. Request access to Azure OpenAI [here](https://aka.ms/oai/access).
2. Create a resource in the Azure portal [here](https://portal.azure.com/?microsoft%5Fazure%5Fmarketplace%5FItemHideKey=microsoft%5Fopenai%5Ftip#create/Microsoft.CognitiveServicesOpenAI). (This will be your **Resource Name**)
3. Deploy a model in Azure OpenAI Studio [here](https://oai.azure.com/). (This will be your **Deployment Name)**
4. Select your `Foundation Model` from the dropdowon on the modal.
5. Now, on Azure OpenAI studio, go to any playground (chat or completions), click on a UI element called "View code". Note down the API version & API key from here. (This will be your **Azure API Version** & **Azure API Key**)
When you input these details, the foundation model will be auto populated. More details in [this guide](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal).
Log view for an image generation request on Azure OpenAI
More information on image generation is available in the [API Reference](https://portkey.ai/docs/api-reference/completions-1#create-image).
***
## Making Requests Without Virtual Keys
Here's how you can pass your Azure OpenAI details & secrets directly without using the Virutal Keys feature.
### Key Mapping
In a typical Azure OpenAI request,
```sh
curl https://{YOUR_RESOURCE_NAME}.openai.azure.com/openai/deployments/{YOUR_DEPLOYMENT_NAME}/chat/completions?api-version={API_VERSION} \
-H "Content-Type: application/json" \
-H "api-key: {YOUR_API_KEY}" \
-d '{
"model": "gpt-4o",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant"
},
{
"role": "user",
"content": "what is a portkey?"
}
]
}'
```
| Parameter | Node SDK | Python SDK | REST Headers |
| --------------------- | ----------------------------------- | ------------------------------------ | ----------------------------- |
| AZURE RESOURCE NAME | azureResourceName | azure\_resource\_name | x-portkey-azure-resource-name |
| AZURE DEPLOYMENT NAME | azureDeploymentId | azure\_deployment\_id | x-portkey-azure-deployment-id |
| API VERSION | azureApiVersion | azure\_api\_version | x-portkey-azure-api-version |
| AZURE API KEY | Authorization: "Bearer + {API_KEY}" | Authorization = "Bearer + {API_KEY}" | Authorization |
| AZURE MODEL NAME | azureModelName | azure\_model\_name | x-portkey-azure-model-name |
### Example
* On the same [page](https://us-east-1.console.aws.amazon.com/iam/home#/security%5Fcredentials) under the '**Access keys'** section, where you created your Secret Access key, you will also find your **Access Key ID.**
* And lastly, get Your `AWS Region` from the Home Page of[ AWS Bedrock](https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/overview) as shown in the image below.
***
## Next Steps
The complete list of features supported in the SDK are available on the link below.
In addition to the `user` parameter, Portkey allows you to send arbitrary custom metadata with your requests. This powerful feature enables you to associate additional context or information with each request, which can be useful for analysis, debugging, or other custom use cases.
## Codestral v/s Mistral API Endpoint
Here's a handy guide for when you might want to make your requests to the Codestral endpoint v/s the original Mistral API endpoint:
[For more, check out Mistral's Code Generation guide here](https://docs.mistral.ai/capabilities/code%5Fgeneration/#operation/listModels).
***
## Managing Mistral AI Prompts
You can manage all prompts to Mistral AI in the [Prompt Library](/product/prompt-library). All the current models of Mistral AI are supported and you can easily start testing different prompts.
Once you're ready with your prompt, you can use the `portkey.prompts.completions.create` interface to use the prompt in your application.
## Next Steps
The complete list of features supported in the SDK are available on the link below.
In addition to the `user` parameter, Portkey allows you to send arbitrary custom metadata with your requests. This powerful feature enables you to associate additional context or information with each request, which can be useful for analysis, debugging, or other custom use cases.
2. Use this prompt in your codebase using the Portkey SDK.
More information on image generation is available in the [API Reference](/provider-endpoints/images/create-image#create-image).
### Audio - Transcription, Translation, and Text-to-Speech
Portkey's multimodal Gateway also supports the `audio` methods on OpenAI API. Check out the below guides for more info:
Check out the below guides for more info:
All requests, including those with fewer than 1024 tokens, will display a `cached_tokens` field of the `usage.prompt_tokens_details` [chat completions object](https://platform.openai.com/docs/api-reference/chat/object) indicating how many of the prompt tokens were a cache hit.
For requests under 1024 tokens, `cached_tokens` will be zero.
Portkey runs on our popular [open source Gateway](https://git.new/portkey). You can spin it up locally to make requests without sending them to the Portkey API.
***
### OpenAI Structured Outputs
Use structured outputs for more consistent and parseable responses:
**Uploading Base64 encoded images**
If you have an image or set of images locally, you can pass those to the model in base 64 encoded format. [Check out this example from OpenAI on how to do this](https://platform.openai.com/docs/guides/vision#uploading-base64-encoded-images).
Based on your access level, you might see the relevant permissions on the API key modal - tick the ones you'd like, name your API key, and save it.
## 2. Integrate Portkey
Portkey offers a variety of integration options, including SDKs, REST APIs, and native connections with platforms like OpenAI, Langchain, and LlamaIndex, among others.
### Through the OpenAI SDK
If you're using the **OpenAI SDK**, import the Portkey SDK and configure it within your OpenAI client object:
## Logs vs. Logs Metadata
* **Logs**: Complete log entries including request and response payloads
* **Logs Metadata**: Information such as timestamps, model used, tokens consumed, and other metrics without the actual content
## Related Features
### Alert Thresholds
You can configure alert thresholds to receive notifications before reaching your full budget:
1. Enter a value in the **Alert Threshold** field
2. When usage reaches this threshold, notifications will be sent to configured recipients
3. The API key continues to function until the full budget limit is reached
### Periodic Reset Options
Budget limits can be set to automatically reset at regular intervals:
* **No Periodic Reset**: The budget limit applies until exhausted
* **Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC
* **Reset Monthly**: Budget limits reset on the 1st of each month at 12 AM UTC
## Rate Limits
Rate limits control how frequently an API key can be used, helping you maintain application performance and prevent unexpected usage spikes.
### Setting Up Rate Limits
When creating a new API key or editing an existing one:
1. Toggle on **Add Rate Limit**
2. Choose your limit type:
* **Requests**: Limit based on number of API calls
* **Tokens**: Limit based on token consumption
3. Specify the limit value and time interval
### Time Intervals
Rate limits can be applied using three different time intervals:
* **Per Minute**: For granular control of high-frequency applications
* **Per Hour**: For balanced control of moderate usage
* **Per Day**: For broader usage management
When a rate limit is reached, subsequent requests are rejected until the time interval resets.
## Email Notifications
Email notifications keep relevant stakeholders informed about API key usage and when limits are approached or reached.
### Configuring Notifications
To set up email notifications for an API key with budget limits:
1. Toggle on **Email Notifications** when creating/editing an API key
2. Add recipient email addresses:
* Type an email address and click **New** or press Enter
* Add multiple recipients as needed
### Default Recipients
When limits are reached or thresholds are crossed, Portkey automatically sends notifications to:
* Organization administrators
* Organization owners
* The API key creator/owner
You can add additional recipients such as finance team members, department heads, or project managers who need visibility into AI usage.
## Availability
These features are available to Portkey Enterprise customers and select Pro users. To enable these features for your account, please contact [support@portkey.ai](mailto:support@portkey.ai) or join the [Portkey Discord](https://portkey.ai/community) community.
To learn more about the Portkey Enterprise plan, [schedule a consultation](https://portkey.sh/demo-16).
# Enforcing Default Configs on API Keys
Source: https://docs.portkey.ai/docs/product/administration/enforce-default-config
Learn how to attach default configs to API keys for enforcing governance controls across your organization
## Overview
Portkey allows you to attach default configs to API keys, enabling you to enforce specific routing rules, security controls, and other governance measures across all API calls made with those keys. This feature provides a powerful way to implement organization-wide policies without requiring changes to individual application code.
You can also [self-host](https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md) the gateway and then connect it to Portkey. Please reach out on [hello@portkey.ai](mailto:hello@portkey.ai) and we'll help you set this up!
# Automatic Retries
Source: https://docs.portkey.ai/docs/product/ai-gateway/automatic-retries
LLM APIs often have inexplicable failures. With Portkey, you can rescue a substantial number of your requests with our in-built automatic retries feature.
For each request we also calculate and show the cache response time and how much money you saved with each hit.
***
## How Cache works with Configs
You can set cache at two levels:
* **Top-level** that works across all the targets.
* **Target-level** that works when that specific target is triggered.
## Enabling Fallback on LLMs
To enable fallbacks, you can modify the [config object](/api-reference/config-object) to include the `fallback` mode.
Here's a quick example of a config to **fallback** to Anthropic's `claude-3.5-sonnet` if OpenAI's `gpt-4o` fails.
```JSON
{
"strategy": {
"mode": "fallback"
},
"targets": [
{
"virtual_key": "openai-virtual-key",
"override_params": {
"model": "gpt-4o"
}
},
{
"virtual_key": "anthropic-virtual-key",
"override_params": {
"model": "claude-3.5-sonnet-20240620"
}
}
]
}
```
In this scenario, if the OpenAI model encounters an error or fails to respond, Portkey will automatically retry the request with Anthropic.
[Using Configs in your Requests](/product/ai-gateway/configs#using-configs)
## Triggering fallback on specific error codes
By default, fallback is triggered on any request that returns a **non-2xx** status code.
You can change this behaviour by setting the optional `on_status_codes` param in your fallback config and manually inputting the status codes on which fallback will be triggered.
```sh
{
"strategy": {
"mode": "fallback",
"on_status_codes": [ 429 ]
},
"targets": [
{
"virtual_key": "openai-virtual-key"
},
{
"virtual_key": "azure-openai-virtual-key"
}
]
}
```
Here, fallback from OpenAI to Azure OpenAI will only be triggered when there is a `429` error code from the OpenAI request (i.e. rate limiting error)
## Tracing Fallback Requests on Portkey
Portkey logs all the requests that are sent as a part of your fallback config. This allows you to easily trace and see which targets failed and see which ones were eventually successful.
To see your fallback trace,
1. On the Logs page, first filter the logs with the specific `Config ID` where you've setup the fallback - this will show all the requests that have been sent with that config.
2. Now, trace an individual request and all the failed + successful logs for it by filtering further on `Trace ID` - this will show all the logs originating from a single request.
## Caveats and Considerations
While the Fallback on LLMs feature greatly enhances the reliability and resilience of your application, there are a few things to consider:
1. Ensure the LLMs in your fallback list are compatible with your use case. Not all LLMs offer the same capabilities.
2. Keep an eye on your usage with each LLM. Depending on your fallback list, a single request could result in multiple LLM invocations.
3. Understand that each LLM has its own latency and pricing. Falling back to a different LLM could have implications on the cost and response time.
# Files
Source: https://docs.portkey.ai/docs/product/ai-gateway/files
Upload files to Portkey and reuse the content in your requests
Portkey supports managing files in two ways:
1. Uploading and managing files to any provider using the unified signature
2. \[Enterprise Only] Uploading files to Portkey and using them for batching/fine-tuning requests with any provider
## 1. Uploading and managing files to any provider using the unified signature
Please refer to the [Provider Specific Files](/integrations/llms/openai/files) documentation for more details.
1. [OpenAI](/integrations/llms/openai/files)
2. [Bedrock](/integrations/llms/bedrock/files)
3. [Azure OpenAI](/integrations/llms/azure-openai/files)
4. [Fireworks](/integrations/llms/fireworks/files)
5. [Vertex](/integrations/llms/vertex-ai/files)
## 2. \[Enterprise Only] Uploading files to Portkey and using them for batching/fine-tuning requests with any provider
With Portkey, you can upload files to Portkey and reuse them for [batching inference](/product/ai-gateway/batches) with any provider and [fine-tuning](/product/ai-gateway/fine-tuning) with supported providers.
In this way, you can test your data with different foundation models, perform A/B testing with different foundation models, and perform batch inference with different foundation models.
### Uploading Files
```sh
curl --location --request POST 'https://api.portkey.ai/v1/files' \
--header 'x-portkey-api-key: 
#### Explore the AI Gateway's Multimodal capabilities below:
## Managing Functions and Tools in Prompts
Portkey's Prompt Library supports creating prompt templates with function/tool definitions, as well as letting you set the `tool choice` param. Portkey will also validate your tool definition on the fly, eliminating syntax errors.
## Supported Providers and Models
The following providers are supported for function calling with more providers getting added soon. Please raise a [request](/integrations/llms/suggest-a-new-integration) or a [PR](https://github.com/Portkey-AI/gateway/pulls) to add model or provider to the AI gateway.
| Provider | Models |
| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| [OpenAI](/integrations/llms/openai) | gpt-4 series of modelsgpt-3.5-turbo series of models |
| [Azure OpenAI](/integrations/llms/azure-openai) | gpt-4 series of modelsgpt-3.5-turbo series of models |
| [Anyscale](/integrations/llms/anyscale-llama2-mistral-zephyr) | mistralai/Mistral-7B-Instruct-v0.1 mistralai/Mixtral-8x7B-Instruct-v0.1 |
| [Together AI](/integrations/llms/together-ai) | mistralai/Mixtral-8x7B-Instruct-v0.1 mistralai/Mistral-7B-Instruct-v0.1 togethercomputer/CodeLlama-34b-Instruct |
| [Fireworks AI](/integrations/llms/fireworks) | firefunction-v1fw-function-call-34b-v0 |
| [Google Gemini](/integrations/llms/gemini) / [Vertex AI](/integrations/llms/vertex-ai) | gemini-1.0-progemini-1.0-pro-001gemini-1.5-pro-latest |
## Cookbook
[**Here's a detailed cookbook on function calling using Portkey.**](/guides/getting-started/function-calling)
# Image Generation
Source: https://docs.portkey.ai/docs/product/ai-gateway/multimodal-capabilities/image-generation
Portkey's AI gateway supports image generation capabilities that many foundational model providers offer.
The most common use case is that of **text-to-image** where the user sends a prompt which the image model processes and returns an image.
## Supported Providers and Models
The following providers are supported for image generation with more providers getting added soon. Please raise a [request](/integrations/llms/suggest-a-new-integration) or a [PR](https://github.com/Portkey-AI/gateway/pulls) to add model or provider to the AI gateway.
| Provider | Models | Functions |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------- |
| [OpenAI](/integrations/llms/openai) | dall-e-2, dall-e-3 | Create Image (text to image) |
| [Azure OpenAI](/integrations/llms/azure-openai) | dall-e-2, dall-e-3 | Create Image (text to image) |
| [Stability](/integrations/llms/stability-ai) | stable-diffusion-v1-6, stable-diffusion-xl-1024-v1-0 | Create Image (text to image) |
| [AWS Bedrock](/integrations/llms/aws-bedrock) | `Stable Image Ultra, Stable Diffusion 3 Large (SD3 Large), Stable Image Core, Stable Diffusion XL 1.0` | Create Image (text to image) |
| Segmind | [Refer here](/integrations/llms/segmind) | Create Image (text to image) |
| Together AI (Coming Soon) | | |
| Monster API (Coming Soon) | | |
| Replicate (Coming Soon) | | |
## Cookbook
[**Here's a detailed cookbook on image generation using Portkey**](https://github.com/Portkey-AI/portkey-cookbook/blob/main/examples/image-generation.ipynb) which demonstrates the use of multiple providers and routing between them through Configs.
# Speech-to-Text
Source: https://docs.portkey.ai/docs/product/ai-gateway/multimodal-capabilities/speech-to-text
Portkey's AI gateway supports STT models like Whisper by OpenAI.
## Transcription & Translation Usage
Portkey supports both `Transcription` and `Translation` methods for STT models and follows the OpenAI signature where you can send the file (in `flac`, `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `ogg`, `wav`, or `webm` formats) as part of the API request.
Here's an example:
OpenAI NodeJSOpenAI PythonREST
## Creating prompt templates for vision models
Portkey's prompt library supports creating templates with image inputs. If the same image will be used in all prompt calls, you can save it as part of the template's image URL itself. Or, if the image will be sent via the API as a variable, add a variable to the image link.
## Supported Providers and Models
Portkey supports all vision models from its integrated providers as they become available. The table below shows some examples of supported vision models. Please raise a [request](/integrations/llms/suggest-a-new-integration) or a [PR](https://github.com/Portkey-AI/gateway/pulls) to add a provider to the AI gateway.
| Provider | Models | Functions |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------- | ---------------------- |
| [OpenAI](/integrations/llms/openai) | `gpt-4-vision-preview`, `gpt-4o`, `gpt-4o-mini ` | Create Chat Completion |
| [Azure OpenAI](/integrations/llms/azure-openai) | `gpt-4-vision-preview`, `gpt-4o`, `gpt-4o-mini ` | Create Chat Completion |
| [Gemini](/integrations/llms/gemini) | `gemini-1.0-pro-vision `, `gemini-1.5-flash`, `gemini-1.5-flash-8b`, `gemini-1.5-pro` | Create Chat Completion |
| [Anthropic](/integrations/llms/anthropic) | `claude-3-sonnet`, `claude-3-haiku`, `claude-3-opus`, `claude-3.5-sonnet`, `claude-3.5-haiku` | Create Chat Completion |
| [AWS Bedrock](/integrations/llms/aws-bedrock) | `anthropic.claude-3-5-sonnet anthropic.claude-3-5-haiku anthropic.claude-3-5-sonnet-20240620-v1:0` | Create Chat Completion |
For a complete list of all supported provider (including non-vision LLMs), check out our [providers documentation](/integrations/llms).
# Realtime API
Source: https://docs.portkey.ai/docs/product/ai-gateway/realtime-api
Use OpenAI's Realtime API with logs, cost tracking, and more!
## Next Steps
* [For more info on realtime API, refer here](https://platform.openai.com/docs/guides/realtime)
* [Portkeys OpenAI Integration](/integrations/llms/openai)
* [Logs](/product/observability/logs)
* [Traces](/product/observability/traces)
* [Guardrails](/product/ai-gateway/guardrails)
# Request Timeouts
Source: https://docs.portkey.ai/docs/product/ai-gateway/request-timeouts
Manage unpredictable LLM latencies effectively with Portkey's **Request Timeouts**.
To use the required deployment, simply pass the `alias` of the deployment as the `model` in LLM request body. In case the models is left empty or the specified alias does not exist, the default deployment is used.
## How are the provider API keys stored?
Your API keys are encrypted and stored in secure vaults, accessible only at the moment of a request. Decryption is performed exclusively in isolated workers and only when necessary, ensuring the highest level of data security.
## How are the provider keys linked to the virtual key?
We randomly generate virtual keys and link them separately to the securely stored keys. This means, your raw API keys can not be reverse engineered from the virtual keys.
## Using Virtual Keys
### Using the Portkey SDK
Add the virtual key directly to the initialization configuration for Portkey.
## Create an AWS Role for Portkey to Assume
This role you create will be used by Porktey to execute InvokeModel commands on Bedrock models in your AWS account. The setup process will establish a minimal-permission ("least privilege") role and set it up to allow Porktey to assume this role.
### Create a permission policy in your AWS account using the following JSON
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BedrockConsole",
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream"
],
"Resource": "*"
}
]
}
```
### Create a new IAM role
Choose *AWS account* as the trusted entity type. If you set an external ID be sure to copy it, we will need it later.
### Add the above policy to the role
Search for the policy you created above and add it to the role.
### Configure Trust Relationship for the role
Once the role is created, open the role and navigate to the *Trust relationships* tab and click *Edit trust policy*.
This is where you will add the Portkey AWS account as a trusted entity.
```sh Portkey Account ARN
arn:aws:iam::299329113195:role/portkey-app
```
You're all set! You can now use the virtual key to invoke Bedrock models.
# Budget Limits
Source: https://docs.portkey.ai/docs/product/ai-gateway/virtual-keys/budget-limits
Budget Limits lets you set cost limits on virtual keys
Budget Limits lets you set cost or token limits on virtual keys
With the ability to create and manage multiple organizations, you can tailor access control to match your company's structure and project requirements. Users can be assigned to specific organizations, and they can seamlessly switch between them using Portkey's intuitive user interface.
## 2. Fine-Grained User Roles and Permissions
Portkey offers a comprehensive Role-Based Access Control (RBAC) system that allows you to define and assign user roles with granular permissions. By default, Portkey provides three roles: `Owner`, `Admin`, and `Member`, each with a predefined set of permissions across various features.
* `Owners` have complete control over the organization, including user management, billing, and all platform features.
* `Admins` have elevated privileges, allowing them to manage users, prompts, configs, guardrails, virtual keys, and API keys.
* `Members` have access to essential features like logs, analytics, prompts, configs, and virtual keys, with limited permissions.
| Feature | Owner Role | Admin Role | Member Role |
| ------------------ | ------------------------------------------- | ------------------------------------------- | -------------------------- |
| Logs and Analytics | View, Filter, Group | View, Filter, Group | View, Filter, Group |
| Prompts | List, View, Create, Update, Delete, Publish | List, View, Create, Update, Delete, Publish | List, View, Create, Update |
| Configs | List, View, Create, Update, Delete | List, View, Create, Update, Delete | List, View, Create |
| Guardrails | List, View, Create, Update, Delete | List, View, Create, Update, Delete | List, View, Create, Update |
| Virtual Keys | List, Create, Edit, Duplicate, Delete, Copy | List, Create, Edit, Duplicate, Delete, Copy | List, Copy |
| Team | Add users, assign roles | Add users, assign roles | - |
| Organisation | Update | Update | - |
| API Keys | Create, Edit, Delete, Update, Rotate | Create, Edit, Delete, Update, Rotate | - |
| Billing | Manage | - | - |
You can easily add team members to your organization and assign them appropriate roles based on their responsibilities. Portkey's user-friendly interface simplifies the process of inviting users and managing their roles, ensuring that the right people have access to the right resources.
## 3. Secure and Customizable API Key Management
Portkey provides a secure and flexible API key management system that allows you to create and manage multiple API keys with fine-grained permissions. Each API key can be customized to grant specific access levels to different features, such as metrics, completions, prompts, configs, guardrails, virtual keys, team management, and API key management.
| Feature | Permissions | Default |
| --------------------------- | ----------------------------- | -------- |
| Metrics | Disabled, Enabled | Disabled |
| Completions (all LLM calls) | Disabled, Enabled | Enabled |
| Prompts | Disabled, Read, Write, Delete | Read |
| Configs | Disabled, Read, Write, Delete | Disabled |
| Guardrails | Disabled, Read, Write, Delete | Disabled |
| Virtual Keys | Disabled, Read, Write, Delete | Disabled |
| Users (Team Management) | Disabled, Read, Write, Delete | Disabled |
By default, a new organization is provisioned with a master API key that has all permissions enabled. Owners and admins can edit and manage these keys, as well as create new API keys with tailored permissions. This granular control enables you to enforce the principle of least privilege, ensuring that each API key has access only to the necessary resources.
Portkey's API key management system provides a secure and auditable way to control access to your organization's data and resources, reducing the risk of unauthorized access and data breaches.
## Audit Logs
Portkey maintains detailed audit logs that capture all administrative activities across the platform. These logs provide visibility into actions related to prompts, configs, guardrails, virtual keys, team management, organization updates, and API key modifications.
Each log entry includes information about the user, the action performed, the affected resource, and a timestamp. This ensures traceability and accountability, helping teams monitor changes and investigate any unauthorized activity.
Audit logs can be filtered by user, action type, resource, and time range, making it easy to track specific events. Organizations can use this data to enforce security policies, ensure compliance, and maintain operational integrity.
Portkey’s audit logging system provides a clear and structured way to review platform activity, ensuring security and compliance across all operations.
# Audit Logs
Source: https://docs.portkey.ai/docs/product/enterprise-offering/audit-logs
Track and monitor all administrative activities across your Portkey organization with comprehensive audit logging.
### Available Filters
* **Method**: Filter by HTTP method (PUT, POST, DELETE)
* **Request ID**: Search for a specific request by its unique identifier
* **Resource Type**: Filter by type of resource affected:
* Workspaces
* API Keys
* Virtual Keys
* Configs
* Prompts
* Guardrails
* Integrations
* Collections
* Organization
* Labels
* Custom Resource Types
* **Action**: Filter by the type of action performed:
* Create
* Update
* Delete
* Publish
* Export
* Rotate
* Manage
* Duplicate
* **Response Status**: Filter by HTTP response status codes
* **Workspace**: Filter by specific workspace
* **User**: Filter by the user who performed the action
* **Client IP**: Filter by originating IP address
* **Country**: Filter by geographic location of requests
* **Time Range**: Filter logs within a specific time period
## Enterprise Features
Portkey's Audit Logs include enterprise-grade capabilities:
### 1. Complete Visibility
* Full user attribution for every action
* Detailed timestamps and change history
* Cross-workspace tracking
* Searchable audit trail
### 2. Compliance & Security
* SOC 2, ISO 27001, GDPR, and HIPAA compliant
* PII data protection
* Indefinite log retention
### 3. Enterprise-Grade Features
* Role-based access control
* Cross-organization visibility
* Custom retention policies
> As a Director of AI Infrastructure at a Fortune 100 Healthcare company explained: *"Having a detailed audit trail isn't just about compliance. It's about being able to debug production issues quickly, understand usage patterns, and make data-driven decisions about our AI infrastructure."*
## Related Features
## Components and Sizing Recommendations
| Component | Options | Sizing Recommendations |
| --------------------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| AI Gateway | Deploy as a Docker container in your Kubernetes cluster using Helm Charts | AWS NodeGroup t4g.medium instance, with at least 4GiB of memory and two vCPUs For high reliability, deploy across multiple Availability Zones. |
| Logs store | AWS S3 | Each log document is \~10kb in size (uncompressed) |
| Cache (Prompts, Configs & Virtual Keys) | Elasticache or self-hosted Redis | Deploy in the same VPC as the Portkey Gateway. |
## Helm Chart
This deployment uses the Portkey AI hybrid Helm chart to deploy the Portkey AI Gateway. You can find more information about the Helm chart in the [Portkey AI Helm Chart GitHub Repository](https://github.com/Portkey-AI/helm/blob/main/charts/portkey-gateway/README.md).
## Prerequisites
1. Create a Portkey account on [Portkey AI](https://app.portkey.ai)
2. Portkey team will share the credentials for the private Docker registry.
## Markeplace Listing
### Visit Portkey AI AWS Marketplace Listing
You can find the Portkey AI AWS Marketplace listing [here](https://aws.amazon.com/marketplace/pp/prodview-o2leb4xcrkdqa).
### Subscribe to Portkey AI Enterprise Edition
Subscribe to the Portkey AI Enterprise Edition to gain access to the Portkey AI Gateway.
### Quick Launch
Upon subscribing to the Portkey AI Enterprise Edition, you will be able to select Quick Launch from within your AWS Console Subscriptions.
### Launch the Cloud Formation Template
Select the Portkey AI Enterprise Edition and click on Quick Launch.
### Run the Cloud Formation Template
Fill the required parameters and click on Next and run the Cloud Formation Template.
## Cloud Formation Steps
* Creates a new EKS cluster and NodeGroup in your selected VPC and Subnets
* Sets up IAM Roles needed for S3 bucket access using STS and Lambda execution
* Uses AWS Lambda to:
* Install the Portkey AI Helm chart to your EKS cluster
* Upload the values.yaml file to the S3 bucket
* Allows for changes to the values file or helm chart deployment by updating and re-running the same Lambda function in your AWS account
### Cloudformation Template
> Our cloudformation template has passed the AWS Marketplace validation and security review.
```yaml portkey-hybrid-eks-cloudformation.template.yaml [expandable]
AWSTemplateFormatVersion: "2010-09-09"
Description: Portkey deployment template for AWS Marketplace
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Required Parameters"
Parameters:
- VPCID
- Subnet1ID
- Subnet2ID
- ClusterName
- NodeGroupName
- NodeGroupInstanceType
- SecurityGroupID
- CreateNewCluster
- HelmChartVersion
- PortkeyDockerUsername:
NoEcho: true
- PortkeyDockerPassword:
NoEcho: true
- PortkeyClientAuth:
NoEcho: true
- Label:
default: "Optional Parameters"
Parameters:
- PortkeyOrgId
- PortkeyGatewayIngressEnabled
- PortkeyGatewayIngressSubdomain
- PortkeyFineTuningEnabled
Parameters:
# Required Parameters
VPCID:
Type: AWS::EC2::VPC::Id
Description: VPC where the EKS cluster will be created
Default: Select a VPC
Subnet1ID:
Type: AWS::EC2::Subnet::Id
Description: First subnet ID for EKS cluster
Default: Select your subnet
Subnet2ID:
Type: AWS::EC2::Subnet::Id
Description: Second subnet ID for EKS cluster
Default: Select your subnet
# Optional Parameters with defaults
ClusterName:
Type: String
Description: Name of the EKS cluster (if not provided, a new EKS cluster will be created)
Default: portkey-eks-cluster
NodeGroupName:
Type: String
Description: Name of the EKS node group (if not provided, a new EKS node group will be created)
Default: portkey-eks-cluster-node-group
NodeGroupInstanceType:
Type: String
Description: EC2 instance type for the node group (if not provided, t3.medium will be used)
Default: t3.medium
AllowedValues:
- t3.medium
- t3.large
- t3.xlarge
PortkeyDockerUsername:
Type: String
Description: Docker username for Portkey (provided by the Portkey team)
Default: portkeyenterprise
PortkeyDockerPassword:
Type: String
Description: Docker password for Portkey (provided by the Portkey team)
Default: ""
NoEcho: true
PortkeyClientAuth:
Type: String
Description: Portkey Client ID (provided by the Portkey team)
Default: ""
NoEcho: true
PortkeyOrgId:
Type: String
Description: Portkey Organisation ID (provided by the Portkey team)
Default: ""
HelmChartVersion:
Type: String
Description: Version of the Helm chart to deploy
Default: "latest"
AllowedValues:
- latest
SecurityGroupID:
Type: String
Description: Optional security group ID for the EKS cluster (if not provided, a new security group will be created)
Default: ""
CreateNewCluster:
Type: String
AllowedValues: [true, false]
Default: true
Description: Whether to create a new EKS cluster or use an existing one
PortkeyGatewayIngressEnabled:
Type: String
AllowedValues: [true, false]
Default: false
Description: Whether to enable the Portkey Gateway ingress
PortkeyGatewayIngressSubdomain:
Type: String
Description: Subdomain for the Portkey Gateway ingress
Default: ""
PortkeyFineTuningEnabled:
Type: String
AllowedValues: [true, false]
Default: false
Description: Whether to enable the Portkey Fine Tuning
Conditions:
CreateSecurityGroup: !Equals [!Ref SecurityGroupID, ""]
ShouldCreateCluster: !Equals [!Ref CreateNewCluster, true]
Resources:
PortkeyAM:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
RoleName: PortkeyAM
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
Action: sts:AssumeRole
Policies:
- PolicyName: PortkeyEKSAccess
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "eks:DescribeCluster"
- "eks:ListClusters"
- "eks:ListNodegroups"
- "eks:ListFargateProfiles"
- "eks:ListNodegroups"
- "eks:CreateCluster"
- "eks:CreateNodegroup"
- "eks:DeleteCluster"
- "eks:DeleteNodegroup"
- "eks:UpdateClusterConfig"
- "eks:UpdateKubeconfig"
Resource: !Sub "arn:aws:eks:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}"
- Effect: Allow
Action:
- "sts:AssumeRole"
Resource: !Sub "arn:aws:iam::${AWS::AccountId}:role/PortkeyAM"
- Effect: Allow
Action:
- "sts:GetCallerIdentity"
Resource: "*"
- Effect: Allow
Action:
- "iam:ListRoles"
- "iam:GetRole"
Resource: "*"
- Effect: Allow
Action:
- "bedrock:InvokeModel"
- "bedrock:InvokeModelWithResponseStream"
Resource: "*"
- Effect: Allow
Action:
- "s3:GetObject"
- "s3:PutObject"
Resource:
- !Sub "arn:aws:s3:::${AWS::AccountId}-${AWS::Region}-portkey-logs/*"
PortkeyLogsBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
Properties:
BucketName: !Sub "${AWS::AccountId}-${AWS::Region}-portkey-logs"
VersioningConfiguration:
Status: Enabled
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
# EKS Cluster Role
EksClusterRole:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
RoleName: EksClusterRole-Portkey
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: eks.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
# EKS Cluster Security Group (if not provided)
EksSecurityGroup:
Type: AWS::EC2::SecurityGroup
Condition: CreateSecurityGroup
DeletionPolicy: Delete
Properties:
GroupDescription: Security group for Portkey EKS cluster
VpcId: !Ref VPCID
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 8787
ToPort: 8787
CidrIp: PORTKEY_IP
SecurityGroupEgress:
- IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: 0.0.0.0/0
# EKS Cluster
EksCluster:
Type: AWS::EKS::Cluster
Condition: ShouldCreateCluster
DeletionPolicy: Delete
DependsOn: EksClusterRole
Properties:
Name: !Ref ClusterName
Version: "1.32"
RoleArn: !GetAtt EksClusterRole.Arn
ResourcesVpcConfig:
SecurityGroupIds:
- !If
- CreateSecurityGroup
- !Ref EksSecurityGroup
- !Ref SecurityGroupID
SubnetIds:
- !Ref Subnet1ID
- !Ref Subnet2ID
AccessConfig:
AuthenticationMode: API_AND_CONFIG_MAP
LambdaExecutionRole:
Type: AWS::IAM::Role
DeletionPolicy: Delete
DependsOn: EksCluster
Properties:
RoleName: PortkeyLambdaRole
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: EKSAccess
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- ec2:DescribeInstances
- ec2:DescribeRegions
Resource: "*"
- Effect: Allow
Action:
- "sts:AssumeRole"
Resource: !GetAtt PortkeyAM.Arn
- Effect: Allow
Action:
- "s3:GetObject"
- "s3:PutObject"
Resource:
- !Sub "arn:aws:s3:::${AWS::AccountId}-${AWS::Region}-portkey-logs/*"
- Effect: Allow
Action:
- "eks:DescribeCluster"
- "eks:ListClusters"
- "eks:ListNodegroups"
- "eks:ListFargateProfiles"
- "eks:ListNodegroups"
- "eks:CreateCluster"
- "eks:CreateNodegroup"
- "eks:DeleteCluster"
- "eks:DeleteNodegroup"
- "eks:CreateFargateProfile"
- "eks:DeleteFargateProfile"
- "eks:DescribeFargateProfile"
- "eks:UpdateClusterConfig"
- "eks:UpdateKubeconfig"
Resource: !Sub "arn:aws:eks:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}"
LambdaClusterAdmin:
Type: AWS::EKS::AccessEntry
DependsOn: EksCluster
Properties:
ClusterName: !Ref ClusterName
PrincipalArn: !GetAtt LambdaExecutionRole.Arn
Type: STANDARD
KubernetesGroups:
- system:masters
AccessPolicies:
- PolicyArn: "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
AccessScope:
Type: "cluster"
# Node Group Role
NodeGroupRole:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
RoleName: NodeGroupRole-Portkey
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
# EKS Node Group
EksNodeGroup:
Type: AWS::EKS::Nodegroup
DependsOn: EksCluster
DeletionPolicy: Delete
Properties:
CapacityType: ON_DEMAND
ClusterName: !Ref ClusterName
NodegroupName: !Ref NodeGroupName
NodeRole: !GetAtt NodeGroupRole.Arn
InstanceTypes:
- !Ref NodeGroupInstanceType
ScalingConfig:
MinSize: 1
DesiredSize: 1
MaxSize: 1
Subnets:
- !Ref Subnet1ID
- !Ref Subnet2ID
PortkeyInstallerFunction:
Type: AWS::Lambda::Function
DependsOn: EksNodeGroup
DeletionPolicy: Delete
Properties:
FunctionName: portkey-eks-installer
Runtime: nodejs18.x
Handler: index.handler
MemorySize: 1024
EphemeralStorage:
Size: 1024
Code:
ZipFile: |
const fs = require('fs');
const zlib = require('zlib');
const { pipeline } = require('stream');
const path = require('path');
const https = require('https');
const { promisify } = require('util');
const { execSync } = require('child_process');
const { EKSClient, DescribeClusterCommand } = require('@aws-sdk/client-eks');
async function unzipAwsCli(zipPath, destPath) {
// ZIP file format: https://en.wikipedia.org/wiki/ZIP_(file_format)
const data = fs.readFileSync(zipPath);
let offset = 0;
// Find end of central directory record
const EOCD_SIGNATURE = 0x06054b50;
for (let i = data.length - 22; i >= 0; i--) {
if (data.readUInt32LE(i) === EOCD_SIGNATURE) {
offset = i;
break;
}
}
// Read central directory info
const numEntries = data.readUInt16LE(offset + 10);
let centralDirOffset = data.readUInt32LE(offset + 16);
// Process each file
for (let i = 0; i < numEntries; i++) {
// Read central directory header
const signature = data.readUInt32LE(centralDirOffset);
if (signature !== 0x02014b50) {
throw new Error('Invalid central directory header');
}
const fileNameLength = data.readUInt16LE(centralDirOffset + 28);
const extraFieldLength = data.readUInt16LE(centralDirOffset + 30);
const fileCommentLength = data.readUInt16LE(centralDirOffset + 32);
const localHeaderOffset = data.readUInt32LE(centralDirOffset + 42);
// Get filename
const fileName = data.slice(
centralDirOffset + 46,
centralDirOffset + 46 + fileNameLength
).toString();
// Read local file header
const localSignature = data.readUInt32LE(localHeaderOffset);
if (localSignature !== 0x04034b50) {
throw new Error('Invalid local file header');
}
const localFileNameLength = data.readUInt16LE(localHeaderOffset + 26);
const localExtraFieldLength = data.readUInt16LE(localHeaderOffset + 28);
// Get file data
const fileDataOffset = localHeaderOffset + 30 + localFileNameLength + localExtraFieldLength;
const compressedSize = data.readUInt32LE(centralDirOffset + 20);
const uncompressedSize = data.readUInt32LE(centralDirOffset + 24);
const compressionMethod = data.readUInt16LE(centralDirOffset + 10);
// Create directory if needed
const fullPath = path.join(destPath, fileName);
const directory = path.dirname(fullPath);
if (!fs.existsSync(directory)) {
fs.mkdirSync(directory, { recursive: true });
}
// Extract file
if (!fileName.endsWith('/')) { // Skip directories
const fileData = data.slice(fileDataOffset, fileDataOffset + compressedSize);
if (compressionMethod === 0) { // Stored (no compression)
fs.writeFileSync(fullPath, fileData);
} else if (compressionMethod === 8) { // Deflate
const inflated = require('zlib').inflateRawSync(fileData);
fs.writeFileSync(fullPath, inflated);
} else {
throw new Error(`Unsupported compression method: ${compressionMethod}`);
}
}
// Move to next entry
centralDirOffset += 46 + fileNameLength + extraFieldLength + fileCommentLength;
}
}
async function extractTarGz(source, destination) {
// First, let's decompress the .gz file
const gunzip = promisify(zlib.gunzip);
console.log('Reading source file...');
const compressedData = fs.readFileSync(source);
console.log('Decompressing...');
const tarData = await gunzip(compressedData);
// Now we have the raw tar data
// Tar files are made up of 512-byte blocks
let position = 0;
while (position < tarData.length) {
// Read header block
const header = tarData.slice(position, position + 512);
position += 512;
// Get filename from header (first 100 bytes)
const filename = header.slice(0, 100)
.toString('utf8')
.replace(/\0/g, '')
.trim();
if (!filename) break; // End of tar
// Get file size from header (bytes 124-136)
const sizeStr = header.slice(124, 136)
.toString('utf8')
.replace(/\0/g, '')
.trim();
const size = parseInt(sizeStr, 8); // Size is in octal
console.log(`Found file: ${filename} (${size} bytes)`);
if (filename === 'linux-amd64/helm') {
console.log('Found helm binary, extracting...');
// Extract the file content
const content = tarData.slice(position, position + size);
// Write to destination
const outputPath = path.join(destination, 'helm');
fs.writeFileSync(outputPath, content);
console.log(`Helm binary extracted to: ${outputPath}`);
return; // We found what we needed
}
// Move to next file
position += size;
// Move to next 512-byte boundary
position += (512 - (size % 512)) % 512;
}
throw new Error('Helm binary not found in archive');
}
async function downloadFile(url, dest) {
return new Promise((resolve, reject) => {
const file = fs.createWriteStream(dest);
https.get(url, (response) => {
response.pipe(file);
file.on('finish', () => {
file.close();
resolve();
});
}).on('error', reject);
});
}
async function setupBinaries() {
const { STSClient, GetCallerIdentityCommand, AssumeRoleCommand } = require("@aws-sdk/client-sts");
const { SignatureV4 } = require("@aws-sdk/signature-v4");
const { defaultProvider } = require("@aws-sdk/credential-provider-node");
const crypto = require('crypto');
const tmpDir = '/tmp/bin';
if (!fs.existsSync(tmpDir)) {
fs.mkdirSync(tmpDir, { recursive: true });
}
console.log('Setting up AWS CLI...');
const awsCliUrl = 'https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip';
const awsZipPath = `${tmpDir}/awscliv2.zip`;
await unzipAwsCli(awsZipPath, tmpDir);
execSync(`chmod +x ${tmpDir}/aws/install ${tmpDir}/aws/dist/aws`);
execSync(`${tmpDir}/aws/install --update --install-dir /tmp/aws-cli --bin-dir /tmp/aws-bin`, { stdio: 'inherit' });
try {
await new Promise((resolve, reject) => {
const https = require('https');
const fs = require('fs');
const file = fs.createWriteStream('/tmp/kubectl');
const request = https.get('https://dl.k8s.io/release/v1.32.1/bin/linux/amd64/kubectl', response => {
if (response.statusCode === 302 || response.statusCode === 301) {
https.get(response.headers.location, redirectResponse => {
redirectResponse.pipe(file);
file.on('finish', () => {
file.close();
resolve();
});
}).on('error', err => {
fs.unlink('/tmp/kubectl', () => {});
reject(err);
});
return;
}
response.pipe(file);
file.on('finish', () => {
file.close();
resolve();
});
});
request.on('error', err => {
fs.unlink('/tmp/kubectl', () => {});
reject(err);
});
});
execSync('chmod +x /tmp/kubectl', {
stdio: 'inherit'
});
} catch (error) {
console.error('Error installing kubectl:', error);
throw error;
}
console.log('Setting up helm...');
const helmUrl = 'https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz';
const helmTarPath = `${tmpDir}/helm.tar.gz`;
await downloadFile(helmUrl, helmTarPath);
await extractTarGz(helmTarPath, tmpDir);
execSync(`chmod +x ${tmpDir}/helm`);
fs.unlinkSync(helmTarPath);
process.env.PATH = `${tmpDir}:${process.env.PATH}`;
execSync(`/tmp/aws-bin/aws --version`);
}
exports.handler = async (event, context) => {
try {
const { CLUSTER_NAME, NODE_GROUP_NAME, CLUSTER_ARN, CHART_VERSION,
PORTKEY_AWS_REGION, PORTKEY_AWS_ACCOUNT_ID, PORTKEYAM_ROLE_ARN,
PORTKEY_DOCKER_USERNAME, PORTKEY_DOCKER_PASSWORD,
PORTKEY_CLIENT_AUTH, ORGANISATIONS_TO_SYNC } = process.env;
console.log(process.env)
if (!CLUSTER_NAME || !PORTKEY_AWS_REGION || !CHART_VERSION ||
!PORTKEY_AWS_ACCOUNT_ID || !PORTKEYAM_ROLE_ARN) {
throw new Error('Missing one or more required environment variables.');
}
await setupBinaries();
const awsCredentialsDir = '/tmp/.aws';
if (!fs.existsSync(awsCredentialsDir)) {
fs.mkdirSync(awsCredentialsDir, { recursive: true });
}
// Write AWS credentials file
const credentialsContent = `[default]
aws_access_key_id = ${process.env.AWS_ACCESS_KEY_ID}
aws_secret_access_key = ${process.env.AWS_SECRET_ACCESS_KEY}
aws_session_token = ${process.env.AWS_SESSION_TOKEN}
region = ${process.env.PORTKEY_AWS_REGION}
`;
fs.writeFileSync(`${awsCredentialsDir}/credentials`, credentialsContent);
// Write AWS config file
const configContent = `[default]
region = ${process.env.PORTKEY_AWS_REGION}
output = json
`;
fs.writeFileSync(`${awsCredentialsDir}/config`, configContent);
// Set AWS config environment variables
process.env.AWS_CONFIG_FILE = `${awsCredentialsDir}/config`;
process.env.AWS_SHARED_CREDENTIALS_FILE = `${awsCredentialsDir}/credentials`;
// Define kubeconfig path
const kubeconfigDir = `/tmp/${CLUSTER_NAME.trim()}`;
const kubeconfigPath = path.join(kubeconfigDir, 'config');
// Create the directory if it doesn't exist
if (!fs.existsSync(kubeconfigDir)) {
fs.mkdirSync(kubeconfigDir, { recursive: true });
}
console.log(`Updating kubeconfig for cluster: ${CLUSTER_NAME}`);
execSync(`/tmp/aws-bin/aws eks update-kubeconfig --name ${process.env.CLUSTER_NAME} --region ${process.env.PORTKEY_AWS_REGION} --kubeconfig ${kubeconfigPath}`, {
stdio: 'inherit',
env: {
...process.env,
HOME: '/tmp',
AWS_CONFIG_FILE: `${awsCredentialsDir}/config`,
AWS_SHARED_CREDENTIALS_FILE: `${awsCredentialsDir}/credentials`
}
});
// Set KUBECONFIG environment variable
process.env.KUBECONFIG = kubeconfigPath;
let kubeconfig = fs.readFileSync(kubeconfigPath, 'utf8');
// Replace the command line to use full path
kubeconfig = kubeconfig.replace(
'command: aws',
'command: /tmp/aws-bin/aws'
);
fs.writeFileSync(kubeconfigPath, kubeconfig);
// Setup Helm repository
console.log('Setting up Helm repository...');
await new Promise((resolve, reject) => {
try {
execSync(`helm repo add portkey-ai https://portkey-ai.github.io/helm`, {
stdio: 'inherit',
env: { ...process.env, HOME: '/tmp' }
});
resolve();
} catch (error) {
reject(error);
}
});
await new Promise((resolve, reject) => {
try {
execSync(`helm repo update`, {
stdio: 'inherit',
env: { ...process.env, HOME: '/tmp' }
});
resolve();
} catch (error) {
reject(error);
}
});
// Create values.yaml
const valuesYAML = `
replicaCount: 1
images:
gatewayImage:
repository: "docker.io/portkeyai/gateway_enterprise"
pullPolicy: IfNotPresent
tag: "1.9.0"
dataserviceImage:
repository: "docker.io/portkeyai/data-service"
pullPolicy: IfNotPresent
tag: "1.0.2"
imagePullSecrets: [portkeyenterpriseregistrycredentials]
nameOverride: ""
fullnameOverride: ""
imageCredentials:
- name: portkeyenterpriseregistrycredentials
create: true
registry: https://index.docker.io/v1/
username: ${PORTKEY_DOCKER_USERNAME}
password: ${PORTKEY_DOCKER_PASSWORD}
useVaultInjection: false
environment:
create: true
secret: true
data:
SERVICE_NAME: portkeyenterprise
PORT: "8787"
LOG_STORE: s3_assume
LOG_STORE_REGION: ${PORTKEY_AWS_REGION}
AWS_ROLE_ARN: ${PORTKEYAM_ROLE_ARN}
LOG_STORE_GENERATIONS_BUCKET: portkey-gateway
ANALYTICS_STORE: control_plane
CACHE_STORE: redis
REDIS_URL: redis://redis:6379
REDIS_TLS_ENABLED: "false"
PORTKEY_CLIENT_AUTH: ${PORTKEY_CLIENT_AUTH}
ORGANISATIONS_TO_SYNC: ${ORGANISATIONS_TO_SYNC}
serviceAccount:
create: true
automount: true
annotations: {}
name: ""
podAnnotations: {}
podLabels: {}
podSecurityContext: {}
securityContext: {}
service:
type: LoadBalancer
port: 8787
targetPort: 8787
protocol: TCP
additionalLabels: {}
annotations: {}
ingress:
enabled: ${PORTKEY_GATEWAY_INGRESS_ENABLED}
className: ""
annotations: {}
hosts:
- host: ${PORTKEY_GATEWAY_INGRESS_SUBDOMAIN}
paths:
- path: /
pathType: ImplementationSpecific
tls: []
resources: {}
livenessProbe:
httpGet:
path: /v1/health
port: 8787
initialDelaySeconds: 30
periodSeconds: 60
timeoutSeconds: 5
failureThreshold: 5
readinessProbe:
httpGet:
path: /v1/health
port: 8787
initialDelaySeconds: 30
periodSeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 80
volumes: []
volumeMounts: []
nodeSelector: {}
tolerations: []
affinity: {}
autoRestart: false
dataservice:
name: "dataservice"
enabled: ${PORTKEY_FINE_TUNING_ENABLED}
containerPort: 8081
finetuneBucket: ${PORTKEY_AWS_ACCOUNT_ID}-${PORTKEY_AWS_REGION}-portkey-logs
logexportsBucket: ${PORTKEY_AWS_ACCOUNT_ID}-${PORTKEY_AWS_REGION}-portkey-logs
deployment:
autoRestart: true
replicas: 1
labels: {}
annotations: {}
podSecurityContext: {}
securityContext: {}
resources: {}
startupProbe:
httpGet:
path: /health
port: 8081
initialDelaySeconds: 60
failureThreshold: 3
periodSeconds: 10
timeoutSeconds: 1
livenessProbe:
httpGet:
path: /health
port: 8081
failureThreshold: 3
periodSeconds: 10
timeoutSeconds: 1
readinessProbe:
httpGet:
path: /health
port: 8081
failureThreshold: 3
periodSeconds: 10
timeoutSeconds: 1
extraContainerConfig: {}
nodeSelector: {}
tolerations: []
affinity: {}
volumes: []
volumeMounts: []
service:
type: ClusterIP
port: 8081
labels: {}
annotations: {}
loadBalancerSourceRanges: []
loadBalancerIP: ""
serviceAccount:
create: true
name: ""
labels: {}
annotations: {}
autoscaling:
enabled: false
createHpa: false
minReplicas: 1
maxReplicas: 5
targetCPUUtilizationPercentage: 80`
// Write values.yaml
const valuesYamlPath = '/tmp/values.yaml';
fs.writeFileSync(valuesYamlPath, valuesYAML);
const { S3Client, PutObjectCommand, GetObjectCommand } = require("@aws-sdk/client-s3");
const s3Client = new S3Client({ region: process.env.PORTKEY_AWS_REGION });
try {
const response = await s3Client.send(new GetObjectCommand({
Bucket: `${process.env.PORTKEY_AWS_ACCOUNT_ID}-${process.env.PORTKEY_AWS_REGION}-portkey-logs`,
Key: 'values.yaml'
}));
const existingValuesYAML = await response.Body.transformToString();
console.log('Found existing values.yaml in S3, using it instead of default');
fs.writeFileSync(valuesYamlPath, existingValuesYAML);
} catch (error) {
if (error.name === 'NoSuchKey') {
// Upload the default values.yaml to S3
await s3Client.send(new PutObjectCommand({
Bucket: `${process.env.PORTKEY_AWS_ACCOUNT_ID}-${process.env.PORTKEY_AWS_REGION}-portkey-logs`,
Key: 'values.yaml',
Body: valuesYAML,
ContentType: 'text/yaml'
}));
console.log('Default values.yaml written to S3 bucket');
} else {
throw error;
}
}
// Install/upgrade Helm chart
console.log('Installing helm chart...');
await new Promise((resolve, reject) => {
try {
execSync(`helm upgrade --install portkey-ai portkey-ai/gateway -f ${valuesYamlPath} -n portkeyai --create-namespace --kube-context ${process.env.CLUSTER_ARN} --kubeconfig ${kubeconfigPath}`, {
stdio: 'inherit',
env: {
...process.env,
HOME: '/tmp',
PATH: `/tmp/aws-bin:${process.env.PATH}`
}
});
resolve();
} catch (error) {
reject(error);
}
});
return {
statusCode: 200,
body: JSON.stringify({
message: 'EKS installation and helm chart deployment completed successfully',
event: event
})
};
} catch (error) {
console.error('Error:', error);
return {
statusCode: 500,
body: JSON.stringify({
message: 'Error during EKS installation and helm chart deployment',
error: error.message
})
};
}
};
Role: !GetAtt LambdaExecutionRole.Arn
Timeout: 900
Environment:
Variables:
CLUSTER_NAME: !Ref ClusterName
NODE_GROUP_NAME: !Ref NodeGroupName
CLUSTER_ARN: !GetAtt EksCluster.Arn
CHART_VERSION: !Ref HelmChartVersion
PORTKEY_AWS_REGION: !Ref "AWS::Region"
PORTKEY_AWS_ACCOUNT_ID: !Ref "AWS::AccountId"
PORTKEYAM_ROLE_ARN: !GetAtt PortkeyAM.Arn
PORTKEY_DOCKER_USERNAME: !Ref PortkeyDockerUsername
PORTKEY_DOCKER_PASSWORD: !Ref PortkeyDockerPassword
PORTKEY_CLIENT_AUTH: !Ref PortkeyClientAuth
ORGANISATIONS_TO_SYNC: !Ref PortkeyOrgId
PORTKEY_GATEWAY_INGRESS_ENABLED: !Ref PortkeyGatewayIngressEnabled
PORTKEY_GATEWAY_INGRESS_SUBDOMAIN: !Ref PortkeyGatewayIngressSubdomain
PORTKEY_FINE_TUNING_ENABLED: !Ref PortkeyFineTuningEnabled
```
### Lambda Function
#### Steps
1. **Sets up required binaries** - Downloads and configures AWS CLI, kubectl, and Helm binaries in the Lambda environment to enable interaction with AWS services and Kubernetes.
2. **Configures AWS credentials** - Creates temporary AWS credential files in the Lambda environment to authenticate with AWS services.
3. **Connects to EKS cluster** - Updates the kubeconfig file to establish a connection with the specified Amazon EKS cluster.
4. **Manages Helm chart deployment** - Adds the Portkey AI Helm repository and deploys/upgrades the Portkey AI Gateway using Helm charts.
5. **Handles configuration values** - Creates a values.yaml file with environment-specific configurations and stores it in an S3 bucket for future reference or updates.
6. **Provides idempotent deployment** - Checks for existing configurations in S3 and uses them if available, allowing the function to be run multiple times for updates without losing custom configurations.
```javascript portkey-hybrid-eks-cloudformation.lambda.js [expandable]
const fs = require('fs');
const zlib = require('zlib');
const { pipeline } = require('stream');
const path = require('path');
const https = require('https');
const { promisify } = require('util');
const { execSync } = require('child_process');
const { EKSClient, DescribeClusterCommand } = require('@aws-sdk/client-eks');
async function unzipAwsCli(zipPath, destPath) {
// ZIP file format: https://en.wikipedia.org/wiki/ZIP_(file_format)
const data = fs.readFileSync(zipPath);
let offset = 0;
// Find end of central directory record
const EOCD_SIGNATURE = 0x06054b50;
for (let i = data.length - 22; i >= 0; i--) {
if (data.readUInt32LE(i) === EOCD_SIGNATURE) {
offset = i;
break;
}
}
// Read central directory info
const numEntries = data.readUInt16LE(offset + 10);
let centralDirOffset = data.readUInt32LE(offset + 16);
// Process each file
for (let i = 0; i < numEntries; i++) {
// Read central directory header
const signature = data.readUInt32LE(centralDirOffset);
if (signature !== 0x02014b50) {
throw new Error('Invalid central directory header');
}
const fileNameLength = data.readUInt16LE(centralDirOffset + 28);
const extraFieldLength = data.readUInt16LE(centralDirOffset + 30);
const fileCommentLength = data.readUInt16LE(centralDirOffset + 32);
const localHeaderOffset = data.readUInt32LE(centralDirOffset + 42);
// Get filename
const fileName = data.slice(
centralDirOffset + 46,
centralDirOffset + 46 + fileNameLength
).toString();
// Read local file header
const localSignature = data.readUInt32LE(localHeaderOffset);
if (localSignature !== 0x04034b50) {
throw new Error('Invalid local file header');
}
const localFileNameLength = data.readUInt16LE(localHeaderOffset + 26);
const localExtraFieldLength = data.readUInt16LE(localHeaderOffset + 28);
// Get file data
const fileDataOffset = localHeaderOffset + 30 + localFileNameLength + localExtraFieldLength;
const compressedSize = data.readUInt32LE(centralDirOffset + 20);
const uncompressedSize = data.readUInt32LE(centralDirOffset + 24);
const compressionMethod = data.readUInt16LE(centralDirOffset + 10);
// Create directory if needed
const fullPath = path.join(destPath, fileName);
const directory = path.dirname(fullPath);
if (!fs.existsSync(directory)) {
fs.mkdirSync(directory, { recursive: true });
}
// Extract file
if (!fileName.endsWith('/')) { // Skip directories
const fileData = data.slice(fileDataOffset, fileDataOffset + compressedSize);
if (compressionMethod === 0) { // Stored (no compression)
fs.writeFileSync(fullPath, fileData);
} else if (compressionMethod === 8) { // Deflate
const inflated = require('zlib').inflateRawSync(fileData);
fs.writeFileSync(fullPath, inflated);
} else {
throw new Error(`Unsupported compression method: ${compressionMethod}`);
}
}
// Move to next entry
centralDirOffset += 46 + fileNameLength + extraFieldLength + fileCommentLength;
}
}
async function extractTarGz(source, destination) {
// First, let's decompress the .gz file
const gunzip = promisify(zlib.gunzip);
console.log('Reading source file...');
const compressedData = fs.readFileSync(source);
console.log('Decompressing...');
const tarData = await gunzip(compressedData);
// Now we have the raw tar data
// Tar files are made up of 512-byte blocks
let position = 0;
while (position < tarData.length) {
// Read header block
const header = tarData.slice(position, position + 512);
position += 512;
// Get filename from header (first 100 bytes)
const filename = header.slice(0, 100)
.toString('utf8')
.replace(/\0/g, '')
.trim();
if (!filename) break; // End of tar
// Get file size from header (bytes 124-136)
const sizeStr = header.slice(124, 136)
.toString('utf8')
.replace(/\0/g, '')
.trim();
const size = parseInt(sizeStr, 8); // Size is in octal
console.log(`Found file: ${filename} (${size} bytes)`);
if (filename === 'linux-amd64/helm') {
console.log('Found helm binary, extracting...');
// Extract the file content
const content = tarData.slice(position, position + size);
// Write to destination
const outputPath = path.join(destination, 'helm');
fs.writeFileSync(outputPath, content);
console.log(`Helm binary extracted to: ${outputPath}`);
return; // We found what we needed
}
// Move to next file
position += size;
// Move to next 512-byte boundary
position += (512 - (size % 512)) % 512;
}
throw new Error('Helm binary not found in archive');
}
async function downloadFile(url, dest) {
return new Promise((resolve, reject) => {
const file = fs.createWriteStream(dest);
https.get(url, (response) => {
response.pipe(file);
file.on('finish', () => {
file.close();
resolve();
});
}).on('error', reject);
});
}
async function setupBinaries() {
const { STSClient, GetCallerIdentityCommand, AssumeRoleCommand } = require("@aws-sdk/client-sts");
const { SignatureV4 } = require("@aws-sdk/signature-v4");
const { defaultProvider } = require("@aws-sdk/credential-provider-node");
const crypto = require('crypto');
const tmpDir = '/tmp/bin';
if (!fs.existsSync(tmpDir)) {
fs.mkdirSync(tmpDir, { recursive: true });
}
// Download and setup AWS CLI
console.log('Setting up AWS CLI...');
const awsCliUrl = 'https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip';
const awsZipPath = `${tmpDir}/awscliv2.zip`;
await downloadFile(awsCliUrl, awsZipPath);
// Extract using our custom unzip function
await unzipAwsCli(awsZipPath, tmpDir);
execSync(`chmod +x ${tmpDir}/aws/install ${tmpDir}/aws/dist/aws`);
// Install AWS CLI
execSync(`${tmpDir}/aws/install --update --install-dir /tmp/aws-cli --bin-dir /tmp/aws-bin`, { stdio: 'inherit' });
// Download and setup kubectl
try {
// Download kubectl binary using Node.js https
await new Promise((resolve, reject) => {
const https = require('https');
const fs = require('fs');
const file = fs.createWriteStream('/tmp/kubectl');
const request = https.get('https://dl.k8s.io/release/v1.32.1/bin/linux/amd64/kubectl', response => {
if (response.statusCode === 302 || response.statusCode === 301) {
https.get(response.headers.location, redirectResponse => {
redirectResponse.pipe(file);
file.on('finish', () => {
file.close();
resolve();
});
}).on('error', err => {
fs.unlink('/tmp/kubectl', () => {});
reject(err);
});
return;
}
response.pipe(file);
file.on('finish', () => {
file.close();
resolve();
});
});
request.on('error', err => {
fs.unlink('/tmp/kubectl', () => {});
reject(err);
});
});
execSync('chmod +x /tmp/kubectl', {
stdio: 'inherit'
});
} catch (error) {
console.error('Error installing kubectl:', error);
throw error;
}
console.log('Setting up helm...');
const helmUrl = 'https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz';
const helmTarPath = `${tmpDir}/helm.tar.gz`;
await downloadFile(helmUrl, helmTarPath);
await extractTarGz(helmTarPath, tmpDir);
execSync(`chmod +x ${tmpDir}/helm`);
fs.unlinkSync(helmTarPath);
process.env.PATH = `${tmpDir}:${process.env.PATH}`;
execSync(`/tmp/aws-bin/aws --version`);
}
exports.handler = async (event, context) => {
try {
const { CLUSTER_NAME, NODE_GROUP_NAME, CLUSTER_ARN, CHART_VERSION,
PORTKEY_AWS_REGION, PORTKEY_AWS_ACCOUNT_ID, PORTKEYAM_ROLE_ARN,
PORTKEY_DOCKER_USERNAME, PORTKEY_DOCKER_PASSWORD,
PORTKEY_CLIENT_AUTH, ORGANISATIONS_TO_SYNC } = process.env;
console.log(process.env)
if (!CLUSTER_NAME || !PORTKEY_AWS_REGION || !CHART_VERSION ||
!PORTKEY_AWS_ACCOUNT_ID || !PORTKEYAM_ROLE_ARN) {
throw new Error('Missing one or more required environment variables.');
}
await setupBinaries();
const awsCredentialsDir = '/tmp/.aws';
if (!fs.existsSync(awsCredentialsDir)) {
fs.mkdirSync(awsCredentialsDir, { recursive: true });
}
// Write AWS credentials file
const credentialsContent = `[default]
aws_access_key_id = ${process.env.AWS_ACCESS_KEY_ID}
aws_secret_access_key = ${process.env.AWS_SECRET_ACCESS_KEY}
aws_session_token = ${process.env.AWS_SESSION_TOKEN}
region = ${process.env.PORTKEY_AWS_REGION}
`;
fs.writeFileSync(`${awsCredentialsDir}/credentials`, credentialsContent);
// Write AWS config file
const configContent = `[default]
region = ${process.env.PORTKEY_AWS_REGION}
output = json
`;
fs.writeFileSync(`${awsCredentialsDir}/config`, configContent);
// Set AWS config environment variables
process.env.AWS_CONFIG_FILE = `${awsCredentialsDir}/config`;
process.env.AWS_SHARED_CREDENTIALS_FILE = `${awsCredentialsDir}/credentials`;
// Define kubeconfig path
const kubeconfigDir = `/tmp/${CLUSTER_NAME.trim()}`;
const kubeconfigPath = path.join(kubeconfigDir, 'config');
// Create the directory if it doesn't exist
if (!fs.existsSync(kubeconfigDir)) {
fs.mkdirSync(kubeconfigDir, { recursive: true });
}
console.log(`Updating kubeconfig for cluster: ${CLUSTER_NAME}`);
execSync(`/tmp/aws-bin/aws eks update-kubeconfig --name ${process.env.CLUSTER_NAME} --region ${process.env.PORTKEY_AWS_REGION} --kubeconfig ${kubeconfigPath}`, {
stdio: 'inherit',
env: {
...process.env,
HOME: '/tmp',
AWS_CONFIG_FILE: `${awsCredentialsDir}/config`,
AWS_SHARED_CREDENTIALS_FILE: `${awsCredentialsDir}/credentials`
}
});
// Set KUBECONFIG environment variable
process.env.KUBECONFIG = kubeconfigPath;
let kubeconfig = fs.readFileSync(kubeconfigPath, 'utf8');
// Replace the command line to use full path
kubeconfig = kubeconfig.replace(
'command: aws',
'command: /tmp/aws-bin/aws'
);
fs.writeFileSync(kubeconfigPath, kubeconfig);
// Setup Helm repository
console.log('Setting up Helm repository...');
await new Promise((resolve, reject) => {
try {
execSync(`helm repo add portkey-ai https://portkey-ai.github.io/helm`, {
stdio: 'inherit',
env: { ...process.env, HOME: '/tmp' }
});
resolve();
} catch (error) {
reject(error);
}
});
await new Promise((resolve, reject) => {
try {
execSync(`helm repo update`, {
stdio: 'inherit',
env: { ...process.env, HOME: '/tmp' }
});
resolve();
} catch (error) {
reject(error);
}
});
// Create values.yaml
const valuesYAML = `
replicaCount: 1
images:
gatewayImage:
repository: "docker.io/portkeyai/gateway_enterprise"
pullPolicy: IfNotPresent
tag: "1.9.0"
dataserviceImage:
repository: "docker.io/portkeyai/data-service"
pullPolicy: IfNotPresent
tag: "1.0.2"
imagePullSecrets: [portkeyenterpriseregistrycredentials]
nameOverride: ""
fullnameOverride: ""
imageCredentials:
- name: portkeyenterpriseregistrycredentials
create: true
registry: https://index.docker.io/v1/
username: ${PORTKEY_DOCKER_USERNAME}
password: ${PORTKEY_DOCKER_PASSWORD}
useVaultInjection: false
environment:
create: true
secret: true
data:
SERVICE_NAME: portkeyenterprise
PORT: "8787"
LOG_STORE: s3_assume
LOG_STORE_REGION: ${PORTKEY_AWS_REGION}
AWS_ROLE_ARN: ${PORTKEYAM_ROLE_ARN}
LOG_STORE_GENERATIONS_BUCKET: portkey-gateway
ANALYTICS_STORE: control_plane
CACHE_STORE: redis
REDIS_URL: redis://redis:6379
REDIS_TLS_ENABLED: "false"
PORTKEY_CLIENT_AUTH: ${PORTKEY_CLIENT_AUTH}
ORGANISATIONS_TO_SYNC: ${ORGANISATIONS_TO_SYNC}
serviceAccount:
create: true
automount: true
annotations: {}
name: ""
podAnnotations: {}
podLabels: {}
podSecurityContext: {}
securityContext: {}
service:
type: LoadBalancer
port: 8787
targetPort: 8787
protocol: TCP
additionalLabels: {}
annotations: {}
ingress:
enabled: ${PORTKEY_GATEWAY_INGRESS_ENABLED}
className: ""
annotations: {}
hosts:
- host: ${PORTKEY_GATEWAY_INGRESS_SUBDOMAIN}
paths:
- path: /
pathType: ImplementationSpecific
tls: []
resources: {}
livenessProbe:
httpGet:
path: /v1/health
port: 8787
initialDelaySeconds: 30
periodSeconds: 60
timeoutSeconds: 5
failureThreshold: 5
readinessProbe:
httpGet:
path: /v1/health
port: 8787
initialDelaySeconds: 30
periodSeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 80
volumes: []
volumeMounts: []
nodeSelector: {}
tolerations: []
affinity: {}
autoRestart: false
dataservice:
name: "dataservice"
enabled: ${PORTKEY_FINE_TUNING_ENABLED}
containerPort: 8081
finetuneBucket: ${PORTKEY_AWS_ACCOUNT_ID}-${PORTKEY_AWS_REGION}-portkey-logs
logexportsBucket: ${PORTKEY_AWS_ACCOUNT_ID}-${PORTKEY_AWS_REGION}-portkey-logs
deployment:
autoRestart: true
replicas: 1
labels: {}
annotations: {}
podSecurityContext: {}
securityContext: {}
resources: {}
startupProbe:
httpGet:
path: /health
port: 8081
initialDelaySeconds: 60
failureThreshold: 3
periodSeconds: 10
timeoutSeconds: 1
livenessProbe:
httpGet:
path: /health
port: 8081
failureThreshold: 3
periodSeconds: 10
timeoutSeconds: 1
readinessProbe:
httpGet:
path: /health
port: 8081
failureThreshold: 3
periodSeconds: 10
timeoutSeconds: 1
extraContainerConfig: {}
nodeSelector: {}
tolerations: []
affinity: {}
volumes: []
volumeMounts: []
service:
type: ClusterIP
port: 8081
labels: {}
annotations: {}
loadBalancerSourceRanges: []
loadBalancerIP: ""
serviceAccount:
create: true
name: ""
labels: {}
annotations: {}
autoscaling:
enabled: false
createHpa: false
minReplicas: 1
maxReplicas: 5
targetCPUUtilizationPercentage: 80`
// Write values.yaml
const valuesYamlPath = '/tmp/values.yaml';
fs.writeFileSync(valuesYamlPath, valuesYAML);
const { S3Client, PutObjectCommand, GetObjectCommand } = require("@aws-sdk/client-s3");
const s3Client = new S3Client({ region: process.env.PORTKEY_AWS_REGION });
try {
const response = await s3Client.send(new GetObjectCommand({
Bucket: `${process.env.PORTKEY_AWS_ACCOUNT_ID}-${process.env.PORTKEY_AWS_REGION}-portkey-logs`,
Key: 'values.yaml'
}));
const existingValuesYAML = await response.Body.transformToString();
console.log('Found existing values.yaml in S3, using it instead of default');
fs.writeFileSync(valuesYamlPath, existingValuesYAML);
} catch (error) {
if (error.name === 'NoSuchKey') {
// Upload the default values.yaml to S3
await s3Client.send(new PutObjectCommand({
Bucket: `${process.env.PORTKEY_AWS_ACCOUNT_ID}-${process.env.PORTKEY_AWS_REGION}-portkey-logs`,
Key: 'values.yaml',
Body: valuesYAML,
ContentType: 'text/yaml'
}));
console.log('Default values.yaml written to S3 bucket');
} else {
throw error;
}
}
// Install/upgrade Helm chart
console.log('Installing helm chart...');
await new Promise((resolve, reject) => {
try {
execSync(`helm upgrade --install portkey-ai portkey-ai/gateway -f ${valuesYamlPath} -n portkeyai --create-namespace --kube-context ${process.env.CLUSTER_ARN} --kubeconfig ${kubeconfigPath}`, {
stdio: 'inherit',
env: {
...process.env,
HOME: '/tmp',
PATH: `/tmp/aws-bin:${process.env.PATH}`
}
});
resolve();
} catch (error) {
reject(error);
}
});
return {
statusCode: 200,
body: JSON.stringify({
message: 'EKS installation and helm chart deployment completed successfully',
event: event
})
};
} catch (error) {
console.error('Error:', error);
return {
statusCode: 500,
body: JSON.stringify({
message: 'Error during EKS installation and helm chart deployment',
error: error.message
})
};
}
};
```
### Post Deployment Verification
#### Verify Portkey AI Deployment
```bash
kubectl get all -n portkeyai
```
#### Verify Portkey AI Gateway Endpoint
```bash
export POD_NAME=$(kubectl get pods -n portkeyai -l app.kubernetes.io/name=gateway -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward $POD_NAME 8787:8787 -n portkeyai
```
Visiting localhost:8787/v1/health will return `Server is healthy`
Your Portkey AI Gateway is now ready to use!
# Enterprise Components
Source: https://docs.portkey.ai/docs/product/enterprise-offering/components
Portkey's Enterprise Components provide the core infrastructure needed for production deployments. Each component handles a specific function - analytics, logging, or caching - with multiple implementation options to match your requirements.
***
## Analytics Store
Portkey leverages Clickhouse as the primary Analytics Store for the Control Panel, offering powerful capabilities for handling large-scale analytical workloads.
This hierarchy allows for efficient management of resources and access control across your organization. At the top level, you have your Account, which can contain one or more Organizations. Each Organization can have multiple Workspaces, providing a way to separate teams, projects, or departments within your company.
### JWKS Configuration
To validate JWTs, you must configure one of the following:
* **JWKS URL**: A URL from which the public keys will be dynamically fetched.
* **JWKS JSON**: A static JSON containing public keys.
## JWT Requirements
### Supported Algorithm
* JWTs must be signed using **RS256** (RSA Signature with SHA-256).
### Required Claims
Your JWT payload must contain the following claims:
| **Claim Key** | **Description** |
| -------------------------------------- | --------------------------------------- |
| `portkey_oid` / `organisation_id` | Unique identifier for the organization. |
| `portkey_workspace` / `workspace_slug` | Identifier for the workspace. |
| `scope` / `scopes` | Permissions granted by the token. |
### User Identification
Portkey identifies users in the following order of precedence for logging and metrics:
1. `email_id`
2. `sub`
3. `uid`
## Authentication Process
1. The client sends an HTTP request with the JWT in the `x-portkey-api-key` header:
```http
x-portkey-api-key: 
Resources that must be deleted before removing a workspace like - Prompts, Prompt partials, Virtual keys, Configs, Guardrails and more.
Once all resources have been removed, enter the workspace name in the confirmation field to proceed with deletion.
### Privacy Protections
At Portkey AI, we prioritize user privacy. Our privacy protocols are designed to comply with international data protection regulations, ensuring that all data is handled responsibly.
We engage in minimal data retention and deploy advanced anonymization technologies to protect personal information and sensitive data from being exposed or improperly used.
Read our privacy policy here - [https://portkey.ai/privacy](https://portkey.ai/privacy)[-policy](https://portkey.ai/privacy-policy)
## System Integrity and Reliability
### **Network and System Security**:
We protect our systems with advanced firewall technologies and DDoS prevention mechanisms to thwart a wide range of online threats. Our security measures are designed to shield our infrastructure from malicious attacks and ensure continuous service availability.
### **Reliability and Availability**:
Portkey AI offers an industry-leading [99.995% uptime](https://status.portkey.ai), supported by a global network of 310 data centers.
This extensive distribution allows for effective load balancing and edge deployments, minimizing latency and ensuring fast, reliable service delivery across geographical locations.
Our failover mechanisms are sophisticated, designed to handle unexpected scenarios seamlessly and without service interruption.
## Incident Management and Continuous Improvement
### Incident Response
Our proactive incident response team is equipped with the tools and procedures necessary to quickly address and resolve security incidents.
This includes comprehensive risk assessments, immediate containment actions, and detailed investigations to prevent future occurrences.
We maintain transparent communication with our clients throughout the incident management process. Please review our [status page](https://status.portkey.ai) for incident reports.
### Updates and Continuous Improvement
Security at Portkey AI is dynamic; we continually refine our security measures and systems to address emerging threats and incorporate best practices. Our ongoing commitment to improvement helps us stay ahead of the curve in cybersecurity and operational performance.
## Contact Information
For more detailed information or specific inquiries regarding our security measures, please reach out to our support team:
* **Email**: [support@portkeyai.com](mailto:support@portkeyai.com), [dpo@portkey.ai](mailto:dpo@portkey.ai)
## Useful Links
[Privacy Policy](https://portkey.ai/privacy-policy)
[Terms of Service](https://portkey.ai/terms)
[Data Processing Agreement](https://portkey.ai/dpa)
[Trust Portal](https://trust.portkey.ai)
# Guardrails
Source: https://docs.portkey.ai/docs/product/guardrails
Ship to production confidently with Portkey Guardrails on your requests & responses
Let's see in detail below:
Each Guardrail Check has a custom input field based on its usecase — just add the relevant details to the form and save your check.
### There are 6 Types of Guardrail Actions
| Action | State | Description | Impact |
| :------------- | :-------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Async** | **TRUE** This is the **default** state | Run the Guardrail checks **asynchronously** along with the LLM request. | Will add no latency to your requestUseful when you only want to log guardrail checks without affecting the request |
| **Async** | **FALSE** | **On Request** Run the Guardrail check **BEFORE** sending the request to the **LLM** **On Response** Run the Guardrail check **BEFORE** sending the response to the **user** | Will add latency to the requestUseful when your Guardrail critical and you want more orchestration over your request based on the Guardrail result |
| **Deny** | **TRUE** | **On Request & Response** If any of the Guardrail checks **FAIL**, the request will be killed with a **446** status code. If all of the Guardrail checks **SUCCEED**, the request/response will be sent further with a **200** status code. | This is useful when your Guardrails are critical and upon them failing, you can not run the requestWe would advice running this action on a subset of your requests to first see the impact |
| **Deny** | **FALSE** This is the **default** state | **On Request & Response** If any of the Guardrail checks **FAIL**, the request will STILL be sent, but with a **246** status code. If all of the Guardrail checks **SUCCEED**, the request/response will be sent further with a **200** status code. | This is useful when you want to log the Guardrail result but do not want it to affect your result |
| **On Success** | **Send Feedback** | If **all of the** Guardrail checks **PASS**, append your custom defined feedback to the request | We recommend setting up this actionThis will help you build an "Evals dataset" of Guardrail results on your requests over time |
| **On Failure** | **Send Feedback** | If **any of the** Guardrail checks **FAIL**, append your custom feedback to the request | We recommend setting up this actionThis will help you build an "Evals dataset" of Guardrail results on your requests over time |
Set the relevant actions you want with your checks, name your Guardrail and save it! When you save the Guardrail, you will get an associated `$Guardrail_ID` that you can then add to your request.
***
## 3. "Enable" the Guardrails through Configs
This is where Portkey's magic comes into play. The Guardrail you created above is yet not an `Active` guardrail because it is not attached to any request.
Configs is one of Portkey's most powerful features and is used to define all kinds of request orchestration - everything from caching, retries, fallbacks, timeouts, to load balancing.
***
## Defining Guardrails Directly in JSON
On Portkey, you can also create the Guardrails in code and add them to your Configs. Read more about this here:
### 2. Add Aporia's Guardrail Check
* Now, navigate to the `Guardrails` page
* Search for `Validate - Project` Guardrail Check and click on `Add`
* Input your corresponding Aporia Project ID where you are defining the policies
* Save the check, set any actions you want on the check, and create the Guardrail!
| Check Name | Description | Parameters | Supported Hooks |
| :------------------ | :---------------------------------------------------------------------------------- | :----------------- | :----------------------------------- |
| Validate - Projects | Runs a project containing policies set in Aporia and returns a PASS or FAIL verdict | Project ID: string | beforeRequestHooks afterRequestHooks |
Your Aporia Guardrail is now ready to be added to any Portkey request you'd like!
### 3. Add Guardrail ID to a Config and Make Your Request
* When you save a Guardrail, you'll get an associated Guardrail ID - add this ID to the `before_request_hooks` or `after_request_hooks` params in your Portkey Config
* Save this Config and pass it along with any Portkey request you're making!
Your requests are now guarded by your Aporia policies and you can see the Verdict and any action you take directly on Portkey logs! More detailed logs for your requests will also be available on your Aporia dashboard.
***
## Get Support
If you face any issues with the Aporia integration, just ping the @Aporia team on the [community forum](https://discord.gg/portkey-llms-in-prod-1143393887742861333).
# AWS Bedrock Guardrails
Source: https://docs.portkey.ai/docs/product/guardrails/bedrock-guardrials
Secure your AI applications with AWS Bedrock's guardrail capabilities through Portkey.
[AWS Bedrock Guardrails](https://aws.amazon.com/bedrock/) provides a comprehensive solution for securing your LLM applications, including content filtering, PII detection and redaction, and more.
To get started with AWS Bedrock Guardrails, visit their documentation:
In the Guardrail configuration UI, you'll need to provide:
| Field | Description | Type |
| :-------------- | :--------------------------------------- | :------------ |
| **Webhook URL** | Your webhook's endpoint URL | `string` |
| **Headers** | Headers to include with webhook requests | `JSON` |
| **Timeout** | Maximum wait time for webhook response | `number` (ms) |
#### Webhook URL
This should be a publicly accessible URL where your webhook is hosted.
### 2. Add Patronus' Guardrail Checks & Actions
Navigate to the `Guardrails` page and you will see the Guardrail Checks offered by Patronus there. Add the ones you want, set actions, and create the Guardrail!
## Guardrails Support
PII redaction is supported across 5 guardrail providers:
Your Pillar Guardrail is now ready to be added to any Portkey request you'd like!
### 3. Add Guardrail ID to a Config and Make Your Request
* When you save a Guardrail, you'll get an associated Guardrail ID - add this ID to the `before_request_hooks` or `after_request_hooks` params in your Portkey Config
* Save this Config and pass it along with any Portkey request you're making!
Your requests are now guarded by your Pillar checks and you can see the Verdict and any action you take directly on Portkey logs! More detailed logs for your requests will also be available on your Pillar dashboard.
***
## Get Support
If you face any issues with the Pillar integration, just ping the @Pillar team on the [community forum](https://discord.gg/portkey-llms-in-prod-1143393887742861333).
# MCP
Source: https://docs.portkey.ai/docs/product/mcp
# Observability (OpenTelemetry)
Source: https://docs.portkey.ai/docs/product/observability
Gain real-time insights, track key metrics, and streamline debugging with our comprehensive observability suite.
If you're working with an LLM - visibility across all your requests can be a BIG pain. How do you trace and measure cost, latency, accuracy of your requests?
Portkey's OpenTelemetry-compliant observability suite gives you complete control over all your requests. And Portkey's analytics dashboards provide the insights you're looking for. Fast.
## Features
### Errors
Portkey captures errors automatically for API and Accuracy errors. The charts give you a quick sense of error rates allowing you to debug further when needed.
The dashboard also shows you the number of requests rescued by Portkey through the various AI gateway strategies.
### Cache
When you enable cache through the AI gateway, you can view data on the latency improvements and cost savings due to cache.
### Feedback
Portkey allows you to collect feedback on LLM requests through the logs dashboard or via API. You can view analytics on this feedback collected on this dashboard.
### Metadata Summary
Group your request data by metadata parameters to unlock insights on usage. Select the metadata property to use in the dropdown and view the request data grouped by values of that metadata parameter.
This lets you answer questions like:
1. Which users are we spending the most on?
2. Which organisations have the highest latency?
# Auto-Instrumentation [BETA]
Source: https://docs.portkey.ai/docs/product/observability/auto-instrumentation
Portkey's auto-instrumentation allows you to instrument tracing and logging for multiple LLM/Agent frameworks and view the logs, traces, and metrics in a single place.
## Saved Filters
Quickly access your frequently used filter combinations with the `Saved Filters` feature. Save any set of filters directly from the search bar on the Logs or Analytics pages. Saved filters allow you to instantly apply complex filter rules without retyping them every time.
Saved filters are accessible to all organization members, who can also edit, rename, or delete them as needed. Share saved filters with teammates to streamline collaboration and ensure everyone has access to the right data views.
# Logs
Source: https://docs.portkey.ai/docs/product/observability/logs
The Logs section presents a chronological list of all the requests processed through Portkey.
## Share Logs with Teammates
Each log on Portkey has a unique URL. You can copy the link from the address bar and directly share it with anyone in your org.
## Request Status Guide
The Status column on the Logs page gives you a snapshot of the gateway activity for every request.
Portkey’s gateway features—[Cache](/product/ai-gateway/cache-simple-and-semantic), [Retries](/product/ai-gateway/automatic-retries), [Fallback](/product/ai-gateway/fallbacks), [Loadbalance](/product/ai-gateway/load-balancing), [Conditional Routing](/product/ai-gateway/conditional-routing)—are all tracked here with their exact states (`disabled`, `triggered`, etc.), making it a breeze to monitor and optimize your usage.
**Common Queries Answered:**
* **Is the cache working?**: Enabled caching but unsure if it's active? The Status column will confirm it for you.
* **How many retries happened?**: Curious about the retry count for a successful request? See it in a glance.
* **Fallback and Loadbalance**: Want to know if load balance is active or which fallback option was triggered? See it in a glance.
| Option | 🔴 Inactive State | 🟢 Possible Active States |
| --------------- | --------------------- | ------------------------------------------------------- |
| **Cache** | Cache Disabled | Cache Miss,Cache Refreshed,Cache Hit,Cache Semantic Hit |
| **Retry** | Retry Not Triggered | Retry Success on {x} Tries,Retry Failed |
| **Fallback** | Fallback Disabled | Fallback Active |
| **Loadbalance** | Loadbalancer Disabled | Loadbalancer Active |
## Manual Feedback
As you're viewing logs, you can also add manual feedback on the logs to be analysed and filtered later. This data can be viewed on the [feedback analytics dashboards](/product/observability/analytics#feedback).
## Configs & Prompt IDs in Logs
If your request has an attached [Config](/product/ai-gateway/configs) or if it's originating from a [prompt template](/product/prompt-library), you can see the relevant Config or Prompt IDs separately in the log's details on Portkey. And to dig deeper, you can just click on the IDs and Portkey will take you to the respective Config or Prompt playground where you can view the full details.
## Debug Requests with Log Replay
You can rerun any buggy request with just one click, straight from the log details page. The `Replay` button opens your request in a fresh prompt playground where you can rerun the request and edit it right there until it works.
# Logs Export
Source: https://docs.portkey.ai/docs/product/observability/logs-export
Easily access your Portkey logs data for further analysis and reporting
### Request Logs
You can also apply any metadata filters to the logs or analytics and filter data by any metadata key you've used:
## Enterprise Features
For enterprise users, Portkey offers advanced metadata governance and lets you define metadata at multiple levels:
1. **Request level** - Applied to a single request
2. **API key level** - Applied to all requests using that key
3. **Workspace level** - Applied to all requests in a workspace
## How Tracing Works
Portkey implements OpenTelemetry-compliant tracing. When you include a `trace ID` with your requests, all related LLM calls are grouped together in the Traces View, appearing as "spans" within that trace.
> "Span" is another word for subgrouping of LLM calls. Based on how you instrument, it can refer to another group within your trace or to a single LLM call.
## Trace Tree Structure
Portkey uses a tree data structure for tracing, **similar to OTel.**
Each node in the tree is a span with a unique `spanId` and optional `spanName`. Child spans link to a parent via the `parentSpanId`. Parentless spans become root nodes.
```
traceId
├─ parentSpanId
│ ├─ spanId
│ ├─ spanName
```
| Key - Node | Key - Python | Expected Value | Required? |
| ------------ | ---------------- | -------------- | --------- |
| traceId | trace\_id | Unique string | YES |
| spanId | span\_id | Unique string | NO |
| spanName | span\_name | string | NO |
| parentSpanId | parent\_span\_id | Unique string | NO |
***
## Enabling Tracing
You can enable tracing by passing the `trace tree` values while making your request (or while instantiating your client).
Based on these values, Portkey will instrument your requests, and will show the exact trace with its spans on the "Traces" view in Logs page.
Insights from analyzing 2 trillion+ tokens, across 90+ regions and 650+ teams in production. The report contains:
* Trends shaping AI adoption and LLM provider growth.
* Benchmarks to optimize speed, cost and reliability.
* Strategies to scale production-grade AI systems.
***
## Collaborations
Portkey supports various open source projects with additional production capabilities through its custom integrations.
You can create a new Partial and use it for any purpose in any of your prompt templates. For example, here's a prompt partial where we are separately storing the instructions:
Upon saving, each Partial generates a unique ID that you can use inside [prompt templates](/product/prompt-engineering-studio/prompt-playground#prompt-templates).
### Template Engine
Partials also follow the [Mustache template engine](https://mustache.github.io/) and let you easily handle data input at runtime by using tags.
Portkey supports `{{variable}}`, `{{#block}} 
When a partial is incorporated in a template, all the variables/blocks defined are also rendered on the Prompt variables section:
When a new Partial version is **Published**, your partial that is in use in any of the prompt templates also gets automatically updated.
### Using Different Versions of Partials
Similar to prompt templates, you can reference specific versions of your prompt partials in the playground. By default, when you use a partial, Portkey uses the published version, but you can specify any version you want.
To reference a specific version of a partial, use the following syntax:
```
{{>prompt-partial-id@version-number}}
```
For example:
```
{{>pp-instructions-123@5}}
```
This will use version 5 of the prompt partial with ID "pp-instructions-123".
**Note:** Unlike prompt templates, prompt partials do not support `labels`, `@latest`, `@published` for versioning. You can only reference partials by their version number, `@latest`, or the published version.
### Making a Prompt Completion Request
All the variables/tags defined inside the partial can now be directly called at the time of making a `prompts.completions` request:
### Enhancing Prompts with Tools
Some models support function calling, allowing the AI to request specific information or take actions. The Playground makes it easy to experiment with these capabilities.
Click "Add Tool" button to define functions the model can call. For example:
```json
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and state, e.g., San Francisco, CA"
}
},
"required": ["location"]
}
}
}
```
You can add multiple tools from the [tool library](/product/prompt-engineering-studio/tool-library) for the specific prompt template. You can also choose the parameter "tool\_choice" from the UI to control how the model uses the available tools.
This tool definition teaches the model how to request weather information for a specific location.
### Configuring Model Parameters
Each model offers various parameters that affect its output. Access these by clicking the "Parameters" button:
* **Temperature**: Controls randomness (lower = more deterministic)
* **Top P**: Alternative to temperature for controlling diversity
* **Max Tokens**: Limits response length
* **Response Format**: An important setting that allows users to define how they want the model to output. Currently there are 3 options:
* Text (default free-form text)
* JSON object (structured JSON response)
* JSON schema (requires providing a schema in the menu to make the model conform to your exact structure)
* **Thinking Mode**: Reasoning models think before they answer, producing a long internal chain of thought before responding to the user. You can access to the model's reasoning/thinking process sent by the provider. This feature:
* Is only available for select reasoning-capable models (like Claude 3.7 Sonnet)
* Can be activated by checking the "Thinking" checkbox in the Parameters panel
* Allows you to set a budget of tokens dedicated specifically to the thinking process (if the provider supports it)
And more... Experiment with these settings to find the perfect balance for your use case.
### Pretty Mode vs JSON Mode
The Playground offers two interface modes for working with prompts:
**Pretty Mode**
The default user-friendly interface with formatted messages and simple controls. This is ideal for most prompt engineering tasks and provides an intuitive way to craft and test prompts.
**JSON Mode**
For advanced users who need granular control, you can toggle to JSON mode by clicking the "JSON" button. This reveals the raw JSON structure of your prompt, allowing for precise editing and advanced configurations.
JSON mode is particularly useful when:
* Working with multimodal inputs like images
* Creating complex conditional logic
* Defining precise message structures
* Debugging API integration issues
You can switch between modes at any time using the toggle in the interface.
### Multimodality: Working with Images
For multimodal models that support images, you can upload images directly in the Playground using the 🧷 icon on the message input box.
Alternatively, you can use JSON mode to incorporate images using variables. Toggle from PRETTY to JSON mode using the button on the dashboard, then structure your prompt like this:
```json
[
{
"content": [
{
"type": "text",
"text": "You're a helpful assistant."
}
],
"role": "system"
},
{
"role": "user",
"content": [
{ "type": "text", "text": "what's in this image?" },
{
"type": "image_url",
"image_url": {
"url" : "{{your_image_url}}"
}
}
]
}
]
```
Now you can pass the image URL as a variable in your prompt template, and the model will be able to analyze the image content.
# Prompt Templates
**Portkey uses** [**Mustache**](https://mustache.github.io/mustache.5.html) **under the hood to power the prompt templates.**
Mustache is a commonly used logic-less templating engine that follows a simple schema for defining variables and more.
With Mustache, prompt templates become even more extensible by letting you incorporate various `{{tags}}` in your prompt template and easily pass your data.
The most common usage of mustache templates is for `{{variables}}`, used to pass a value at runtime.
### Using Variables in Prompt Templates
Let's look at the following template:
As you can see, `{{customer_data}}` and `{{chat_query}}` are defined as variables in the template and you can pass their value at runtime:
**And the prompt template uses the partial like this:**
**We can pass the data object inside the variables:**
### Publishing Prompts
Publishing a prompt version marks it as the default version that will be used when no specific version is requested. This is especially important for production environments.
Updating the Prompt does not automatically update your prompt in production. While updating, you can tick `Publish prompt changes` which will also update your prompt deployment to the latest version.
1. Create and test your new prompt version
2. When ready for production, click "Update" and check "Publish prompt changes"
3. Portkey will save the new version and mark it as the published version
4. All default API calls will now use this version
### Viewing Version History
**All** of your prompt versions can be seen by clicking the `Version History` button on the playground:
You can `Restore` or `Publish` any of the previous versions by clicking on the ellipsis menu.
### Comparing Versions
To compare different versions of your prompt:
1. Select the versions you want to compare from the version history panel
2. Click "Compare on playground" to see a side-by-side of different prompt versions
This helps you understand how prompts have evolved and which changes might have impacted performance.
## Using Different Prompt Versions
By default, when you pass the `PROMPT_ID` in `prompts.completions.create` method, Portkey sends the request to the `Published` version of your prompt.
You can also call any specific prompt version by appending version identifiers to your `PROMPT_ID`.
### Version Number References
**For example:**
```js
response = portkey.prompts.completions.create(
prompt_id="pp-classification-prompt@12",
variables={ }
)
```
Here, the request is sent to **Version 12** of the prompt template.
### Special Version References
Portkey supports special version references:
```js
// Latest version (may not be published)
response = portkey.prompts.completions.create(
prompt_id="pp-classification-prompt@latest",
variables={ }
)
// Published version (default when no suffix is provided)
response = portkey.prompts.completions.create(
prompt_id="pp-classification-prompt",
variables={ }
)
```
**Important Notes:**
* `@latest` refers to the most recent version, regardless of publication status
* When no suffix is provided, Portkey defaults to the `Published` version
* Each version is immutable once created - to make changes, you must create a new version
## Prompt Labels
Labels provide a more flexible and meaningful way to reference prompt versions compared to version numbers. You can add version tags/labels like `staging`, `production` to any prompt version to track changes and call them directly: