Comprehensive documentation for Portkey's AI Gateway, Guardrails, Observability, Prompts, and Governance features.

Portkey Docs

Portkey AI is a comprehensive platform designed to streamline and enhance AI integration for developers and organizations. It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in your Generative AI apps.

What is Portkey?

Integrate Portkey and analyze your first LLM call in 2 minutes!

Make Your First Request

Portkey Features

Automatically create, manage, and execute fine-tuning jobs for Large Language Models (LLMs) across multiple providers.

Autonomous Fine-tuning

Open Source

Comparing Portkey's Open-source version and Dev, Pro, Enterprise plans.

Feature Comparison

Contact Us

Are you navigating the challenging journey of transitioning LLMs from prototype stages to full-scale production? You're not alone. As this frontier of technology continues to expand, the roadmap isn't always clear. Best practices, guidelines, and efficient methodologies are still on the horizon.

Developer Forum

Since Portkey functions as a gateway - you may encounter Portkey-related, as well as non-Portkey related erros while using our services.

Common Errors & Resolutions

December '23 Migration

Integrations

Submit an Integration

Portkey connects with all major LLM providers and orchestration frameworks.

Overview

Google Gemini

Route to your AWS Sagemaker models through Portkey

AWS SageMaker

Ollama

Bring Your Own LLM

Milvus

Qdrant

Portkey helps bring your agents to production

Use Portkey with Autogen to take your AI Agents to production

Autogen

Use Portkey with Control Flow to take your AI Agents to production

Control Flow

Use Portkey with CrewAI to take your AI Agents to production

CrewAI

Langchain Agents

Use Portkey with LangGraph to take your AI Agents to production

LangGraph Agents

Use Portkey with Llama Agents to take your AI Agents to production

Llama Agents by Llamaindex

The Portkey x Swarm integration brings advanced AI gateway capabilities, full-stack observability, and reliability features to build production-ready AI agents.

OpenAI Swarm

Use Portkey with Phidata to take your AI Agents to production

Phidata

You can also use Portkey if you are doing custom agent orchestration!

Bring Your own Agents

AutoGen is a framework that enables the development of LLM applications using multiple agents that can converse with each other to solve tasks.

Integrate DSPy with Portkey for production-ready LLM pipelines

DSPy

With Portkey, you can confidently take your Instructor pipelines to production and get complete observability over all of your calls + make them reliable - all with a 2 LOC change!

Instructor

Portkey adds core production capabilities to any Langchain app.

Langchain (Python)

Langchain (JS/TS)

Cost tracking, observability, and more for LibreChat

LibreChat

Cost tracking, observability, and more for Open WebUI

Open WebUI

Add usage tracking, cost controls, and security guardrails to your Anything LLM deployment

Anything LLM

The **Portkey x LlamaIndex** integration brings advanced **AI gateway** capabilities, full-stack **observability**, and **prompt management** to apps built on LlamaIndex.

LlamaIndex (Python)

Portkey brings advanced **AI gateway** capabilities, full-stack **observability**, and **prompt management** + **versioning** to your **Promptfoo** projects. This document provides an overview of how to leverage the strengths of both the platforms to streamline your AI development workflow.

Promptfoo

Integrate Portkey with Vercel AI SDK for production-ready and reliable AI apps

Vercel

Integrate MindsDB with Portkey to build enterprise-grade AI use-cases

MindsDb

ToolJet is a low-code platform that lets you build apps by connecting APIs and data sources, with Portkey integration adding AI features like chat interfaces and automation.

ToolJet

MongoDB

Supabase

Learn how to integrate Portkey's enterprise features with Zed for enhanced observability, reliability and governance.

Learn how to integrate Portkey's enterprise features with any OpenAI Compliant project for enhanced observability, reliability and governance.

Portkey with Any OpenAI Compatible Project

This documentation provides detailed information about the various ways you can access and interact with Portkey - **a robust AI gateway** designed to simplify and enhance your experience with Large Language Models (LLMs) like OpenAI's GPT models. 

Introduction

Authentication

Header requirements and options for the Portkey API

Headers

Errors

Supported Providers

Chat

Execute your saved prompt templates on Portkey


Prompt Completions

Renders a prompt template with its variable values filled in


Prompt Render

Insert a Log

This endpoint allows users to submit feedback for a particular interaction or response.

Create Feedback

This endpoint allows users to update existing feedback.

Update Feedback

Embeddings

Create Image

Create Image Edit

Create Image Variation

Create Speech

Create Transcription

Create Translation

Access any custom provider endpoint through Portkey API

Gateway to Other APIs

Completions

Moderations

Create Fine-tuning Job

List Fine-tuning Jobs

Retrieve Fine-tuning Job

List Fine-tuning Events

List Fine-tuning Checkpoints

Cancel Fine-tuning

Create Batch

List Batch

Retrieve Batch

Cancel Batch

Upload File

List Files

Retrieve File

Retrieve File Content

Delete File

Manage your Portkey organization and workspaces programmatically

Create Config

List Configs

Retrieve Config

Update Config

Create Virtual Key

List Virtual Key

Retrieve Virtual Key

Update Virtual Key

Delete Virtual Key

Update API Key

Create API Key

Delete an API Key

Retrieve and API Key

List API Key

Retrieve a user

Retrieve all users

Update a user

Remove a user

Send an invite to user for your organization

Invite a User

Retrieve an user invite

Retrieve all user invite

Delete a user invite

Resend an invite to user for your organization

Resend a user invite

Create Workspace

Retrieve all Workspaces

Retrieve a Workspace

Update Workspace

Delete a Workspace

Add a Workspace Member

Retrieve all Workspace Member

Retrieve a Workspace Member

Update Workspace Member

Remove Workspace Member

OpenAPI Specification

Building a sophisticated AI SDR agent leveraging internet search and evals to draft personalized outreach emails in 15 seconds

Ultimate AI SDR

Portkey's prompt templates offer a powerful solution for testing and building chatbots.

Build a chatbot using Portkey's Prompt Templates

Building an LLM-as-a-Judge System for AI (Customer Support) Agent

A/B testing with large language models in production is crucial for driving optimal performance and user satisfaction.

A/B Test Prompts and Models

LLMs are **costly** to run. As their usage increaases, the providers have to balance serving user requests v/s straining their GPU resources too thin. They generally deal with this by putting _rate limits_ on how many requests a user can send in a minute or in a day.

Tackling Rate Limiting

Get the LLM to interact with external APIs!

Function Calling

Image Generation

Getting started with AI Gateway

Llama 3 on Groq

If you have multiple users of your GenAI app triggering the same or similar queries to your models, fetching LLM response from the models can be slow and expensive.

Return Repeat Requests from Cache

Trigger Automatic Retries on LLM Failures

You are likely familiar with how to make an API call to GPT4 for chat completions. 

101 on Portkey's Gateway Configs

Try out the new Llama 3 model directly using the OpenAI SDK

Llama 3 on Portkey + Together AI

Introduction to GPT-4o

Portkey helps bring Anyscale APIs to production with its abstractions for observability, fallbacks, caching, and more. Use the Anyscale API **through** Portkey for.

Anyscale

Portkey helps bring Mistral's APIs to production with its observability suite & AI Gateway.

Mistral

Portkey is a control panel for your Vercel AI app. It makes your LLM integrations prod-ready, reliable, fast, and cost-efficient.

Vercel AI

Deepinfra

Groq

Langchain

Mixtral 8x22b

Segmind

LLMs are highly capable of following a given structure. By providing a few examples of how the assistant should respond to a given prompt, the LLM can generate responses that closely follow the format of these examples.

Few-Shot Prompting

Get the LLM to adhere to your JSON schema using Anyscale & Together AI's newly introduced JSON modes

Enforcing JSON Schema with Anyscale & Together

Detecting Emotions with GPT-4o

 Consider that you have list of support articles that you want to suggest it to users when users search for it. You want to suggest as best fit as possible. With the availability of tools like Large Language Model (LLMs) and Vector Databases, the approach towards suggestions & recommendation systems has significantly evolved.

Build an article suggestion app with Supabase pgvector, and Portkey

Companies often face challenges of scaling their services efficiently as the traffic to their applications grow - when you’re consuming APIs, the first point of failure is that if you hit the API too much, you can get rate limited. Loadbalancing is a proven way to scale usage horizontally without overburdening any one provider and thus staying within rate limits.

Setting up resilient Load balancers with failure-mitigating Fallbacks

Writing the right prompt is often hard to get a quality LLM response. You want the prompt to be specialized and exhaustive enough for your problem. There is a high chance someone else might’ve stumbled across a similar situation and written the prompt you’ve been figuring out all this while. 

Run Portkey on Prompts from Langchain Hub

Portkey can help you easily create fallbacks from one LLM to another, making your application more reliable. While Fallback ensures reliability, it also means that you'll be running a prompt optimized for one LLM on another, which can often lead to significant differences in the final output.

Prompt Engineering

Whitepapers

Getting Started

Integrations

Use Cases

Prompts

Ultimate AI SDR

LLM as a Judge

Build a Chatbot using Prompt Templates