Advanced Prompting with JSON Mode

Comprehensive documentation for Portkey's AI Gateway, Guardrails, Observability, Prompts, and Governance features.

Portkey Docs

Portkey AI is a comprehensive platform designed to streamline and enhance AI integration for developers and organizations. It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in your Generative AI apps.

What is Portkey?

Integrate Portkey and analyze your first LLM call in 2 minutes!

Make Your First Request

Portkey Features

Automatically create, manage, and execute fine-tuning jobs for Large Language Models (LLMs) across multiple providers.

Autonomous Fine-tuning

Open Source

Comparing Portkey's Open-source version and Dev, Pro, Enterprise plans.

Feature Comparison

Contact Us

Are you navigating the challenging journey of transitioning LLMs from prototype stages to full-scale production? You're not alone. As this frontier of technology continues to expand, the roadmap isn't always clear. Best practices, guidelines, and efficient methodologies are still on the horizon.

Developer Forum

Since Portkey functions as a gateway - you may encounter Portkey-related, as well as non-Portkey related erros while using our services.

Common Errors & Resolutions

December '23 Migration

Integrations

Submit an Integration

Portkey connects with all major LLM providers and orchestration frameworks.

Overview

Google Gemini

Route to your AWS Sagemaker models through Portkey

AWS SageMaker

Ollama

Bring Your Own LLM

Milvus

Qdrant

Portkey helps bring your agents to production

Use Portkey with Autogen to take your AI Agents to production

Autogen

Use Portkey with Control Flow to take your AI Agents to production

Control Flow

Use Portkey with CrewAI to take your AI Agents to production

CrewAI

Langchain Agents

Use Portkey with LangGraph to take your AI Agents to production

LangGraph Agents

Use Portkey with Llama Agents to take your AI Agents to production

Llama Agents by Llamaindex

The Portkey x Swarm integration brings advanced AI gateway capabilities, full-stack observability, and reliability features to build production-ready AI agents.

OpenAI Swarm

Use Portkey with Phidata to take your AI Agents to production

Phidata

You can also use Portkey if you are doing custom agent orchestration!

Bring Your own Agents

AutoGen is a framework that enables the development of LLM applications using multiple agents that can converse with each other to solve tasks.

Integrate DSPy with Portkey for production-ready LLM pipelines

DSPy

With Portkey, you can confidently take your Instructor pipelines to production and get complete observability over all of your calls + make them reliable - all with a 2 LOC change!

Instructor

Portkey adds core production capabilities to any Langchain app.

Langchain (Python)

Langchain (JS/TS)

Cost tracking, observability, and more for LibreChat

LibreChat

Cost tracking, observability, and more for Open WebUI

Open WebUI

Add usage tracking, cost controls, and security guardrails to your Anything LLM deployment

Anything LLM

The **Portkey x LlamaIndex** integration brings advanced **AI gateway** capabilities, full-stack **observability**, and **prompt management** to apps built on LlamaIndex.

LlamaIndex (Python)

Portkey brings advanced **AI gateway** capabilities, full-stack **observability**, and **prompt management** + **versioning** to your **Promptfoo** projects. This document provides an overview of how to leverage the strengths of both the platforms to streamline your AI development workflow.

Promptfoo

Integrate Portkey with Vercel AI SDK for production-ready and reliable AI apps

Vercel

Integrate MindsDB with Portkey to build enterprise-grade AI use-cases

MindsDb

ToolJet is a low-code platform that lets you build apps by connecting APIs and data sources, with Portkey integration adding AI features like chat interfaces and automation.

ToolJet

MongoDB

Supabase

Learn how to integrate Portkey's enterprise features with Zed for enhanced observability, reliability and governance.

Learn how to integrate Portkey's enterprise features with any OpenAI Compliant project for enhanced observability, reliability and governance.

Portkey with Any OpenAI Compatible Project

This documentation provides detailed information about the various ways you can access and interact with Portkey - **a robust AI gateway** designed to simplify and enhance your experience with Large Language Models (LLMs) like OpenAI's GPT models. 

Introduction

Authentication

Header requirements and options for the Portkey API

Headers

Errors

Supported Providers

Chat

Execute your saved prompt templates on Portkey


Prompt Completions

Renders a prompt template with its variable values filled in


Prompt Render

Embeddings

Create Image

Create Image Edit

Create Image Variation

Create Speech

Create Transcription

Create Translation

Create Realtime Session

Access any custom provider endpoint through Portkey API

Gateway to Other APIs

Completions

Moderations

Create Fine-tuning Job

List Fine-tuning Jobs

Retrieve Fine-tuning Job

List Fine-tuning Events

List Fine-tuning Checkpoints

Cancel Fine-tuning

Create Batch

List Batch

Retrieve Batch

Cancel Batch

Upload File

List Files

Retrieve File

Retrieve File Content

Delete File

Guardrails

OpenAPI Specification

Building a sophisticated AI SDR agent leveraging internet search and evals to draft personalized outreach emails in 15 seconds

Ultimate AI SDR

A/B testing with large language models in production is crucial for driving optimal performance and user satisfaction.

A/B Test Prompts and Models

LLMs are **costly** to run. As their usage increaases, the providers have to balance serving user requests v/s straining their GPU resources too thin. They generally deal with this by putting _rate limits_ on how many requests a user can send in a minute or in a day.

Tackling Rate Limiting

Get the LLM to interact with external APIs!

Function Calling

Image Generation

Getting started with AI Gateway

Llama 3 on Groq

If you have multiple users of your GenAI app triggering the same or similar queries to your models, fetching LLM response from the models can be slow and expensive.

Return Repeat Requests from Cache

Trigger Automatic Retries on LLM Failures

You are likely familiar with how to make an API call to GPT4 for chat completions. 

101 on Portkey's Gateway Configs

Try out the new Llama 3 model directly using the OpenAI SDK

Llama 3 on Portkey + Together AI

Introduction to GPT-4o

Portkey helps bring Anyscale APIs to production with its abstractions for observability, fallbacks, caching, and more. Use the Anyscale API **through** Portkey for.

Anyscale

Portkey helps bring Mistral's APIs to production with its observability suite & AI Gateway.

Mistral

Portkey is a control panel for your Vercel AI app. It makes your LLM integrations prod-ready, reliable, fast, and cost-efficient.

Vercel AI

Deepinfra

Groq

Langchain

Mixtral 8x22b

Segmind

LLMs are highly capable of following a given structure. By providing a few examples of how the assistant should respond to a given prompt, the LLM can generate responses that closely follow the format of these examples.

Few-Shot Prompting

Get the LLM to adhere to your JSON schema using Anyscale & Together AI's newly introduced JSON modes

Enforcing JSON Schema with Anyscale & Together

Detecting Emotions with GPT-4o

 Consider that you have list of support articles that you want to suggest it to users when users search for it. You want to suggest as best fit as possible. With the availability of tools like Large Language Model (LLMs) and Vector Databases, the approach towards suggestions & recommendation systems has significantly evolved.

Build an article suggestion app with Supabase pgvector, and Portkey

Companies often face challenges of scaling their services efficiently as the traffic to their applications grow - when you’re consuming APIs, the first point of failure is that if you hit the API too much, you can get rate limited. Loadbalancing is a proven way to scale usage horizontally without overburdening any one provider and thus staying within rate limits.

Setting up resilient Load balancers with failure-mitigating Fallbacks

Writing the right prompt is often hard to get a quality LLM response. You want the prompt to be specialized and exhaustive enough for your problem. There is a high chance someone else might’ve stumbled across a similar situation and written the prompt you’ve been figuring out all this while. 

Run Portkey on Prompts from Langchain Hub

Portkey can help you easily create fallbacks from one LLM to another, making your application more reliable. While Fallback ensures reliability, it also means that you'll be running a prompt optimized for one LLM on another, which can often lead to significant differences in the final output.

Smart Fallback with Model-Optimized Prompts

Portkeys Prompt Playground allows you to test and tinker with various hyperparameters without any external dependencies and deploy them to production seamlessly. Moreover, all team members can use the same prompt template, ensuring that everyone works from the same source of truth.

How to use OpenAI SDK with Portkey Prompt Templates

Portkey Fallbacks can automatically switch your app's requests from one LLM provider to another, ensuring reliability by allowing you to fallback among multiple LLMs.

Setup OpenAI -> Azure OpenAI Fallback

Generative AI models have revolutionized text generation and opened up new possibilities for developers.

Fallback from SDXL to Dall-e-3

Comparing Top10 LMSYS Models with Portkey

Portkey's prompt templates offer a powerful solution for testing and building chatbots.

Build a chatbot using Portkey's Prompt Templates

Monitor and analyze user-level LLM costs across 1600+ models using Portkey's metadata and analytics API.

Tracking LLM Costs Per User with Portkey

Enterprise Gateway [1.10.10]

Enterprise Gateway

Helm Chart [1.0.5]

Helm Chart

Latest Updates

Learn to integrate OpenAI with Portkey, enabling seamless completions, prompt management, and advanced functionalities like streaming, function calling and fine-tuning.

OpenAI

Structured Outputs ensure that the model always follows your supplied [JSON schema](https://json-schema.org/overview/what-is-jsonschema). Portkey supports OpenAI's Structured Outputs feature out of the box with our SDKs & APIs. 

Structured Outputs

Prompt Caching

Files

Batches

Anthropic

Google Vertex AI

Controlled Generations ensure that the model always follows your supplied [JSON schema](https://json-schema.org/overview/what-is-jsonschema). Portkey supports Vertex AI's Controlled Generations feature out of the box with our SDKs & APIs. 

Controlled Generations

Azure OpenAI is a great alternative to accessing the best models including GPT-4 and more in your private environments. Portkey provides complete support for Azure OpenAI.

Azure OpenAI

AWS Bedrock

Upload files to S3 for Bedrock batch inference

LocalAI

Integrate vLLM-hosted custom models with Portkey and take them to production

vLLM

Integrate Trtiton-hosted custom models with Portkey and take them to production

Triton

AI21

Integrate Anyscale endpoints with Portkey seamlessly and make your OSS models production-ready

Introduction

Product

Support