Portkey Docs

Inference API

Admin API

Changelog ↗

Sign in →

Support

Github

Retrieve Run Steps

Usage statistics related to the run step. This value will be `null` while the run step's status is `in_progress`.

Number of completion tokens used over the course of the run step.

Number of prompt tokens used over the course of the run step.

Total number of tokens used (prompt + completion).

Details of the message creation by the run step.

The ID of the message that was created by this run step.

Message creation

Details of the Code Interpreter tool call the run step was involved in.

The Code Interpreter tool call definition.

The input to the Code Interpreter tool call.

The outputs from the Code Interpreter tool call. Code Interpreter can output one or more items, including text (`logs`) or images (`image`). Each of these are represented by a different object type.

The type of tool call. This is always going to be `code_interpreter` for this type of tool call.

Code Interpreter tool call

The [file](https://platform.openai.com/docs/api-reference/files) ID of the image.

Code Interpreter image output

Text output from the Code Interpreter tool call as part of a run step.

The text output from the Code Interpreter tool call.

Code Interpreter log output

For now, this is always going to be an empty object.

The type of tool call. This is always going to be `file_search` for this type of tool call.

File search tool call

The definition of the function that was called.

The output of the function. This will be `null` if the outputs have not been [submitted](https://platform.openai.com/docs/api-reference/runs/submitToolOutputs) yet.

The type of tool call. This is always going to be `function` for this type of tool call.

Function tool call

An array of tool calls the run step was involved in. These can be associated with one of three types of tools: `code_interpreter`, `file_search`, or `function`.


Tool calls

Portkey Key & Virtual Key

Portkey Key & Provider Auth & Provider Name

Portkey Key & Config

Portkey Key & Provider Auth & Provider Name & Custom Host

Represents a step in execution of a run.


Run steps

Retrieves a run step.

Portkey AI is a comprehensive platform designed to streamline and enhance AI integration for developers and organizations. It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in your Generative AI apps.

What is Portkey?

Integrate Portkey and analyze your first LLM call in 2 minutes!

Make Your First Request

Portkey Features

Portkey connects with all major LLM providers and orchestration frameworks.

Supported LLMs

Google Gemini

Azure OpenAI is a great alternative to accessing the best models including GPT-4 and more in your private environments. Portkey provides complete support for Azure OpenAI.

Azure OpenAI

AWS Bedrock

Google Vertex AI

Ollama

LocalAI

AI21

Integrate Anyscale endpoints with Portkey seamlessly and make your OSS models production-ready

Anyscale

Cerebras

Cohere

Fireworks

Deepinfra

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including [Deepbricks](https://deepbricks.ai/). 

Deepbricks

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including DeepSeek models. 

DeepSeek

Groq

Hugging Face

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including the models hosted on [Inference.net](https://www.inference.net/). 

Inference.net

Jina AI

Lingyi (01.ai)

Mistral AI

MonsterAPIs provides access to generative AI model APIs at 80% lower costs. Connect to MonsterAPI LLM APIs seamlessly through Portkey's AI gateway.

Monster API

Moonshot

Nomic

Novita AI

OpenRouter

Perplexity AI

Predibase

Reka AI

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including [SambaNova AI](https://sambanova.ai/).

SambaNova

Segmind

Stability AI

SiliconFlow

Together AI

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including Voyage AI's embedding and Re-rank endpoints. 

Voyage AI

Workers AI

ZhipuAI / ChatGLM / BigModel

Bring Your Own LLM

Suggest a new integration!

Portkey helps bring your agents to production!

Agents

Use Portkey with Autogen to take your AI Agents to production

Autogen

Use Portkey with Control Flow to take your AI Agents to production

Control Flow

Use Portkey with CrewAI to take your AI Agents to production

CrewAI

Langchain Agents

Use Portkey with LangGraph to take your AI Agents to production

LangGraph Agents

Use Portkey with Llama Agents to take your AI Agents to production

Llama Agents by Llamaindex

Use Portkey with Phidata to take your AI Agents to production

Phidata

You can also use Portkey if you are doing custom agent orchestration!

Bring Your own Agents

Libraries

AutoGen is a framework that enables the development of LLM applications using multiple agents that can converse with each other to solve tasks.

Integrate DSPy with Portkey for production-ready LLM pipelines

DSPy

With Portkey, you can confidently take your Instructor pipelines to production and get complete observability over all of your calls + make them reliable - all with a 2 LOC change!

Instructor

Portkey adds core production capabilities to any Langchain app.

Langchain (Python)

Langchain (JS/TS)

Integrate Portkey with LibreChat for enhanced AI capabilities

LibreChat

The **Portkey x LlamaIndex** integration brings advanced **AI gateway** capabilities, full-stack **observability**, and **prompt management** to apps built on LlamaIndex.

LlamaIndex (Python)

Portkey brings advanced **AI gateway** capabilities, full-stack **observability**, and **prompt management** + **versioning** to your **Promptfoo** projects. This document provides an overview of how to leverage the strengths of both the platforms to streamline your AI development workflow.

Promptfoo

Integrate Portkey with Vercel AI SDK for production-ready and reliable AI apps

Vercel

Gain real-time insights, track key metrics, and streamline debugging with our comprehensive observability suite.

Observability (OpenTelemetry)

The Logs section presents a chronological list of all the requests processed through Portkey.

Logs

The **Tracing** capabilities in Portkey empowers you to monitor the lifecycle of your LLM requests in a unified, chronological view.

Tracing

Analytics

Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app.

Feedback

Metadata

Filters

Easily access your Portkey logs data for further analysis and reporting

Logs Export

Budget Limits

Portkey provides the world's fastest multimodal AI gateway that enables working with multiple LLM providers as well as different modalities very easy. The gateway providers features to improve your app's reliability, cost efficiency and accuracy.

AI Gateway

Portkey's Universal API provides a consistent interface to integrate a wide range of modalities (text, vision, audio) and LLMs (hosted OR local) into your apps.

Universal API

This feature is available on all Portkey plans.

Configs

Conditional Routing

Cache (Simple & Semantic)

Fallbacks

LLM APIs often have inexplicable failures. With Portkey, you can rescue a substantial number of your requests with our in-built automatic retries feature. 

Automatic Retries

Load Balancing feature efficiently distributes network traffic across multiple LLMs.

Load Balancing

You can use Portkey's AI gateway to also canary test new models or prompts in different environments. 

Canary Testing

Manage unpredictable LLM latencies effectively with Portkey's **Request Timeouts**.

Request Timeouts

With Portkey's Prompt Library, you can seamlessly create and manage prompts along with all associated model parameters.

Prompt Library

With Prompt Templates, you can seamlessly create and manage your LLM prompts in one place, and deploy them with just an API call.

Prompt Templates

With Prompt Partials, you can save your commonly used templates (which could be your instruction set, data structure explanation, examples etc.) separately from your prompts and flexibly incorporate them wherever required.

Prompt Partials

See how to use Portkey's prompt templates with OpenAI (or any other provider) SDKs

Retrieve Prompts

Advanced Prompting with JSON Mode

Ship to production confidently with Portkey Guardrails on your requests & responses

Guardrails

With the raw Guardrails mode, we let you define your Guardrail checks & actions however you want, directly in code.

Creating Raw Guardrails (in JSON)

Automatically create, manage, and execute fine-tuning jobs for Large Language Models (LLMs) across multiple providers.

Autonomous Fine-tuning

Enterprise Offering

With customizable user roles, API key management, and comprehensive audit logs, Portkey provides the flexibility and control needed to ensure secure collaboration & maintain a strong security posture

Access Control Management

Portkey AI provides a secure, reliable AI gateway for the seamless integration and management of large language models (LLMs).

Security @ Portkey

Open Source

Portkey processes **billions** of tokens everyday for our customers worldwide. All these requests are routed through our AI Gateway.

Product Feature Comparison

Getting Started

A/B testing with large language models in production is crucial for driving optimal performance and user satisfaction.

A/B Test Prompts and Models

LLMs are **costly** to run. As their usage increaases, the providers have to balance serving user requests v/s straining their GPU resources too thin. They generally deal with this by putting _rate limits_ on how many requests a user can send in a minute or in a day.

Tackling Rate Limiting

Get the LLM to interact with external APIs!

Function Calling

Image Generation

Getting started with AI Gateway

Llama 3 on Groq

If you have multiple users of your GenAI app triggering the same or similar queries to your models, fetching LLM response from the models can be slow and expensive.

Return Repeat Requests from Cache

Trigger Automatic Retries on LLM Failures

You are likely familiar with how to make an API call to GPT4 for chat completions. 

101 on Portkey's Gateway Configs

Integration

Try out the new Llama 3 model directly using the OpenAI SDK

Portkey Endpoints

Provider Endpoints

Retrieve Run Steps

Authorizations

Path Parameters

Response