Home
Product
API
Integrations
Changelog
Open Source

Chat Completions

Create Chat Completion

POST /chat/completions

Generate a chat message completion from the selected LLM.

The body is similar to the chat completion request of OpenAI and the response will be the Chat Completion Object. When choosing stream:true the response will be a stream of Chat Completion Chunk objects.

Portkey automatically transforms the parameters for LLMs other than OpenAI. If some parameters don't exist in the other LLMs, they will be dropped.

SDK Usage

The chat.completions.create method in the Portkey SDK enables you to generate chat completions using various Large Language Models (LLMs). This method is designed to be similar to the OpenAI chat completions API, offering a familiar interface for those accustomed to OpenAI's services.

Method Signature

portkey.chat.completions.create(requestParams[, configParams]);

For REST API examples, scroll here.

Parameters

  1. requestParams (Object): Parameters for the chat completion request, detailing the chat interaction. These are similar to the OpenAI request signature. Portkey automatically transforms the parameters for LLMs other than OpenAI. If some parameters don't exist in the other LLMs, they will be dropped. Portkey is multimodal by-default, so parameters relevant to vision models, like image_url, base64 data are also supported.

  2. configParams(Object): Additional configuration options for the request. This is an optional parameter that can include custom config options for this specific request. These will override the configs set in the Portkey Client. A full list of these config parameters can be found here.

Example Usage

1. Default

The chat completions endpoint accepts an array of message objects and returns the completion in a chat message format.

import Portkey from 'portkey-ai';

// Initialize the Portkey client
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",  // Replace with your Portkey API key
    virtualKey: "VIRTUAL_KEY"   // Optional: For virtual key management
});

// Generate a chat completion
async function getChatCompletion() {
    const chatCompletion = await portkey.chat.completions.create({
        messages: [{ role: 'user', content: 'Say this is a test' }],
        model: 'gpt-3.5-turbo',
    });

    console.log(chatCompletion);
}
await getChatCompletion();

To send chat requests to locally or privately hosted models, check out the guide on Ollama.

2. Image Input (Vision Models)

The chat completions API also supports adding images to the request for vision models (GPT-4V, Gemini, etc).

// Generate a chat completion with streaming
async function getChatCompletionVision(){
    const chatCompletion = await portkey.chat.completions.create({
        model: "gpt-4-vision-preview",
        messages: [
          {
            role: "user",
            content: [
              { type: "text", text: "What’s in this image?" },
              {
                type: "image_url",
                image_url:
                  "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
              },
            ],
          },
        ],
    });

    console.log(response.choices[0]);
}
await getChatCompletionVision();

3. Streaming Chat Completions

Pass the stream parameter as true in the request to enable streaming responses from the Chat completions API.

The chat completions endpoint accepts an array of message objects and returns the completion in a chat message format.

// Generate a chat completion with streaming
async function getChatCompletionStream(){
    const chatCompletion = await portkey.chat.completions.create({
        messages: [{ role: 'user', content: 'Say this is a test' }],
        model: 'gpt-3.5-turbo',
        stream: true
    });

    for await (const chunk of chatCompletion) {
        console.log(chunk.choices[0].delta.content);
    }
}
await getChatCompletionStream();

4. Functions

The tools parameter accepts functions which can be sent specifically for models that support function calling.

// Generate a chat completion with streaming
async function getChatCompletionFunctions(){
  const messages = [{"role": "user", "content": "What's the weather like in Boston today?"}];
  const tools = [
      {
        "type": "function",
        "function": {
          "name": "get_current_weather",
          "description": "Get the current weather in a given location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA",
              },
              "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
          },
        }
      }
  ];

    const response = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: messages,
    tools: tools,
    tool_choice: "auto",
  });
  
  console.log(response)

}
await getChatCompletionFunctions();

5. Custom configuration for each request

There might be a need to override config values per request, or send options for trace id and metadata as part of the request being made. This is possible by attaching adding these parameters along with the request being made.

// Generate chat completion with config params
async function getChatCompletionWithConfig() {
    const chatCompletion = await portkey.chat.completions.create({
        messages: [{ role: 'user', content: 'Say this is a test' }],
        model: 'gpt-3.5-turbo',
    }, {config: "sample-7g5tr4"});

    for await (const chunk of chatCompletion) {
        console.log(chunk.choices[0].delta.content);
    }
}
await getChatCompletionWithConfig();

Response Format

The response will conform to the Chat Completions Object schema from the Portkey API, typically including generated messages and relevant metadata.

PreviousConfig ObjectNextCompletions

Last updated