December '23 Migration

Date: 8th Dec, 2023

This December, we’re pushing out some exciting new updates to Portkey’s SDKs, APIs, and Configs.

​

Compatibility & Deprecation List

We recommend upgrading to these new versions promptly to take full advantage of their capabilities. While your existing code will continue to work until the deprecation date around Q2 ‘24, transitioning now ensures you stay ahead of the curve and avoid any future service interruptions. Follow along with this guide!

​

Major Version Release of the SDK

​

Here’s What’s New:

1. More extensible SDK that can be used with many more LLM providers
2. Out-of-the-box support for streaming
3. Completely follows OpenAI’s SDK signature reducing your technical debt
4. Native support for Langchain & Llamaindex within the SDK (Python)
5. Support for the Portkey Feedback endpoint
6. Support for Portkey Prompt Templates
7. Older SDK versions to be deprecated soon

​

Here’s What’s Changed:

• Portkey Python SDK
• Portkey Node SDK

import portkey
from portkey import Config, LLMOptions

portkey.config = Config(
    mode="single",
    llms=LLMOptions(provider="openai", api_key="OPENAI_API_KEY")
)

response = portkey.ChatCompletions.create(
    model="gpt-4",
    messages=[
        {"role": "user","content": "Hello World!"}
    ]
)

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    Authorization="OPENAI_KEY"
)

response = portkey.chat.completions.create(
    messages=[{'role': 'user', 'content': 'Say this is a test'}],
    model='gpt-3.5-turbo'
)

print(response)

pip install -U portkey-ai

​

All-New APIs

​

Here’s What’s New:

1. Introduced 3 new routes /chat/completions, /completions, and /embeddings
2. Simplified the headers:
   1. x-portkey-mode header is deprecated and replaced with x-portkey-provider
      1. Which takes values: openai, anyscale, cohere, palm, azure-openai, and more.
   2. New header x-portkey-virtual-key is introduced.
3. /complete and /chatComplete endpoints to be deprecated soon
4. Prompts endpoint /prompts/$PROMPT_ID/generate is upgraded to /prompts/$PROMPT_ID/completions and the old route will be deprecated soon
   1. We now support updating the model params on-the-fly (i.e. changing temperature etc at the time of making a call)
   2. Prompt response object on the /completions route is now fully OpenAI compliant
5. New /gateway endpoint that lets you make calls to third-party LLM providers easily

​

Here’s What’s Changed

• OpenAI Python SDK
• OpenAI Node SDK
• cURL

from openai import OpenAI

client = OpenAI(
    api_key="OPENAI_API_KEY", # defaults to os.environ.get("OPENAI_API_KEY")
    base_url="https://api.portkey.ai/v1/proxy",
    default_headers= {
        "x-portkey-api-key": "PORTKEY_API_KEY",
        "x-portkey-mode": "proxy openai",
        "Content-Type": "application/json"
    }
)

chat_complete = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Say this is a test"}],
)

print(chat_complete.choices[0].message.content)

# pip install -U portkey-ai

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

client = OpenAI(
    api_key="OPENAI_API_KEY", # defaults to os.environ.get("OPENAI_API_KEY")
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="openai",
        api_key="PORTKEY_API_KEY" # defaults to os.environ.get("PORTKEY_API_KEY")
    )
)

chat_complete = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Say this is a test"}],
)

print(chat_complete.choices[0].message.content)

​

Simlarly, for Prompts

• cURL
• Changing Model Params On-the-fly (Python)

curl https://api.portkey.ai/v1/prompts/$PROMPT_ID/generate \
  -H 'x-portkey-api-key: $PORTKEY_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"variables": {"variable_a": "", "variable_b": ""}}'

curl https://api.portkey.ai/v1/prompts/$PROMPT_ID/completions \
  -H 'x-portkey-api-key: $PORTKEY_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"variables": {"variable_a": "", "variable_b": ""}}'

​

Configs 2.0

​

Here’s What’s New

1. New concept of strategy instead of standalone mode. You can now build bespoke gateway strategies and nest them in a single config.
2. You can also trigger a specific strategy on specific error codes.
3. New concept of targets that replace options in the previous Config
4. If you are adding virtual_key to the target array, you no longer need to add provider,Portkey will pick up the Provider directly from the Virtual Key!
5. For Azure, only now pass the virtual_key - it takes care of all other Azure params like Deployment name, API version etc.

​

Here’s What’s Changed

• Config Builder

{
	"mode": "single",
	"options": [
		{
			"provider": "openai",
			"virtual_key": "open-4110dd",
		}
	]
}

{
	"strategy": {
		"mode":"single"
	},
	"targets": [
		{
			"virtual_key": "open-4110dd"
		}
	]
}

​

Support

Shoot ANY questions or queries you have on the migration to the Portkey team on our Discordand we will try to get back to you ASAP.