Portkey + Langchain
Portkey is the Control Panel for AI apps. With it’s popular AI Gateway and Observability Suite, hundreds of teams ship reliable, cost-efficient, and fast apps.
Portkey brings production readiness to Langchain. With Portkey, you can
- Connect to 150+ models through a unified API,
- View 42+ metrics & logs for all requests,
- Enable semantic cache to reduce latency & costs,
- Implement automatic retries & fallbacks for failed requests,
- Add custom tags to requests for better tracking and analysis and more.
Quickstart
Since Portkey is fully compatible with the OpenAI signature, you can connect to the Portkey AI Gateway through the ChatOpenAI interface.
- Set the
base_url as PORTKEY_GATEWAY_URL
- Add
default_headers to consume the headers needed by Portkey using the createHeaders helper method.
To start, get your Portkey API key by signing up here. (Click the profile icon on the bottom left, then click on ” API Key”)
!pip install -qU portkey-ai langchain-openai
We can now connect to the Portkey AI Gateway by updating the ChatOpenAI model in Langchain
Using OpenAI models with Portkey + ChatOpenAI
from langchain_openai import ChatOpenAI
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL
from google.colab import userdata
portkey_headers = createHeaders(api_key= userdata.get("PORTKEY_API_KEY"), ## Grab from https:
provider="openai"
)
llm = ChatOpenAI(api_key= userdata.get("OPENAI_API_KEY"),
base_url=PORTKEY_GATEWAY_URL,
default_headers=portkey_headers)
llm.invoke("What is the meaning of life, universe and everything?")
Using Together AI models with Portkey + ChatOpenAI
from langchain_openai import ChatOpenAI
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL
from google.colab import userdata
portkey_headers = createHeaders(api_key= userdata.get("PORTKEY_API_KEY"), ## Grab from https:
provider="together-ai"
)
llm = ChatOpenAI(model = "meta-llama/Llama-3-8b-chat-hf",
api_key= userdata.get("TOGETHER_API_KEY"), ## Replace it with your provider key
base_url=PORTKEY_GATEWAY_URL,
default_headers=portkey_headers)
llm.invoke("What is the meaning of life, universe and everything?")
Advanced Routing - Load Balancing, Fallbacks, Retries
The Portkey AI Gateway brings capabilities like load-balancing, fallbacks, experimentation and canary testing to Langchain through a configuration-first approach.
Let’s take an example where we might want to split traffic between llama-3-70b and gpt-3.5 50:50 to test the two large models. The gateway configuration for this would look like the following:
config = {
"strategy": {
"mode": "loadbalance"
},
"targets": [{
"virtual_key": "gpt3-8070a6", # OpenAI's virtual key
"override_params": {"model": "gpt-3.5-turbo"},
"weight": 0.5
}, {
"virtual_key": "together-1c20e9", # Together's virtual key
"override_params": {"model": "meta-llama/Llama-3-8b-chat-hf"},
"weight": 0.5
}]
}
from langchain_openai import ChatOpenAI
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL
from google.colab import userdata
portkey_headers = createHeaders(
api_key= userdata.get("PORTKEY_API_KEY"),
config=config
)
llm = ChatOpenAI(api_key="X", base_url=PORTKEY_GATEWAY_URL, default_headers=portkey_headers)
llm.invoke("What is the meaning of life, universe and everything?")
