LangGraph Agents

Install the required packages

pip install -U langgraph langchain_openai portkey-ai

Configure The ChatOpenAI Object with Portkey Settings

from langchain_openai import ChatOpenAI
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL

llm = ChatOpenAI(
    api_key="dummy", # We'll pass a dummy API key here
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key="PORTKEY_API_KEY",
        virtual_key="YOUR_LLM_PROVIDER_VIRTUAL_KEY" # Pass your virtual key saved on Portkey for any provider you'd like (Anthropic, OpenAI, Groq, etc.)
    )
)

from langchain_openai import ChatOpenAI
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL

llm = ChatOpenAI(
    api_key="dummy", # We'll pass a dummy API key here
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key="PORTKEY_API_KEY",
        virtual_key="YOUR_LLM_PROVIDER_VIRTUAL_KEY" # Pass your virtual key saved on Portkey for any provider you'd like (Anthropic, OpenAI, Groq, etc.)
    )
)

import { ChatOpenAI } from "@langchain/openai";
import { createHeaders, PORTKEY_GATEWAY_URL } from "portkey-ai";

// Configure Portkey settings
const portkeyConf = {
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY",
    virtualKey: "OPENAI_VIRTUAL_KEY"
  })
};

// Initialize the LLM with Portkey configuration
const llm = dummy", #  We'll pass a dummy API key here
  configuration: portkeyConf,
  model: "gpt-4o" // or your preferred model
});

The rest of your LangGraph implementation remains the same! Execute your agent and visit your Portkey dashboard to observe your Agent is performing.

End-to-End Example

Here’s a minimal working example of building a LangGraph agent with Portkey:

We’ll first create a simple chatbot using LangGraph and Portkey. This chatbot will respond directly to user messages. Though simple, it will illustrate the core concepts of building with LangGraph.

from typing import Annotated
from langchain_openai import ChatOpenAI
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL

from typing_extensions import TypedDict
from langgraph.graph import StateGraph
from langgraph.graph.message import add_messages

class State(TypedDict):
    messages: Annotated[list, add_messages]

graph_builder = StateGraph(State)

llm = ChatOpenAI(
    api_key="OpenAI_API_Key",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="openai",
        api_key="PORTKEY_API_KEY"
    )
)

def chatbot(state: State):
    return {"messages": [llm.invoke(state["messages"])]}

graph_builder.add_node("chatbot", chatbot)
graph_builder.set_entry_point("chatbot")
graph_builder.set_finish_point("chatbot")
graph = graph_builder.compile()

def stream_graph_updates(user_input: str):
    for event in graph.stream({"messages": [("user", user_input)]}):
        for value in event.values():
            print("Assistant:", value["messages"][-1].content)

while True:
    try:
        user_input = input("User: ")
        if user_input.lower() in ["quit", "exit", "q"]:
            print("Goodbye!")
            break
        stream_graph_updates(user_input)
    except:
        user_input = "What do you know about LangGraph?"
        print("User: " + user_input)
        stream_graph_updates(user_input)
        break

We’ll first create a simple chatbot using LangGraph and Portkey. This chatbot will respond directly to user messages. Though simple, it will illustrate the core concepts of building with LangGraph.

from typing import Annotated
from langchain_openai import ChatOpenAI
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL

from typing_extensions import TypedDict
from langgraph.graph import StateGraph
from langgraph.graph.message import add_messages

class State(TypedDict):
    messages: Annotated[list, add_messages]

graph_builder = StateGraph(State)

llm = ChatOpenAI(
    api_key="OpenAI_API_Key",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="openai",
        api_key="PORTKEY_API_KEY"
    )
)

def chatbot(state: State):
    return {"messages": [llm.invoke(state["messages"])]}

graph_builder.add_node("chatbot", chatbot)
graph_builder.set_entry_point("chatbot")
graph_builder.set_finish_point("chatbot")
graph = graph_builder.compile()

def stream_graph_updates(user_input: str):
    for event in graph.stream({"messages": [("user", user_input)]}):
        for value in event.values():
            print("Assistant:", value["messages"][-1].content)

while True:
    try:
        user_input = input("User: ")
        if user_input.lower() in ["quit", "exit", "q"]:
            print("Goodbye!")
            break
        stream_graph_updates(user_input)
    except:
        user_input = "What do you know about LangGraph?"
        print("User: " + user_input)
        stream_graph_updates(user_input)
        break

This code sets up a simple chatbot that can respond to user queries about the weather using LangGraph and Portkey in JS.

// agent.ts
import { TavilySearchResults } from "@langchain/community/tools/tavily_search";
import { ChatOpenAI } from "@langchain/openai";
import { MemorySaver } from "@langchain/langgraph";
import { HumanMessage } from "@langchain/core/messages";
import { createReactAgent } from "@langchain/langgraph/prebuilt";

// Configure Portkey settings
const portkeyConf = {
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: PORTKEY_API_KEY,
    virtualKey: "openaiVirtualKey"
  })
};

// Define the tools for the agent to use
const agentTools = [new TavilySearchResults({ maxResults: 3 })];
const agentModel = new ChatOpenAI({
  apiKey: "OPENAI_API_KEY",
  configuration: portkeyConf,
  model: "gpt-4"
});

// Initialize memory to persist state between graph runs
const agentCheckpointer = new MemorySaver();
const agent = createReactAgent({
  llm: agentModel,
  tools: agentTools,
  checkpointSaver: agentCheckpointer,
});

// Example usage with async/await
async function runAgent() {
  // First interaction
  const agentFinalState = await agent.invoke(
    { messages: [new HumanMessage("what is the current weather in sf")] },
    { configurable: { thread_id: "42" } },
  );

  console.log(
    agentFinalState.messages[agentFinalState.messages.length - 1].content,
  );

  // Follow-up interaction using the same thread
  const agentNextState = await agent.invoke(
    { messages: [new HumanMessage("what about ny")] },
    { configurable: { thread_id: "42" } },
  );

  console.log(
    agentNextState.messages[agentNextState.messages.length - 1].content,
  );
}

// Run the agent
runAgent().catch(console.error);

Integration Guide

Here’s a simple Google Colab notebook that demonstrates LangGraph with Portkey integration

Google Colab

LangGraph Cookbook

Make your agents Production-ready with Portkey

Portkey makes your LangGraph agents reliable, robust, and production-grade with its observability suite and AI Gateway. Seamlessly integrate 200+ LLMs with your LangGraph agents using Portkey. Implement fallbacks, gain granular insights into agent performance and costs, and continuously optimize your AI operations—all with just 2 lines of code.

Let’s dive deep! Let’s go through each of the use cases!

1. Interoperability

Easily switch between 200+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the provider and API key in the ChatOpenAI object.

If you are using OpenAI with LangGraph, your code would look like this:

llm = ChatOpenAI(
    api_key="OpenAI_API_Key",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="openai", #choose your provider
        api_key="PORTKEY_API_KEY"
    )
)

To switch to Azure as your provider, add your Azure details to Portley vault (here’s how) and use Azure OpenAI using virtual keys

llm = ChatOpenAI(
    api_key="api-key",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="azure-openai", #choose your provider
        api_key="PORTKEY_API_KEY",
        virtual_key="AZURE_VIRTUAL_KEY"   # Replace with your virtual key for Azure
    )
)

If you are using OpenAI with LangGraph, your code would look like this:

llm = ChatOpenAI(
    api_key="OpenAI_API_Key",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="openai", #choose your provider
        api_key="PORTKEY_API_KEY"
    )
)

To switch to Azure as your provider, add your Azure details to Portley vault (here’s how) and use Azure OpenAI using virtual keys

llm = ChatOpenAI(
    api_key="api-key",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="azure-openai", #choose your provider
        api_key="PORTKEY_API_KEY",
        virtual_key="AZURE_VIRTUAL_KEY"   # Replace with your virtual key for Azure
    )
)

If you are using Anthropic with LangGraph, your code would look like this:

llm = ChatOpenAI(
    api_key="ANTHROPIC_API_KEY",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="anthropic", #choose your provider
        api_key="PORTKEY_API_KEY"
    )
)

To switch to AWS Bedrock as your provider, add your AWS Bedrock details to Portley vault (here’s how) and use AWS Bedrock using virtual keys,

llm = ChatOpenAI(
    api_key="api-key",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="bedrock", #choose your provider
        api_key="PORTKEY_API_KEY",
        virtual_key="AWS_Bedrock_VIRTUAL_KEY"   # Replace with your virtual key for Bedrock
    )
)

2. Reliability

Agents are brittle. Long agentic pipelines with multiple steps can fail at any stage, disrupting the entire process. Portkey solves this by offering built-in fallbacks between different LLMs or providers, load-balancing across multiple instances or API keys, and implementing automatic retries and request timeouts. This makes your agents more reliable and resilient.

Here’s how you can implement these features using Portkey’s config

{
  "retry": {
    "attempts": 5
  },
  "strategy": {
    "mode": "loadbalance" // Choose between "loadbalance" or "fallback"
  },
  "targets": [
    {
      "provider": "openai",
      "api_key": "OpenAI_API_Key"
    },
    {
      "provider": "anthropic",
      "api_key": "Anthropic_API_Key"
    }
  ]
}

3. Metrics

Agent runs can be costly. Tracking agent metrics is crucial for understanding the performance and reliability of your AI agents. Metrics help identify issues, optimize runs, and ensure that your agents meet their intended goals.

Portkey automatically logs comprehensive metrics for your AI agents, including cost, tokens used, latency, etc. Whether you need a broad overview or granular insights into your agent runs, Portkey’s customizable filters provide the metrics you need. For agent-specific observability, add Trace-id to the request headers for each agent.

llm = ChatOpenAI(
    api_key="Anthropic_API_Key",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key="PORTKEY_API_KEY",
        provider="anthropic",
        trace_id="research_agent1" #Add individual trace-id for your agent analytics
    )
)

4. Logs

Agent runs are complex. Logs are essential for diagnosing issues, understanding agent behavior, and improving performance. They provide a detailed record of agent activities and tool use, which is crucial for debugging and optimizing processes.

Portkey offers comprehensive logging features that capture detailed information about every action and decision made by your AI agents. Access a dedicated section to view records of agent executions, including parameters, outcomes, function calls, and errors. Filter logs based on multiple parameters such as trace ID, model, tokens used, and metadata.

Auto-Instrumentation

Portkey also supports auto-instrumentation for LangGraph. This allows you to trace and log your Agents execution with minimal code changes.

Add the following to the top of your code execution to start tracing and logging your Agents execution to Portkey:

from portkey import Portkey

Portkey(api_key="{{PORTKEY_API_KEY}}", instrumentation=True)

5. Traces

With traces, you can see each agent run granularly on Portkey. Tracing your LangGraph agent runs helps in debugging, performance optimzation, and visualizing how exactly your agents are running.

Using Traces in LangGraph Agents

Import & Initialize the Portkey Langchain Callback Handler

from portkey_ai.langchain import LangchainCallbackHandler

portkey_handler = LangchainCallbackHandler(
    api_key="YOUR_PORTKEY_API_KEY",
    metadata={
        "session_id": "session_1",  # Use consistent metadata across your application
        "agent_id": "research_agent_1",  # Specific to the current agent
    }
)

Configure Your LLM with the Portkey Callback

from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI(
    api_key="YOUR_OPENAI_API_KEY_HERE",
    callbacks=[portkey_handler],
    # ... other parameters
)

With Portkey tracing, you can encapsulate the complete execution of your agent workflow.

6. Guardrails

LLMs are brittle - not just in API uptimes or their inexplicable 400/500 errors, but also in their core behavior. You can get a response with a 200 status code that completely errors out for your app’s pipeline due to mismatched output. With Portkey’s Guardrails, we now help you enforce LLM behavior in real-time with our Guardrails on the Gateway pattern.

Using Portkey’s Guardrail platform, you can now verify your LLM inputs AND outputs to be adhering to your specifed checks; and since Guardrails are built on top of our Gateway, you can orchestrate your request exactly the way you want - with actions ranging from denying the request, logging the guardrail result, creating an evals dataset, falling back to another LLM or prompt, retrying the request, and more.

7. Continuous Improvement

Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey’s Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.

8. Caching

Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.

{
 "cache": {
    "mode": "semantic" // Choose between "simple" or "semantic"
 }
}

9. Security & Compliance

Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.

Portkey Config

Many of these features are driven by Portkey’s Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.

For more information on using these features and setting up your Config, please refer to the Portkey documentation.

Ecosystem

LLMs

Vector Databases

Agents

Libraries

End-to-End Example

Integration Guide

Google Colab

Make your agents Production-ready with Portkey

1. Interoperability

2. Reliability

3. Metrics

4. Logs

Auto-Instrumentation

5. Traces

Using Traces in LangGraph Agents

6. Guardrails

7. Continuous Improvement

8. Caching

9. Security & Compliance

Portkey Config

Ecosystem

LLMs

Vector Databases

Agents

Libraries

​End-to-End Example

​Integration Guide

Google Colab

​Make your agents Production-ready with Portkey

​1. Interoperability

​2. Reliability

​3. Metrics

​4. Logs

​Auto-Instrumentation

​5. Traces

​Using Traces in LangGraph Agents

​6. Guardrails

​7. Continuous Improvement

​8. Caching

​9. Security & Compliance

​Portkey Config

End-to-End Example

Integration Guide

Make your agents Production-ready with Portkey

1. Interoperability

2. Reliability

3. Metrics

4. Logs

Auto-Instrumentation

5. Traces

Using Traces in LangGraph Agents

6. Guardrails

7. Continuous Improvement

8. Caching

9. Security & Compliance

Portkey Config