Langchain (JS/TS) - Portkey Docs

This guide covers Langchain JavaScript/TypeScript. For Python, see Langchain Python.

Langchain provides a unified interface for building LLM applications. Add the AI Gateway to get production-grade features: full observability, automatic fallbacks, semantic caching, and cost controls—all without changing your Langchain code.

Quick Start

Add the AI Gateway to any Langchain app with 3 parameters:

import { ChatOpenAI } from "@langchain/openai";

const model = new ChatOpenAI({
    model: "@openai-prod/gpt-4o",           // Provider slug from Model Catalog
    configuration: {
        baseURL: "https://aigw.portkey.ai/v1"
    },
    apiKey: "PORTKEY_API_KEY"               // Your AI Gateway API key
});

const response = await model.invoke("Tell me a joke");
console.log(response.content);

That’s it! You now get:

✅ Full observability (costs, latency, logs)
✅ Dynamic model selection per request
✅ Automatic fallbacks and retries (via configs)
✅ Budget controls per team/project

Why Add the AI Gateway to Langchain?

Langchain handles application orchestration. The AI Gateway adds production features:

Enterprise Observability

Every request logged with costs, latency, tokens. Team-level analytics and debugging.

Dynamic Model Selection

Switch models per request. Route simple queries to cheap models, complex to advanced—automatically tracked.

Production Reliability

Automatic fallbacks, smart retries, load balancing—configured once, works everywhere.

Cost & Access Control

Budget limits per team/project. Rate limiting. Centralized credential management.

Setup

1. Install Packages

npm install @langchain/openai

2. Add Provider in Model Catalog

Go to Model Catalog → Add Provider
Select your provider (OpenAI, Anthropic, Google, etc.)
Choose existing credentials or create new by entering your API keys
Name your provider (e.g., openai-prod)

Your provider slug will be @openai-prod (or whatever you named it).

Complete Model Catalog Guide →

Set up budgets, rate limits, and manage credentials

3. Get AI Gateway API Key

Create your AI Gateway API key at stratacloudmanager.paloaltonetworks.com

4. Use in Your Code

Replace your existing ChatOpenAI initialization:

// Before (direct to OpenAI)
const model = new ChatOpenAI({
    model: "gpt-4o",
    apiKey: "OPENAI_API_KEY"
});

// After (via the AI Gateway)
const model = new ChatOpenAI({
    model: "@openai-prod/gpt-4o",
    configuration: {
        baseURL: "https://aigw.portkey.ai/v1"
    },
    apiKey: "PORTKEY_API_KEY"
});

That’s the only change needed! All your existing Langchain code (agents, chains, LCEL, etc.) works exactly the same.

Switching Between Providers

Just change the model string—everything else stays the same:

// OpenAI
const openaiModel = new ChatOpenAI({
    model: "@openai-prod/gpt-4o",
    configuration: { baseURL: "https://aigw.portkey.ai/v1" },
    apiKey: "PORTKEY_API_KEY"
});

// Anthropic
const anthropicModel = new ChatOpenAI({
    model: "@anthropic-prod/claude-sonnet-4",
    configuration: { baseURL: "https://aigw.portkey.ai/v1" },
    apiKey: "PORTKEY_API_KEY"
});

// Google Gemini
const geminiModel = new ChatOpenAI({
    model: "@google-prod/gemini-2.0-flash",
    configuration: { baseURL: "https://aigw.portkey.ai/v1" },
    apiKey: "PORTKEY_API_KEY"
});

The AI Gateway implements OpenAI-compatible APIs for all providers, so you always use ChatOpenAI regardless of which model you’re calling.

Using with Langchain Agents

Langchain agents are the primary use case. The AI Gateway works seamlessly with agent workflows:

import { ChatOpenAI } from "@langchain/openai";
import { tool } from "@langchain/core/tools";
import { createReactAgent } from "@langchain/langgraph/prebuilt";
import { z } from "zod";

// Define tools
const searchTool = tool(
    async ({ query }) => `Results for: ${query}`,
    {
        name: "search",
        description: "Search for information",
        schema: z.object({ query: z.string() })
    }
);

const weatherTool = tool(
    async ({ location }) => `Weather in ${location}: Sunny, 72°F`,
    {
        name: "get_weather",
        description: "Get weather for a location",
        schema: z.object({ location: z.string() })
}
);

// Create model with the AI Gateway
const model = new ChatOpenAI({
    model: "@openai-prod/gpt-4o",
    configuration: { baseURL: "https://aigw.portkey.ai/v1" },
    apiKey: "PORTKEY_API_KEY"
});

// Create agent
const agent = createReactAgent({
    llm: model,
    tools: [searchTool, weatherTool]
});

// Run agent
const result = await agent.invoke({
    messages: [{ role: "user", content: "What's the weather in NYC?" }]
});

Every agent step is logged in the AI Gateway:

Model calls with prompts and responses
Tool executions with inputs and outputs
Full trace of the agent’s reasoning
Costs and latency for each step

Works With All Langchain Features

✅ Agents - Full compatibility with LangGraph agents ✅ LCEL - LangChain Expression Language ✅ Chains - All chain types supported ✅ Streaming - Token-by-token streaming ✅ Tool Calling - Function/tool calling ✅ LangGraph - Complex workflows

Streaming

const model = new ChatOpenAI({
    model: "@openai-prod/gpt-4o",
    configuration: { baseURL: "https://aigw.portkey.ai/v1" },
    apiKey: "PORTKEY_API_KEY",
    streaming: true
});

const stream = await model.stream("Write a short story");
for await (const chunk of stream) {
    process.stdout.write(chunk.content);
}

Chains & Prompts

import { ChatOpenAI } from "@langchain/openai";
import { ChatPromptTemplate } from "@langchain/core/prompts";

const model = new ChatOpenAI({
    model: "@openai-prod/gpt-4o",
    configuration: { baseURL: "https://aigw.portkey.ai/v1" },
    apiKey: "PORTKEY_API_KEY"
});

const prompt = ChatPromptTemplate.fromMessages([
    ["human", "Tell me a short joke about {topic}"]
]);

const chain = prompt.pipe(model);
const response = await chain.invoke({ topic: "ice cream" });
console.log(response.content);

Tool Calling

import { ChatOpenAI } from "@langchain/openai";
import { z } from "zod";
import { tool } from "@langchain/core/tools";

const getWeather = tool(
    async ({ location }) => `Weather in ${location}: Sunny, 72°F`,
    {
        name: "get_weather",
        description: "Get current weather in a location",
        schema: z.object({
            location: z.string().describe("City and state, e.g. San Francisco, CA")
        })
    }
);

const model = new ChatOpenAI({
    model: "@openai-prod/gpt-4o",
    configuration: { baseURL: "https://aigw.portkey.ai/v1" },
    apiKey: "PORTKEY_API_KEY"
});

const modelWithTools = model.bindTools([getWeather]);
const response = await modelWithTools.invoke("What's the weather in NYC?");
console.log(response.tool_calls);

Dynamic Model Selection

For dynamic model routing based on query complexity or task type, use AI Gateway Configs with conditional routing:

import { ChatOpenAI } from "@langchain/openai";

// Define the routing config
const config = {
    strategy: {
        mode: "conditional",
        conditions: [
            {
                query: { "metadata.complexity": { "$eq": "simple" } },
                then: "cheap-model"
            },
            {
                query: { "metadata.complexity": { "$eq": "complex" } },
                then: "advanced-model"
            }
        ],
        default: "cheap-model"
    },
    targets: [
        {
            name: "cheap-model",
            override_params: { model: "@openai-prod/gpt-4o-mini" }
        },
        {
            name: "advanced-model",
            override_params: { model: "@openai-prod/o1" }
        }
    ]
};

const model = new ChatOpenAI({
    model: "gpt-4o",
    configuration: {
        baseURL: "https://aigw.portkey.ai/v1",
        defaultHeaders: { "x-portkey-config": JSON.stringify(config) }
    },
    apiKey: "PORTKEY_API_KEY"
});

// Route to cheap model
const response1 = await model.invoke("What is 2+2?", {
    metadata: { complexity: "simple" }
});

// Route to advanced model
const response2 = await model.invoke("Solve this differential equation...", {
    metadata: { complexity: "complex" }
});

All routing decisions are tracked in the AI Gateway with full observability—see which models were used, costs per model, and performance comparisons.

Conditional Routing Guide →

Learn more about conditional routing and advanced patterns

Advanced Features via Configs

For production features like fallbacks, caching, and load balancing, use AI Gateway Configs:

import { ChatOpenAI } from "@langchain/openai";

const model = new ChatOpenAI({
    model: "@openai-prod/gpt-4o",
    configuration: {
        baseURL: "https://aigw.portkey.ai/v1",
        defaultHeaders: {
            "x-portkey-config": "pc_your_config_id",  // Created in Strata Cloud Manager
        }
    },
    apiKey: "PORTKEY_API_KEY"
});

Example: Load Balancing

const config = {
    strategy: { mode: "loadbalance" },
    targets: [
        {
            override_params: { model: "@openai-prod/gpt-4o" },
            weight: 0.5
        },
        {
            override_params: { model: "@anthropic-prod/claude-sonnet-4" },
            weight: 0.5
        }
    ]
};

const model = new ChatOpenAI({
    model: "gpt-4o",
    configuration: {
        baseURL: "https://aigw.portkey.ai/v1",
        defaultHeaders: { "x-portkey-config": JSON.stringify(config) }
    },
    apiKey: "PORTKEY_API_KEY"
});

// Requests are distributed 50/50 between OpenAI and Anthropic
const response = await model.invoke("Hello!");

Learn About Configs →

Set up fallbacks, retries, caching, load balancing, and more

Embeddings

Create embeddings via the AI Gateway:

import { OpenAIEmbeddings } from "@langchain/openai";

const embeddings = new OpenAIEmbeddings({
    model: "@openai-prod/text-embedding-3-small",
    configuration: {
        baseURL: "https://aigw.portkey.ai/v1",
    },
    apiKey: "PORTKEY_API_KEY"
});

const vectors = await embeddings.embedDocuments(["Hello world", "Goodbye world"]);
console.log(vectors);

The AI Gateway supports OpenAI embeddings via OpenAIEmbeddings. For other providers (Cohere, Voyage), call the embeddings endpoint directly.

Migration from Direct OpenAI

Already using Langchain with OpenAI? Just update 3 parameters:

// Before
import { ChatOpenAI } from "@langchain/openai";

const model = new ChatOpenAI({
    model: "gpt-4o",
    apiKey: process.env.OPENAI_API_KEY,
    temperature: 0.7
});

// After (add configuration, change model and apiKey)
const model = new ChatOpenAI({
    model: "@openai-prod/gpt-4o",            // Add provider slug
    configuration: {
        baseURL: "https://aigw.portkey.ai/v1"     // Add this
    },
    apiKey: "PORTKEY_API_KEY",                // Change to AI Gateway API key
    temperature: 0.7                          // Keep existing params
});

Benefits:

Zero code changes to your existing Langchain logic
Instant observability for all requests
Production-grade reliability features
Cost controls and budgets

Next Steps

Model Catalog

Set up providers, budgets, and access control

Configs

Configure fallbacks, caching, and routing

Observability

Track costs, performance, and usage

Guardrails

Add PII detection and content filtering

​Quick Start

​Why Add the AI Gateway to Langchain?

Enterprise Observability

Dynamic Model Selection

Production Reliability

Cost & Access Control

​Setup

​1. Install Packages

​2. Add Provider in Model Catalog

Complete Model Catalog Guide →

​3. Get AI Gateway API Key

​4. Use in Your Code

​Switching Between Providers

​Using with Langchain Agents

​Works With All Langchain Features

​Streaming

​Chains & Prompts

​Tool Calling

​Dynamic Model Selection

Conditional Routing Guide →

​Advanced Features via Configs

​Example: Load Balancing

Learn About Configs →

​Embeddings

​Migration from Direct OpenAI

​Next Steps

Model Catalog

Configs

Observability

Guardrails

Quick Start

Why Add the AI Gateway to Langchain?

Setup

1. Install Packages

2. Add Provider in Model Catalog

3. Get AI Gateway API Key

4. Use in Your Code

Switching Between Providers

Using with Langchain Agents

Works With All Langchain Features

Streaming

Chains & Prompts

Tool Calling

Dynamic Model Selection

Advanced Features via Configs

Example: Load Balancing

Embeddings

Migration from Direct OpenAI

Next Steps