Introduction
LangGraph is a library for building stateful, multi-actor applications with LLMs, designed to make developing complex agent workflows easier. It provides a flexible framework to create directed graphs where nodes process information and edges define the flow between them. The AI Gateway enhances LangGraph with production-readiness features, turning your experimental agent workflows into robust systems by providing:- Complete observability of every agent step, tool use, and state transition
- Built-in reliability with fallbacks, retries, and load balancing
- Cost tracking and optimization to manage your AI spend
- Access to 3,000+ LLMs through a single integration
- Guardrails to keep agent behavior safe and compliant
- Version-controlled prompts for consistent agent performance
LangGraph Official Documentation
Installation & Setup
Install the required packages
- For search capabilities:
pip install langchain_community - For memory functionality:
pip install langgraph[checkpoint]
Generate API Key
Configure LangChain with the AI Gateway
Basic Agent Implementation
Let’s create a simple LangGraph chatbot using the AI Gateway. This example shows how to set up a basic conversational agent:- Creates a state graph with a message history
- Configures a ChatOpenAI model with the AI Gateway
- Defines a simple chatbot node that processes messages with the LLM
- Compiles the graph and provides a streaming interface for chat
Advanced Features
1. Adding Tools to Your Agent
LangGraph can be enhanced with tools to allow your agent to perform actions. Here’s how to add the Tavily search tool:2. Creating Custom Tools
You can create custom tools for your agents using the@tool decorator. Here’s how to create a simple multiplication tool:
- Defines a Pydantic model for the tool’s input schema
- Creates a custom multiplication tool with the
@tooldecorator - Integrates it into LangGraph with a tool node
3. Adding Memory to Your Agent
For persistent conversations, you can add memory to your LangGraph agents:thread_id in the config allows you to maintain separate conversation threads for different users or contexts.Production Features
1. Enhanced Observability
The AI Gateway provides comprehensive observability for your LangGraph agents, helping you understand exactly what’s happening during each execution.- Traces
- Logs
- Metrics & Dashboards
- Metadata Filtering
2. Reliability - Keep Your Agents Running Smoothly
When running agents in production, things can go wrong - API rate limits, network issues, or provider outages. The AI Gateway’s reliability features ensure your agents keep running smoothly even when problems occur. Enable fallback in your LangGraph agents by using an AI Gateway Config: This configuration will automatically try Claude if the GPT-4o request fails, ensuring your agent can continue operating.Automatic Retries
Request Timeouts
Conditional Routing
Fallbacks
Load Balancing
3. Prompting in LangGraph
The AI Gateway’s Prompt Engineering Studio helps you create, manage, and optimize the prompts used in your LangGraph agents. Instead of hardcoding prompts or instructions, use the AI Gateway’s prompt rendering API to dynamically fetch and apply your versioned prompts.- Prompt Playground
- Using Prompt Templates
- Prompt Versioning
- Mustache Templating for variables
- Iteratively develop prompts before using them in your agents
- Test prompts with different variables and models
- Compare outputs between different prompt versions
- Collaborate with team members on prompt development
4. Guardrails for Safe Agents
Guardrails ensure your LangGraph agents operate safely and respond appropriately in all situations. Why Use Guardrails? LangGraph agents can experience various failure modes:- Generating harmful or inappropriate content
- Leaking sensitive information like PII
- Hallucinating incorrect information
- Generating outputs in incorrect formats
- Detect and redact PII in both inputs and outputs
- Filter harmful or inappropriate content
- Validate response formats against schemas
- Check for hallucinations against ground truth
- Apply custom business logic and rules
Learn More About Guardrails
5. User Tracking with Metadata
Track individual users through your LangGraph agents using the AI Gateway’s metadata system. What is Metadata in the AI Gateway? Metadata allows you to associate custom data with each request, enabling filtering, segmentation, and analytics. The special_user field is specifically designed for user tracking.
Filter Analytics by User
With metadata in place, you can filter analytics by user and analyze performance metrics on a per-user basis:
This enables:
- Per-user cost tracking and budgeting
- Personalized user analytics
- Team or organisation-level metrics
- Environment-specific monitoring (staging vs. production)
Learn More About Metadata
6. Caching for Efficient Agents
Implement caching to make your LangGraph agents more efficient and cost-effective:- Simple Caching
- Semantic Caching
7. Model Interoperability
LangGraph works with multiple LLM providers, and the AI Gateway extends this capability by providing access to over 200 LLMs through a unified interface. You can easily switch between different models without changing your core agent logic: The AI Gateway provides access to LLMs from providers including:- OpenAI (GPT-4o, GPT-4 Turbo, etc.)
- Anthropic (Claude 3.5 Sonnet, Claude 3 Opus, etc.)
- Mistral AI (Mistral Large, Mistral Medium, etc.)
- Google Vertex AI (Gemini 1.5 Pro, etc.)
- Cohere (Command, Command-R, etc.)
- AWS Bedrock (Claude, Titan, etc.)
- Local/Private Models
Supported Providers
Set Up Enterprise Governance for LangGraph
Why Enterprise Governance? If you are using LangGraph inside your organisation, you need to consider several governance aspects:- Cost Management: Controlling and tracking AI spending across teams
- Access Control: Managing which teams can use specific models
- Usage Analytics: Understanding how AI is being used across the organisation
- Security & Compliance: Maintaining enterprise security standards
- Reliability: Ensuring consistent service across all users
Create Integration
Create Config
- Go to Configs in Strata Cloud Manager
- Create new config with:
- Save and note the Config name for the next step
Configure AI Gateway API Key
- Go to API Keys in the AI Gateway and Create new API key
- Select your config from
Step 2 - Generate and save your API key
Connect to LangGraph
Step 1: Implement Budget Controls & Rate Limits
Step 1: Implement Budget Controls & Rate Limits
Step 1: Implement Budget Controls & Rate Limits
Integrations enable granular control over LLM access at the team/department level. This helps you:- Set up budget limits
- Prevent unexpected usage spikes using Rate limits
- Track departmental spending
Setting Up Department-Specific Controls:
- Navigate to Integrations in Strata Cloud Manager and create a new Integration.
- Provision this Integration for each department with their budget limits and rate limits
- Configure model access if required.
Step 2: Define Model Access Rules
Step 2: Define Model Access Rules
Step 2: Define Model Access Rules
As your AI usage scales, controlling which teams can access specific models becomes crucial. AI Gateway Configs provide this control layer with features like:Access Control Features:
- Model Restrictions: Limit access to specific models
- Data Protection: Implement guardrails for sensitive data
- Reliability Controls: Add fallbacks and retry logic
Example Configuration:
Here’s a basic configuration to route requests to OpenAI, specifically using GPT-4o:Step 3: Implement Access Controls
Step 3: Implement Access Controls
Step 3: Implement Access Controls
Create User-specific API keys that automatically:- Track usage per user/team with the help of metadata
- Apply appropriate configs to route requests
- Collect relevant metadata to filter logs
- Enforce access permissions
Step 4: Deploy & Monitor
Step 4: Deploy & Monitor
Step 4: Deploy & Monitor
After distributing API keys to your team members, your enterprise-ready LangGraph setup is ready to go. Each team member can now use their designated API keys with appropriate access levels and budget controls.Monitor usage in Strata Cloud Manager:- Cost tracking by department
- Model usage patterns
- Request volumes
- Error rates
Enterprise Features Now Available
Your LangGraph integration now has:- Departmental budget controls
- Model access governance
- Usage tracking & attribution
- Security guardrails
- Reliability features
Frequently Asked Questions
How does the AI Gateway enhance LangGraph?
How does the AI Gateway enhance LangGraph?
Can I use the AI Gateway with existing LangGraph applications?
Can I use the AI Gateway with existing LangGraph applications?
Does the AI Gateway work with all LangGraph features?
Does the AI Gateway work with all LangGraph features?
How do I filter logs and traces for specific graph runs?
How do I filter logs and traces for specific graph runs?
graph_id, workflow_type, or session_id to easily find and analyze specific graph executions.Can I use LangGraph's memory features with the AI Gateway?
Can I use LangGraph's memory features with the AI Gateway?
MemorySaver checkpointer with gateway-enabled LLMs. All the memory and state management features work seamlessly with the AI Gateway.Resources
LangGraph Docs
Official LangGraph documentation
AI Gateway Docs
Official AI Gateway documentation

