November
Portkey in November ❄️
We won the NetApp Excellerator Award, launched prompt.new for faster development, added folder organization and AI suggestions for prompt templates, introduced multi-workspace analytics.
Plus, there’s now support for OpenAI’s Realtime API and much more. Let’s dive in!
Summary
| Area | Key Updates |
|---|---|
| Platform | • See multi-workspace analytics & logs on a single dashboard • Support for Realtime API across OpenAI and Azure OpenAI • More granular security & access control settings • Organize your prompts in folders |
| Integrations | • Route to AWS Sagemaker models through Portkey • Support for xAI provider and Llama 3.3 & Gemini 2.0 Flash models • New strictOpenAiCompliance flag on the Gateway |
| Enterprise | • Support for AWS STS with IMDS/IRSA auth • Support for Azure Entra (formerly Active Directory) to manage Azure auth • Set budget limits with periodic resets • Support for any S3-compatible store for logging |
| Community | • Won NetApp’s Best Growth Strategy Award • Hosted first Practitioners Dinner in Singapore • Weekly AI Engineering Office Hours |
Enterprise Spotlight
When API Gateways Don’t Cut It
As AI infrastructure becomes increasingly critical for enterprises, technology leaders are choosing Portkey’s AI Gateway for their AI operations.
Akshay Darbari, Director of Platform Engineering at Premera Blue Cross
When Premera Blue Cross’ Director of Platform Engineering needed an AI Gateway, they chose Portkey. Why? Because traditional API gateways weren’t built for AI-first companies. Are you in the same boat? Schedule an expert consultation here.
Platform
Prompt Management
- Type prompt.new in your browser to spin up a new prompt playground! Try it now →
- Organize your prompt templates with folders and subfolders:
- Use AI to write and improve your prompts - right inside the playground:
- Add custom tags/labels like
staging,productionto any prompt version to track changes, and call them directly:
- Each response inside the playground now gives metrics to monitor LLM throughput and latency
Analytics
Org-wide Executive Reports
Monitor analytics and logs across all workspaces in your organization through a unified dashboard. This centralized view provides comprehensive insights into cost, performance, and accuracy metrics for your deployed AI applications.
- Track token usage patterns across requests & responses
- You can now filter logs and analytics with specific Portkey API keys. This is useful if you are tying a particular key to an internal user and want to see their usage!
- • Implement usage and rate limits on API & Virtual keys. Docs →
Enterprise
We’ve strengthened our enterprise authentication capabilities with comprehensive cloud provider integrations.
- Expanded AWS authentication options, for adding your Bedrock models or Sagemaker deployments:
- IMDS-based auth (recommended for AWS environments)
- IRSA-based auth for Kubernetes workloads
- Role-based auth for non-AWS environments
- STS integration with assumed roles
- Also expanded the Azure Integration:
- Azure Entra (formerly Active Directory)
- Managed identity support
- Granular access permissions for API Keys and Virtual Keys across your organization
- Support for sending Azure
deploymentConfigwhile making Virtual Keys through API. Docs
More
- Added the
strictOpenAiComplianceflag for supporting provider-specific parameters. Docs → - The virtual key API now supports adding direct Azure deployment configuration
More Customer Love
Felipe & team are building beconfident, and here’s what they had to say about Portkey:
“Now that we’ve seen positive results, we’re going to move all our prompts to Portkey.”
Integrations
Providers
AWS Sagemaker
Add your Sagemaker deployments to Portkey easily
xAI
Call Grok models through Portkey!
Ollama Tools
Tool calls are now supported on Ollama!
Vertex AI Controlled Generations
The Controlled Generations (read: Structured Outputs) feature on Vertex AI is now supported!
Libraries
OpenAI Swarm
Complete observability for Swarm agents
Supabase
Add LLM features to your Supabase apps
Semantic Kernel
Use Portkey in your Microsoft Semantic Kernel apps to easily observe your requests and make them reliable
Guardrails
Resources
Essential reading for your AI infrastructure:
- What is an LLM Gateway?: Complete introduction
- O1 Models Analysis: Understanding OpenAI’s latest
- LLM Gateway Guide: Making infrastructure choices
- Chat platform Comparison: LibreChat vs OpenWebUI
- AI vs API Gateway: Key differences
- FinOps for GenAI: Optimization strategies
Community
Office Hour
One thing we keep hearing from the Portkey community: you want to learn how other teams are solving production challenges and get the most out of the platform. Not through docs or tutorials, but through real conversations with fellow practitioners.
That’s why we’ve started a new series of AI Engineering Hours since last week to bring the Portkey community together to discuss exactly this!
Link to join the next office hour
Practitioners’ Dinner
We hosted some of Singapore’s leading Gen AI engineers & leaders for a roundtable conversation - one profound insight emerged: Companies serious about Gen AI have realized it’s as much a platform engineering challenge as it is an AI challenge.
Curious what we mean? Read the meetup note here.
Improvements
Providers
- Gemini: Enhanced message and media handling
- Bedrock: Improved message formatting
- Vertex AI: Added Zod validation
SDK
- Stream support for assistant threads
- Enhanced Pydantic compatibility
- Fixed semantic cache behavior
- Resolved Python Httpx proxy issues
Support
Special thanks to harupy and Ignacio Gleser for their contributions!
Was this page helpful?