Portkey in October 🎃 🪔 October was packed with treats (no tricks!) for Portkey. As we celebrate Halloween and Diwali, we’re lighting up your AI infrastructure with some exciting updates. Let’s dive in!

​
Executive Summary

Guardrails GA ReleaseProduction-ready guardrails to enforce LLM behavior in real-time, with support for PII detection, moderation, and more — are now generally available. (Docs)
Enterprise MomentumRefreshed Portkey’s enterprise offering with enhanced security features, and support for AWS Assume Role Auth. Also onboarded one of the world’s largest tech companies to Portkey.
Provider EcosystemAdded 7 new providers including vLLM, Triton, Lambda Labs, and more.
Image GenerationAdded support for Stable Diffusion v3 and Google Imagen.
IntegrationsAdded MindsDB, ToolJet, LibreChat, and OpenWebUI.
Prompt CachingAnthropic’s prompt caching feature is now available directly in prompt playground. (Docs)
.NETYou can now integrate Portkey with your .NET app
Agent Tooling LeadershipPortkey was recognized for providing 11 critical capabilities for production-grade AI agents, leading the Agent Ops tooling benchmark.
Featured CoverageOur DevOps for AI vision featured in the People+AI Newsletter and Pulse2 publication.

​
Features

  • AWS Assume Role Support: Enhanced Bedrock authentication for enterprise security (Docs)
  • User Management API: New API to resend user invites (Docs). Also updated the API specs for Prompt Completions API, Prompt Render API, and Insert Log API
  • New OpenAI Param: OpenAI’s max_completion_tokens is now supported
  • Caching: Improved cost calculations for OpenAI & Azure OpenAI cached responses, and Anthropic’s prompt caching feature is now available directly in prompt playground
  • Gemini Updates: Added support for Gemini JSON mode and Controlled Generations along with Pydantic support
  • Bedrock: Integrated Converse API for /chat/completions. (Docs)
  • Enterprise: Refreshed Portkey’s enterprise offering with enhanced security features.
  • C# (.NET) Support: You can now integrate Portkey in your .NET apps using the OpenAI official library. (Docs)

​
Models & Providers

7 New Providers: Expanding your model hosting and deployment options.

Lemonfox

Lambda Labs

Dashscope

Upstage

Nvidia Triton

Github

vLLM

2 Image Generation Models: Strengthening our multimodal capabilities with next-gen image models.

Imagen on Google Vertex

Official support for Google’s Imagen model through Vertex AI
2 New LLMs:

Llama 3.2

Now integrated with Fireworks, AWS Bedrock, Groq, and Together AI

Vertex Embeddings

Added support for both English and Multilingual embedding models from Google Vertex AI

​
Integrations

Model Management & Monitoring: Enhance your AI infrastructure with enterprise-grade observability.

OpenWebUI

Portkey is the only plugin you’ll need for model management, cost tracking, observability, metadata logging, and more for your Open WebUI instance.
Data & App Integration: Connect your existing tools and databases to LLMs.

MindsDB

Connect your databases, vector stores, and apps to 250+ LLMs with enterprise-grade monitoring and reliability built-in.

ToolJet

Add AI-powered capabilities such as chat completions and automations into your ToolJet apps easily.

​
Guardrails

The guardrails feature is now generally available - it brings production-ready content filtering and response validation to your LLM apps. Updated Content Safety Guardrails:

PII Detection

Detect sensitive personal information in user messages

Content Moderation

Automated content filtering and moderation
Updated Guardrails to Ensure Response Quality:

Language Detection

Automatically detect and validate response languages

Gibberish Detection

Filter out nonsensical or low-quality responses
And More!

Custom Webhooks

Metadata sent to the Portkey API will now be automatically forwarded to your custom webhook endpoint.

Lowercase Detection

Check if the given string is lowercase or not.

​
Resources

Quick Implementation Guides: Technical Deep Dives for Production Deployments:

OpenAI Swarm + Portkey

Build and secure multi-agent AI systems using OpenAI Swarm and Portkey

RAG with Observability

Enhanced version of Anthropic’s RAG Cookbook with unified API and monitoring
Latest insights on AI infrastructure and tooling: Check out more technical content on our Blog →.

​
Fixes

Model & Provider Enhancements Fixed core provider issues and improved reliability:
  • Enhanced streaming transformer for Perplexity
  • Fixed response transformation for Ollama
  • ⭐️ Added missing logprob mapping for Azure OpenAI (Thanks Avishkar!)
  • Fixed token counting for Vertex embeddings (now using tokens instead of characters)
  • Added support for Bedrock cross-region model IDs with pricing
  • Fixed media file handling for Vertex AI & Gemini
Default Models We’ve also reset the for the following providers:
  • Fireworks: accounts/fireworks/models/llama-v3p1-405b-instruct
  • Together AI: meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo
  • Gemini: gemini-1.5-pro
Dev Ex Improvements
  • Added support for anthropic-beta and anthropic-version headers in the Portkey API
  • In Portkey SDK, the Portkey API key is now optional when you’re calling the self-hosted Gateway
  • Enhanced support for custom provider headers in SDK

​
Community Updates

Upcoming Events
Service Reliability When OpenAI users were hitting usage limits earlier this month, Portkey users remained unaffected thanks to our built-in reliability features. Industry Recognition Recent Events We co-sponsored the TED AI Hackathon! Thanks to everyone who participated and built amazing projects.

​
Support

Bug Report

Found a bug or have a feature request? Open an issue on our GitHub repository.

Join Portkey Discord

Collaborate with Industry Practitioners and get 24x7 support.