• Guardrails are now GA: Our guardrails feature is now generally available, helping you enforce LLM behavior in realtime. (Docs)
• Enterprise Updates: Refreshed our enterprise offering and welcomed one of the world’s largest tech companies to the Portkey family.
• Featured in Media: Check out our why we’re building DevOps for AI in the People+AI Newsletter and our CEO’s interview on Pulse2.
• Anthropic Prompt Caching in Playground: For Anthropic models, you can now enable any message to be cached right on the prompt playground.
• Portkey Tops Agent Ops Tooling Benchmark: Portkey provides 11 critical capabilities to put agents in production. Link

• AWS Auth: AWS Assume Role Support for Bedrock. (Docs)
• Resend User Invite API: Launched a new API to re-invite a user to your org. (Docs)
• New API Specs: We also added API specs for: Prompt Completions API, Prompt Render API, and Inserting Logs API
• .NET (C#) Support: See how you can integrate Portkey in your .NET app easily using the OpenAI library and get advanced monitoring, routing, and enterprise features. (Docs)
• New OpenAI Param: OpenAI’s max_completion_tokens is now supported across all providers.
• Calculating Cached Requests:Updated cost calculations for cached responses for both OpenAI & Azure OpenAI.
• JSON Mode: We now support JSON mode for Gemini models.
• Controlled Generations: Gemini Controlled Generations is now supported on Portkey (alnog with support for Pydantic!)
• Bedrock Converse API: We’ve integrated Bedrock’s Converse API for all /chat/completions requests

We added 6 new providers on the Gateway this month!

In true multimodal fashion, we added the all new SDv3 model across a bunch of providers, as well as Google’s Imagen model, and more!

As we continued to develop more Guardrails

And also pushed out some really important integrations that help everyone from IT Admins to SDEs, to Indie hackers!

2-Min Guides

• Guide to Prompt Caching link
• Building Prod-Ready Apps with Vercel link
• OpenAI Swarm Cheat Sheet link

Longer Guides

• How to Build Multi-Agent AI Systems with OpenAI Swarm & Secure Them Using Portkey link
• We modified Anthropic’s RAG Cookbook to have Observability and unified API link
• End-to-End Guide for Using Vercel with Portkey link
• What is Automated Prompt Engineering? link
• OpenAI’s Prompt Caching: A Deep Dive link
• The Complete Guide to Prompt Engineering link
• Multi-Agent AI Systems: OpenAI Swarm link
• The Developer’s Guide to Opentelemetry link

More awesome content here.

• Enhanced streaming transformer for Perplexity
• Fixed response transformation for Ollama
• ⭐️ Added missing logprob mapping for Azure OpenAI (Community Contribution by Avishkar
• Token counting is now fixed for Vertex embeddings (we now count tokens instead of characters)
• Added default models for Gemini, Together AI, Fireworks AI.
  • Fireworks: accounts/fireworks/models/llama-v3p1-405b-instruct
  • Together AI: meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo
  • Gemini: gemini-1.5-pro
• Support for cross-region model IDs for Bedrock including cost calculations - https://github.com/Portkey-AI/gateway/pull/641 - Pricing for Bedrock cross-region model IDs- https://github.com/Portkey-AI/Winky/pull/278/files
• Fixed response transform for Ollama
• anthropic-beta and anthropic-version headers are now supported
• Fix for sending media files for Vertex AI & Gemini
• Support for additional headers from providers - https://github.com/Portkey-AI/portkey-python-sdk/pull/222/files
• API key is optional when using self-hosted Gateway with the Portkey SDK

• TED: We co-sponsored the TED AI Hackathon!
• LLMs in Prod Dinner: Singapore - We are organising a closed-door dinner with some of the leading tech executives in Singapore, on the preset of OpenAI Dev Day. Link to Register

• OpenAI users were hitting usage limits earlier this month. Though, if you were on Portkey, you should have been fine.