Portkey in September

Portkey is growing, is old news (and also current news).

Amid all the growth, last month, we celebrated a different milestone: consistent customer love.

It warms our heart when one of the world's largest pharmaceuticals companies says this about Portkey:

It has been our privilege to deploy Portkey for some of the world's largest and most respected companies, and if you've already come across Portkey and haven't talked to us, to you I say — what are you waiting for?

Now, onto everything we shipped this month:

Security Takes Center Stage

We've added security settings in the admin section to let you create the right permissions for all members in your org. You can now decide who can view, edit Portkey API keys, virtual keys, and more.

We're enabling for orgs on a rolling basis. Just email us on [email protected] and we'll enable it for your org.

SSO Settings in the UI

If you're using Okta, Azure or any other OIDC SSO, you can now configure Portkey to use your own SSO, right from the admin UI.

We now also support multiple domain connections, automatic sign-ins for new emails, and multiple auth systems!

We Are Joining Hacktoberfest

Make improvements, add new features to our open source project, and get awesome Portkey swag!

New APIs

With a focus on easier administration, we launched 2 new APIs last month:

Analytics: Get time series data for 17 metrics, as well as grouped and summary data across your users, models used, and metadata. API Docs.

Workspaces: If you are using multiple workspaces on Portkey, this API lets you easily create, update, delete your workspace settings and manage, add, delete Workspace members. API Docs.

To enable these APIs for your org,

✨ New Features & Enhancements

1. Anthropic & OpenAI Prompt Caching Support: Save upto 50% cost using prompt caching now enabled on your Portkey requests. (OpenAI, Anthropic)
2. Structured Outputs for Gemini: Have Gemini follow your given JSON schemas in the familiar OpenAI style!
3. Extended Media Support for Gemini: Portkey now supports mp4, pdf, jpg, mp3, and more media file types natively for your Gemini requests. (docs)
4. Vertex AI Multimodal Embedding Models:  Use Vertex's powerful multimodal models through Portkey. (docs)
5. Llama 3.2: We support the latest Llama model via Fireworks, Together AI, Groq, and AWS Bedrock. The special tokens for Llama models are now supported on Bedrock-hosted models.
6. o1 models: Get accurate costs & latency details for all your o1-preview & o1-mini calls on Portkey.
7. 6 New AI Providers: Deepseek AI (makers of Deepseek-coder models), Sambanova, Deepgram's voice API, Lambda Labs, Upstage AI, and Inference.net
8. Improved JSON Guardrails: Improved functionality in JSON checks for validating JSON within code blocks, and verifying multiple JSON objects in a single response. (docs)

🤝 New Integrations & Partnerships

• MongoDB: Use MongoDB for data storage and vector embeddings along with Portkey's LLMOps platform to build robust AI apps. (guide)
• LibreChat: Use Portkey as a provider in LibreChat to get complete observability over all your requests, and make them robust & reliable with our fallback & loadbalancing features. (read more here)
• Koyeb: Deploy Portkey Gateway in minutes, at production scale with this guide. (link to guide)

🙏 Celebrating Contributors

Thrilled to shine a spotlight on the amazing individuals who've contributed to Portkey Gateway this month:

• Sterne Lee added Deepbricks, Siliconflow providers
• Allan fixed a bundling issue with the NPM package
• ilsubyeega added SambaNova provider
• Shaunak wrote up the new allUppercase guardrail
• James made a critical PR that combines system messages into one for Anthropic requests
• Avishkar made a fix to map logprob params for Azure OpenAI calls
• Ignacio added support for embedding models available on Vertex AI

Thanks so much for making the Gateway a whole lot better!

📖 New Guides

• Anthropic recently released a fantastic guide on contextual RAG. Here it is with Portkey features enabled: Link to cookbook
• Here's how you enforce PII guidelines in your chatbot (using Langchain): Link to cookbook
• We wrote up a report on the novel "MAMBA" architecture behind the recent Mistral model Codestral. Link to blog
• We benchmarked the new Moderations endpoint from OpenAI and found it to be significantly better than its predecessor. Hot tip: If you are sending request to the old moderations model, it's high time to switch to this one! Link to the benchmark
• We also wrote about what it takes to put a framework like DSPy in production. Link to the guide.

Did You Know?

A few days ago, a lot of OpenAI users randomly got "Usage limit reached" errors. To tackle such scenarios, it helps to have fallbacks setup for your requests using Portkey.

Also, good news: The Gemini API latencies have come down 3x since June this year: link

Community Events

• We hosted a lunch with Neon DB to discuss Embeddings for RAG: Photos
• Team Portkey attended the Magic Ball AI event in Bangalore to demo all the new things we've been cooking: Link
• Portkey's co-founder Rohit was joined Bala to talk about our journey of building Portkey so far. It was beautiful. Watch it here.

More Customer Love

That's it this month! Follow Portkey on X for regular updates.