March
Introducing the Prompt Engineering Studio! 🧪✨
March brings the official launch of our highly anticipated Prompt Engineering Studio – a comprehensive platform for creating, testing, and deploying production-ready prompts with confidence.
We’re also excited to announce that Portkey is now being evaluated as the official AI Gateway solution by several prestigious universities, including Harvard, Princeton, and UC Berkeley.
Additionally, we’ve expanded our multimodal capabilities with Claude image support, added PDF uploads, and introduced thinking mode across major providers. All this with enhanced enterprise security through AWS KMS integration and SCIM for identity management.
Let’s explore all that’s new:
Our flagship release this month is the official launch of the Prompt Engineering Studio, bringing professional-grade prompt development to teams of all sizes:
Technical Improvements
We’ve made significant improvements to our documentation:
“Describing Portkey as merely useful would be an understatement; it’s a must-have.” - @AManInTech
Summary
| Area | Key Updates |
|---|---|
| Platform | • Prompt Engineering Studio official launch • Support for PDF uploads to Claude • Thinking mode across major providers • University evaluations across Ivy League institutions • 1-click AWS EC2 deployment with CloudFormation |
| Gateway | • Multimodal support for Claude (images via URL) • New providers: ncompass and Snowflake Cortex • Enhanced grounding with cached streaming • Improved retry handling and error detection |
| Security | • Bring your own encryption key with AWS KMS • SCIM integration for Okta & Azure Entra (AD) • Org-level guardrail and metadata enforcement • Email notifications for usage limits |
| Guardrails | • AWS Bedrock Guardrails integration • Mistral Moderations endpoint support • New Guardrail provider: Lasso • New input/output guardrails format |
| Documentation | • Admin API documentation • Updated Enterprise Architecture specs • Prompt documentation revamp • Enterprise code visibility in API docs |
Platform
Prompt Engineering Studio- Version control: Track changes, compare versions, and roll back when needed
- Collaborative workflow: Work together with your team on prompt development
- Variables & templates: Create reusable prompt components and patterns
- Testing framework: Validate performance before production deployment
- Production integration: Seamlessly connect to your applications
- Send image URLs to Claude via Anthropic, Vertex, or Bedrock APIs
- Full support for multimodal conversations and analysis
- Consistent interface across all Claude providers
- Send PDF files directly to Claude requests
- Process long-form documents without manual extraction
- Maintain formatting and structure in analysis
- Support for Anthropic (Bedrock, Vertex), OpenAI, and more
- Full compatibility with streaming responses
- Complete observability of reasoning process
- Consistent interface across all supported models
Enterprise
University Validation We’re proud to announce that Portkey is being evaluated as the official AI Gateway solution by leading academic institutions:- Harvard University
- Princeton University
- University of California, Berkeley
- Cornell University
- New York University
- Lehigh University
- Bowdoin College
- AWS KMS Integration: Bring your own encryption keys for maximum security
- SCIM Support: Automated user provisioning with Okta & Azure Entra (AD)
- Organizational Controls: Enforce guardrails and metadata requirements at the org level
- Usage Limit Notifications: Configure email alerts for rate/budget/usage thresholds
- CloudFormation Template: 1-click deployment of Portkey Gateway on AWS EC2
- Real-Time Model Pricing: Pricing configs now fetched dynamically from control plane
- Internal POD Communication: Secure HTTPS between components
- Enhanced Metrics: Track last byte latency for streaming responses
Gateway & Providers
New ProvidersSnowflake Cortex
Access Snowflake’s AI capabilities through the unified Portkey interface
ncompass
Integration with ncompass AI services
- Enhanced Retry Handling: Better detection of errors in retry process
- Improved Tool Support: Fixed handling of null content for Bedrock tool_calls
- Cached Grounding: Support for cached streaming in grounding requests
- Search Parameters: Support for perplexity.ai search options
- Webhook Enhancement: Return appropriate status codes for streaming webhook failures
Guardrails
We’ve significantly expanded our guardrails capabilities:- AWS Bedrock Guardrails: Native integration with AWS content filtering
- Mistral Moderations: Added support for Mistral’s moderation endpoint
- Lasso Integration: New provider for enhanced content safety
- Input/Output Format: New standardized format for setting guardrails
- Default Headers: Simplified configuration through new API & SDK headers
Documentation
Admin API Introduction
- Admin API Docs: Comprehensive guide to our Control Plane API
- Enterprise Architecture: Updated deployment architecture
- Enterprise Code Visibility: API docs now show code for enterprise deployments
- Prompt Documentation: Complete revamp of our prompt engineering guides
- New Cookbook: Building an LLM as a judge
SDK Updates
- Custom Headers: Send headers with
extra_headersparam in any method - Private Deployment Tracing: Instrument LlamaIndex/LangChain with private deployments
- Support for OpenAI Developer Role: Full compatibility with OpenAI’s new permissions
Analytics
New filtering capabilities in logs & analytics dashboards:- Filter requests by cache status:
- Cache Hit
- Cache Miss
- Cache Disabled
- Cache Semantic Hit
Community