Enterprise Architecture
Comprehensive guide to Portkey’s hybrid deployment architecture for enterprises
Portkey Enterprise offers a secure hybrid deployment model that balances security, flexibility, and fast deployment timelines:
- Data Plane runs within your VPC, keeping sensitive LLM data and AI traffic in your environment
- Control Plane hosted by Portkey handles administration, configs, and analytics
Portkey Enterprise Hybrid Deployment Architecture
Schedule an Enterprise Architecture Demo
Want to learn more about our hybrid deployment model? Schedule a personalized demo with our solutions team to see how Portkey Enterprise can fit your security and compliance requirements.
Core Architecture Components
Data Plane (Your VPC)
The Data Plane is deployed in your cloud environment and processes all your AI traffic:
| Component | Description | Security Benefit |
|---|---|---|
| AI Gateway | Core engine that routes traffic across LLM providers and implements metering, access control, and guardrails | All LLM requests remain in your network perimeter |
| Cache Store | Local cache storage for gateway consumption | Eliminates runtime dependency on Control Plane |
| Data Store | Storage for LLM request/response logs | Keep sensitive LLM data completely in your environment |
The AI Gateway runs as containerized workloads in your infrastructure, deployable via your preferred orchestration method (Kubernetes, ECS, etc.).
Control Plane (Portkey VPC)
The Control Plane is fully managed by Portkey and provides the administrative layer for your deployment:
- Hosts the web dashboard for managing configurations, tracking analytics, and viewing logs
- Maintains routing configs, provider integrations
- Stores non-sensitive metadata and aggregated metrics
- Automatically updates with new features and provider integrations without requiring changes to your infrastructure
Data Flow Between Planes
Deployment Architecture
Portkey AI Gateway is deployed as containerized workloads using Helm charts for Kubernetes environments, with flexible deployment options for various cloud providers.
Infrastructure Components
| Component | Description | Configuration Options |
|---|---|---|
| AI Gateway | Core container running the routing logic | Deployed as stateless containers that can scale horizontally |
| Cache System | Stores routing configs, virtual keys, and more | Redis (in-cluster, AWS ElastiCache, or custom endpoint) |
| Log Storage | Persistence for request/response data | Multiple options (see below) |
Storage Options
S3-compatible storage options including:
- AWS S3 (standard credentials or assumed roles)
- Google Cloud Storage (S3 compatible interoperability mode)
- Azure Blob Storage (key, managed identity, or Entra ID)
- Any S3-compatible Blob Storage
S3-compatible storage options including:
- AWS S3 (standard credentials or assumed roles)
- Google Cloud Storage (S3 compatible interoperability mode)
- Azure Blob Storage (key, managed identity, or Entra ID)
- Any S3-compatible Blob Storage
MongoDB/DocumentDB for structured log storage with options for:
- Direct connection string with username/password
- Certificate-based authentication (PEM)
- Connection pooling configurations
Authentication Methods
- IAM roles for service accounts (IRSA) in Kubernetes
- Instance Metadata Service (IMDS) for EC2/ECS
- Managed identities in Azure environments
- IAM roles for service accounts (IRSA) in Kubernetes
- Instance Metadata Service (IMDS) for EC2/ECS
- Managed identities in Azure environments
- Standard access/secret keys
- Certificate-based authentication
- JWT-based authentication
Infrastructure Requirements
- Kubernetes Cluster: K8s 1.20+ with Helm 3.x
- Outbound Network: HTTPS access to Control Plane endpoints
- Container Registry Access: For pulling gateway container images
- Recommended Resource Requirements:
- CPU: 1-2 cores per gateway instance
- Memory: 2-4GB per gateway instance
- Storage: Dependent on logging configuration
Data Security & Encryption
Your Sensitive Data Stays in Your VPC
- All prompt content and LLM responses remain within your network
- Only anonymized metrics data cross network boundaries
- Log storage location is configurable based on your requirements
Your Sensitive Data Stays in Your VPC
- All prompt content and LLM responses remain within your network
- Only anonymized metrics data cross network boundaries
- Log storage location is configurable based on your requirements
Multi-layered Encryption Approach
- All data in the Control Plane is encrypted at rest
- Communication between planes uses TLS 1.3 encryption in transit
- Sensitive data uses envelope encryption
- Optional BYOK (Bring Your Own Key) support with AWS KMS integration
Defense-in-Depth Security Model
- Network-level controls limit Control Plane access to authorized IPs
- Role-based access control for administrative functions
- Audit logging of all administrative actions
- Access tokens are short-lived with automatic rotation
Advantages of Hybrid Architecture
| Benefit | Technical Implementation | Business Value |
|---|---|---|
| Security & Compliance | - Sensitive data never leaves VPC - Configurable encryption methods - Flexible authentication options | - Meets data residency requirements - Supports regulated industries - Simplifies security reviews |
| Operational Efficiency | - No database management overhead - Automatic model config updates - Horizontally scalable architecture | - Low operational burden - Always up-to-date with LLM ecosystem - Scales with your traffic patterns |
| Deployment Flexibility | - Kubernetes-native deployment - Support for major cloud providers - Multiple storage backend options | - Fits into existing infrastructure - Avoids vendor lock-in - Customizable to specific needs |
| Developer Experience | - OpenAI-compatible API - Simple integration patterns - Comprehensive observability | - Minimal code changes needed - Smooth developer onboarding - Full visibility into system behavior |
Technical Rationale
Sample Files
These samples demonstrate the typical data patterns flowing between systems:
Resources & Next Steps
Have Questions?
Our solution architects are available to discuss your specific deployment requirements and security needs.
Schedule Architecture Discussion
Book a personalized consultation with our enterprise team to explore how Portkey’s architecture can be tailored to your organization’s specific requirements.
Was this page helpful?