Portkey Enterprise offers a secure hybrid deployment model that balances security, flexibility, and fast deployment timelines:
  • Data Plane runs within your VPC, keeping sensitive LLM data and AI traffic in your environment
  • Control Plane hosted by Portkey handles administration, configs, and analytics
Portkey's Hybrid Deployment Architecture showing Data Plane in customer VPC and Control Plane in Portkey VPC

Portkey Enterprise Hybrid Deployment Architecture

Schedule an Enterprise Architecture Demo

Want to learn more about our hybrid deployment model? Schedule a personalized demo with our solutions team to see how Portkey Enterprise can fit your security and compliance requirements.

Core Architecture Components

Data Plane (Your VPC)

The Data Plane is deployed in your cloud environment and processes all your AI traffic:
ComponentDescriptionSecurity Benefit
AI GatewayCore engine that routes traffic across LLM providers and implements metering, access control, and guardrailsAll LLM requests remain in your network perimeter
Cache StoreLocal cache storage for gateway consumptionEliminates runtime dependency on Control Plane
Data StoreStorage for LLM request/response logsKeep sensitive LLM data completely in your environment
The AI Gateway runs as containerized workloads in your infrastructure, deployable via your preferred orchestration method (Kubernetes, ECS, etc.).

Control Plane (Portkey VPC)

The Control Plane is fully managed by Portkey and provides the administrative layer for your deployment:
  • Hosts the web dashboard for managing configurations, tracking analytics, and viewing logs
  • Maintains routing configs, provider integrations
  • Stores non-sensitive metadata and aggregated metrics
  • Automatically updates with new features and provider integrations without requiring changes to your infrastructure

Data Flow Between Planes

All LLM traffic stays within your network boundary:
  1. Your application sends requests to the AI Gateway
  2. The Gateway processes the request (applying routing, caching, guardrails)
  3. The Gateway forwards the request to the appropriate LLM provider
  4. Responses from LLMs return through the same path
Security Benefit: Complete isolation of sensitive prompt data and responses
The AI Gateway periodically synchronizes with the Control Plane:
  • Frequency: 30-second heartbeat intervals
  • Data Retrieved: Prompt templates, routing configs, virtual keys, API keys
  • Process: Data is fetched, decrypted locally, and stored in the Gateway cache
  • Resilience: Gateway operates independently between syncs using cached configs
Security Benefit: Continuous operation even during Control Plane disconnection
The Gateway sends anonymized metrics to the Analytics Store:
  • Data Sent: Non-sensitive operational metrics (model used, token counts, response times)
  • Purpose: Powers analytics dashboards for monitoring performance and costs
  • Example: View sample analytics data
Security Benefit: Provides insights without exposing sensitive information
Option A: Logs in Your VPC (Recommended for high-security environments)
  • Logs stored in your environment’s Blob Store
  • When viewing logs in Dashboard UI, Control Plane requests them from Gateway
Option B: Logs in Portkey Cloud
  • Gateway encrypts and sends logs to Portkey Log Store
  • No connections required from Portkey to your environment for viewing logs
Security Benefit: Flexibility to match your compliance requirements

Deployment Architecture

Portkey AI Gateway is deployed as containerized workloads using Helm charts for Kubernetes environments, with flexible deployment options for various cloud providers.

Infrastructure Components

ComponentDescriptionConfiguration Options
AI GatewayCore container running the routing logicDeployed as stateless containers that can scale horizontally
Cache SystemStores routing configs, virtual keys, and moreRedis (in-cluster, AWS ElastiCache, or custom endpoint)
Log StoragePersistence for request/response dataMultiple options (see below)

Storage Options

S3-compatible storage options including:
  • AWS S3 (standard credentials or assumed roles)
  • Google Cloud Storage (S3 compatible interoperability mode)
  • Azure Blob Storage (key, managed identity, or Entra ID)
  • Any S3-compatible Blob Storage

Authentication Methods

  • IAM roles for service accounts (IRSA) in Kubernetes
  • Instance Metadata Service (IMDS) for EC2/ECS
  • Managed identities in Azure environments

Infrastructure Requirements

  • Kubernetes Cluster: K8s 1.20+ with Helm 3.x
  • Outbound Network: HTTPS access to Control Plane endpoints
  • Container Registry Access: For pulling gateway container images
  • Recommended Resource Requirements:
    • CPU: 1-2 cores per gateway instance
    • Memory: 2-4GB per gateway instance
    • Storage: Dependent on logging configuration

Data Security & Encryption

Your Sensitive Data Stays in Your VPC
  • All prompt content and LLM responses remain within your network
  • Only anonymized metrics data cross network boundaries
  • Log storage location is configurable based on your requirements

Advantages of Hybrid Architecture

BenefitTechnical ImplementationBusiness Value
Security & Compliance- Sensitive data never leaves VPC
- Configurable encryption methods
- Flexible authentication options
- Meets data residency requirements
- Supports regulated industries
- Simplifies security reviews
Operational Efficiency- No database management overhead
- Automatic model config updates
- Horizontally scalable architecture
- Low operational burden
- Always up-to-date with LLM ecosystem
- Scales with your traffic patterns
Deployment Flexibility- Kubernetes-native deployment
- Support for major cloud providers
- Multiple storage backend options
- Fits into existing infrastructure
- Avoids vendor lock-in
- Customizable to specific needs
Developer Experience- OpenAI-compatible API
- Simple integration patterns
- Comprehensive observability
- Minimal code changes needed
- Smooth developer onboarding
- Full visibility into system behavior

Technical Rationale

  1. Real-time Model Updates: LLM providers frequently change model parameters, pricing, and availability. Centralizing this data ensures all gateways operate with current information.
  2. Feature Velocity: AI landscape evolves rapidly. Control Plane architecture allows Portkey to deliver new features multiple times per week without requiring customer-side deployments.
  3. Operational Efficiency: Eliminates need for customers to maintain complex database infrastructure solely for non-sensitive object management.
  1. Performance: Eliminates network latency during LLM requests by having all routing and configs data available locally.
  2. Resilience: Gateway continues operating even if temporarily disconnected from Control Plane.
  3. Security: Reduces attack surface by minimizing runtime external dependencies.

Sample Files

These samples demonstrate the typical data patterns flowing between systems:

Resources & Next Steps

Have Questions?

Our solution architects are available to discuss your specific deployment requirements and security needs.

Schedule Architecture Discussion

Book a personalized consultation with our enterprise team to explore how Portkey’s architecture can be tailored to your organization’s specific requirements.