- Customer Environment — the AI Gateway and the LLM providers it routes to. All LLM traffic is processed within your network boundary. No prompt or response data is forwarded to the Management Plane directly.
- Management Plane — the Dashboard, Backend, MySQL, ClickHouse, Redis, and Blob Store. Handles identity resolution, configuration management, analytics, log storage, and audit logging. It communicates with the Customer Environment over a private link.
Core components
Customer Environment
Management Plane
Identity and authentication
The SCM API Gateway is the sole authentication authority. Standard session cookies are not used on Management Plane routes. Identity is established through three headers that the SCM API Gateway injects into every request it forwards:
mTLS boundary: the load balancer between the SCM API Gateway and the Backend terminates the
mTLS connection and forwards the verified client certificate subject. The Backend re-verifies the
certificate CN so that gateway-injected identity headers are only trusted when the caller is the
genuine SCM API Gateway.
Tenant-to-organisation resolution: every request to a Management Plane route resolves the
Tenant ID to an internal organisation identifier by querying a dedicated tenant mapping table in
MySQL. This resolved identifier scopes all downstream database queries and enforces data
isolation between tenants.
Request lifecycle
Synchronous path (affects response latency)
- Entry — the customer application sends an LLM request to the AI Gateway in the Customer Environment.
- Auth and config hydration — the gateway checks Redis for a cached copy of the API key, organisation context, and routing configuration. On a cache hit this is near-instantaneous. On a cache miss the gateway calls the Backend over the private link; the Backend reads from MySQL and returns the hydrated object, which the gateway then stores in Redis.
- LLM proxying — the gateway forwards the request, with any configured transformations, to the target LLM provider. The provider response is streamed back through the gateway.
- Response return — the LLM response is returned to the customer application. The request lifecycle ends here from the client’s perspective.
Asynchronous path (non-blocking, runs after the response is returned)
- Metrics write to ClickHouse — the gateway writes a lightweight analytics record capturing token counts, cost, latency, model, provider, cache status, and trace identifiers, along with a pointer to the full log body. This write is batched and does not affect response latency.
- Raw log write to Blob Store — the gateway writes the complete request and response payload (headers, body, token breakdown) as a structured document. The file path is the same pointer stored in the ClickHouse record in step 5, letting the Backend cross-reference both stores when serving log details.
- Usage sync — the gateway notifies the Backend of updated usage counters, API key exhaustion events, and quota state. This call is non-blocking.
Data flow between planes
AI traffic (fully contained in your environment)
AI traffic (fully contained in your environment)
All LLM traffic stays within your network boundary:
- Your application sends requests to the AI Gateway
- The gateway processes the request, applying routing, caching, and guardrails
- The gateway forwards the request to the appropriate LLM provider
- Responses from LLMs return through the same path
Config sync (gateway ↔ Management Plane)
Config sync (gateway ↔ Management Plane)
The AI Gateway periodically synchronises with the Management Plane:
- Frequency: 1-minute heartbeat intervals
- Data retrieved: routing configs, integrations, providers, API keys
- Process: deltas (changed items) are fetched, decrypted locally, and the corresponding cache entries are invalidated — they are re-fetched on next use
- Resilience: the gateway operates independently between syncs using cached configs, so it continues serving traffic during a Management Plane disconnection
Cache behaviour
Cache behaviour
The gateway holds all configuration objects — API keys, virtual keys, configs, guardrails — in
a local cache so every request is served without a real-time round-trip to the Management
Plane.
- Lazy-loaded: items enter the cache on first use; their TTL resets to 7 days on every re-population, so actively used objects stay fresh automatically.
- Delta invalidation: every minute the gateway fetches a list of changed items from the Management Plane and deletes the corresponding cache entries.
- Eviction policy: use
volatile-lruso idle entries are evicted first under memory pressure, preserving the objects your live traffic depends on.
Cache Behavior
TTL details, delta sync mechanics, eviction policy, and configuration reference.
Analytics (gateway → Management Plane)
Analytics (gateway → Management Plane)
The gateway writes anonymised operational metrics — model used, token counts, response times —
to ClickHouse. These power the analytics dashboards without exposing prompt or response
content.
Log management options
Log management options
Option A: logs in your environment (recommended for high-security deployments)
- Logs stored in your own Blob Store
- When viewing logs in the Dashboard, the Management Plane requests them from the gateway
- The gateway encrypts and forwards logs
- No inbound connections to your environment are required to view logs
Storage ownership
Which component writes and reads each store determines your network security design and access control.ClickHouse split ownership
ClickHouse has intentionally split write ownership across the two planes:- The AI Gateway writes to the analytics tables at high volume directly after every request. These writes are asynchronous and bypass the Backend entirely, keeping the write path low-latency.
- The Backend writes only to the audit log table, recording every configuration change made through the Dashboard or API, tagged with the caller’s identity and timestamp.
Redis
Redis is the caching layer between the AI Gateway and the Backend. It is the primary mechanism by which the gateway avoids a synchronous Backend call on every request. What the AI Gateway stores:- Hydrated API key and organisation context, populated after a cache-miss Backend call
- Routing configurations, guardrail definitions, and virtual key credentials
- Rate limit counters, tracked as atomic per-key and per-organisation windows
- Circuit breaker state, tracking per-provider failure and recovery status
- Semantic cache entries for prompt similarity matching
- On every configuration mutation — an API key rotation, config update, or workspace change — the Backend invalidates the relevant Redis entry, forcing the next gateway request for that key to re-fetch fresh data from MySQL.
- The Backend also uses Redis for internal background job coordination: analytics provisioning, alert dispatching, and usage syncing.
Blob Store
Blob Store is the raw log storage layer. It holds the full-fidelity record of every LLM interaction and is deliberately kept separate from ClickHouse, which holds only lightweight analytics metrics. Every LLM call results in a structured document containing the complete request body, response body, headers, and token breakdown. The gateway writes it asynchronously after returning the response to the caller, so the write does not affect latency, and stores the file path as a pointer in the corresponding ClickHouse record. The Backend does not read Blob Store in the regular request flow. It reads only on demand, when a user opens a log detail view: it retrieves the storage pointer from ClickHouse and fetches the corresponding document. The two stores are always used together for log detail retrieval; neither is queried in isolation. Supported backends include AWS S3, Google Cloud Storage, Azure Blob Storage, and any S3-compatible store such as MinIO.Deploying the gateway
The AI Gateway is deployed as containerised workloads using Helm charts for Kubernetes, with options for the major cloud providers.Storage options
- Object Storage
- Document DB
S3-compatible storage options including:
- AWS S3 (standard credentials or assumed roles)
- Google Cloud Storage (S3-compatible interoperability mode)
- Azure Blob Storage (key, managed identity, or Entra ID)
- Any S3-compatible Blob Storage
Authentication methods
- Cloud Provider IAM
- Direct Authentication
- IAM roles for service accounts (IRSA) in Kubernetes
- Instance Metadata Service (IMDS) for EC2/ECS
- Managed identities in Azure environments
Infrastructure requirements
- Kubernetes cluster: K8s 1.20+ with Helm 3.x
- Outbound network: HTTPS access to Management Plane endpoints
- Container registry access: for pulling gateway container images
- Recommended resources per gateway instance:
- CPU: 1–2 cores
- Memory: 2–4 GB
- Storage: dependent on logging configuration
Data security and encryption
- Data Residency
- Encryption Methods
- Access Controls
Your sensitive data stays in your environment
- All prompt content and LLM responses remain within your network
- Only anonymised metrics cross network boundaries
- Log storage location is configurable
Why the architecture is split this way
Why the transaction database lives in the Management Plane
Why the transaction database lives in the Management Plane
- Real-time model updates: LLM providers frequently change model parameters, pricing, and availability. Centralising this data ensures every gateway operates with current information.
- Feature velocity: new capabilities can ship without requiring a customer-side deployment.
- Operational efficiency: you do not maintain database infrastructure solely for non-sensitive object management.
Why objects are cached locally
Why objects are cached locally
- Performance: eliminates network latency during LLM requests by keeping routing and config data available locally.
- Resilience: the gateway keeps operating even when temporarily disconnected from the Management Plane.
- Security: reduces attack surface by minimising runtime external dependencies.