> ## Documentation Index > Fetch the complete documentation index at: https://docs.portkey.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # AKS > This enterprise-focused document provides comprehensive instructions for deploying the Portkey software on Azure Kubernetes Service (AKS), tailored to meet the needs of large-scale, mission-critical applications. It includes specific recommendations for component sizing, high availability, and integration with monitoring systems. Portkey is also available on the Azure Marketplace. You can deploy Portkey directly through your Azure console, which streamlines procurement and deployment processes. [Deploy via Azure Marketplace →](https://azuremarketplace.microsoft.com/en-in/marketplace/apps/portkey.enterprise-saas?tab=Overview) ## Components and Sizing Recommendations | Component | Options | Sizing Recommendations | | ------------------------------------ | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | AI Gateway | Deploy in your AKS cluster using Helm charts. | Use AKS B2ms worker nodes, each providing at least 2 vCPUs and 4 GiB of memory. For high availability, deploy them across multiple Availability Zones. | | Logs Store (optional) | Azure Blob Storage or S3-compatible storage | Each log document is \~10kb in size (uncompressed) | | Cache (Prompts, Configs & Providers) | Built-in Redis or Azure Managed Redis | Deployed within the same VNet as the Portkey Gateway. | ## Prerequisites Ensure the following tools and resources are installed and available: * A running [AKS cluster](https://learn.microsoft.com/en-us/azure/aks/tutorial-kubernetes-deploy-cluster?tabs=azure-cli#create-a-kubernetes-cluster) with at least 2 worker nodes. (**Best Practice:** Use 2 nodes, with one node in each Availability Zone, to ensure high availability.) * (Optional) If using Workload Identity for Blob Storage access, ensure [OIDC issuer](https://learn.microsoft.com/en-us/azure/aks/use-oidc-issuer) and [Workload Identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster#update-an-existing-aks-cluster) are enabled on your AKS cluster. * [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/get-started-with-azure-cli?view=azure-cli-latest) * [Kubectl](https://learn.microsoft.com/en-us/azure/aks/tutorial-kubernetes-deploy-cluster?tabs=azure-cli#connect-to-cluster-using-kubectl) * [Helm (v3 or above)](https://helm.sh/docs/intro/install/) ## Create a Portkey Account * Go to the [Portkey](https://app.portkey.ai) website. * Sign up for a Portkey account. * Once logged in, locate and save your `Organisation ID` for future reference. It can be found in the browser URL: `https://app.portkey.ai/organisation//` * Contact the Portkey AI team and provide your Organisation ID and the email address used during signup. * The Portkey team will share the following information with you: * Docker credentials for the Gateway images (username and password). * License: Client Auth Key. ## Setup Project Environment ```sh theme={"system"} cluster_name= # Specify the name of the AKS cluster where the gateway will be deployed. namespace= # Specify the namespace where the gateway should be deployed (for example, portkeyai). service_account_name= # Provide a name for the Service Account to be associated with Gateway Pod (for example, gateway-sa) mkdir portkey-gateway cd portkey-gateway touch values.yaml ``` ### Image Credentials Configuration ```yaml theme={"system"} # Update the values.yaml file imageCredentials: - name: portkey-enterprise-registry-credentials create: true registry: https://index.docker.io/v1/ username: password: images: gatewayImage: repository: "docker.io/portkeyai/gateway_enterprise" pullPolicy: Always tag: "latest" dataserviceImage: repository: "docker.io/portkeyai/data-service" pullPolicy: Always tag: "latest" redisImage: repository: "docker.io/redis" pullPolicy: IfNotPresent tag: "7.2-alpine" environment: create: true secret: true data: ANALYTICS_STORE: control_plane SERVICE_NAME: # Specify a name for the service PORTKEY_CLIENT_AUTH: ORGANISATIONS_TO_SYNC: # This is obtained after signing up for a Portkey account. ``` ## Configure Components Based on the choice of components and their configuration update the `values.yaml`. ### MCP Gateway (Optional) By default, only the AI Gateway is enabled in the deployment. To enable the MCP Gateway, add the following configuration to `values.yaml`: ```yaml theme={"system"} environment: data: SERVER_MODE: "mcp/all" MCP_PORT: "8788" MCP_GATEWAY_BASE_URL: "" ``` **Note:** * `MCP_GATEWAY_BASE_URL` must include the protocol prefix — either `http://` or `https://`. * This value is not required for the initial deployment. After the first deployment, once the MCP Load Balancer is provisioned and a hostname is mapped to the MCP Service, set this value and redeploy. **Server Modes** 1. `""` (empty or not provided): Deploys only the AI Gateway. This is the default configuration. 2. `"mcp"`: Deploys only the MCP Gateway. 3. `"all"`: Deploys both the AI Gateway and MCP Gateway. ### Cache Store The Portkey Gateway deployment includes a Redis instance pre-installed by default. You can either use this built-in Redis or connect to an external cache like `Azure Managed Redis`. #### Built-in Redis No additional permissions or network configurations are required. ```yaml theme={"system"} ## To use the built-in Redis, add the following configuration to the values.yaml file. environment: data: CACHE_STORE: redis REDIS_URL: "redis://redis:6379" REDIS_TLS_ENABLED: "false" ``` #### Azure Managed Redis To enable the gateway to work with an Azure Managed cache, ensure connectivity from AKS cluster. [Set up](#azure-managed-redis) access to the Redis instance based on your chosen authentication method and update `values.yaml` with the following configuration. ```yaml theme={"system"} environment: data: CACHE_STORE: azure-redis REDIS_URL: rediss://: AZURE_REDIS_AUTH_MODE: password REDIS_TLS_ENABLED: "true" ## "true"/"false" REDIS_MODE: cluster ## Add this parameter only if cluster mode is enabled REDIS_PASSWORD: ## Provide Azure Managed Redis Access Key ``` ```yaml theme={"system"} serviceAccount: create: true automount: true name: annotations: azure.workload.identity/client-id: podLabels: azure.workload.identity/use: "true" environment: data: CACHE_STORE: azure-redis REDIS_URL: "redis://:" REDIS_TLS_ENABLED: "true" ## "true"/"false" REDIS_MODE: cluster ## Add this parameter only if cluster mode is enabled AZURE_REDIS_AUTH_MODE: workload ``` ```yaml theme={"system"} environment: data: CACHE_STORE: azure-redis REDIS_URL: "redis://:" REDIS_TLS_ENABLED: "true" ## "true"/"false" REDIS_MODE: cluster ## Add this parameter only if cluster mode is enabled AZURE_REDIS_AUTH_MODE: managed AZURE_REDIS_MANAGED_CLIENT_ID: ## Defaults to AZURE_MANAGED_CLIENT_ID if not provided ``` ```yaml theme={"system"} environment: data: CACHE_STORE: azure-redis REDIS_URL: "redis://:" REDIS_TLS_ENABLED: "true" ## "true"/"false" REDIS_MODE: cluster ## Add this parameter only if cluster mode is enabled AZURE_REDIS_AUTH_MODE: entra AZURE_REDIS_ENTRA_CLIENT_ID: ## Defaults to AZURE_ENTRA_CLIENT_ID if not provided AZURE_REDIS_ENTRA_CLIENT_SECRET: ## Defaults to AZURE_ENTRA_CLIENT_SECRET if not provided AZURE_REDIS_ENTRA_TENANT_ID: ## Defaults to AZURE_ENTRA_TENANT_ID if not provided ``` ### Log Store #### Azure Blob Storage 1. (Optional) If not already done, create an Azure Storage Account and a Blob Container to store LLM logs. 2. [Set up](#setting-up-permission) access to the log store. The Gateway supports the following methods for connecting to the Blob Storage: * Workload Identity * Managed Identity * Entra ID Depending on the chosen Blob Storage access method, update `values.yaml` with the following configuration. To set up Workload Identity authentication for the Portkey Gateway, follow the [steps](#setting-up-permission) and add the following configuration to `values.yaml`. ```yaml theme={"system"} ## To enable Workload Identity update values.yaml with the following details:- serviceAccount: create: true automount: true name: annotations: azure.workload.identity/client-id: podLabels: azure.workload.identity/use: "true" environment: data: LOG_STORE: azure AZURE_STORAGE_ACCOUNT: # Specify the name of the storage account which will be used for storing LLM logs. AZURE_STORAGE_CONTAINER: # Specify the name of the blob store container which will be used for storing LLM logs. AZURE_AUTH_MODE: workload ``` ```yaml theme={"system"} ## To enable Managed Identity update values.yaml with following details:- environment: data: LOG_STORE: azure AZURE_STORAGE_ACCOUNT: # Specify the name of the storage account which will be used for storing LLM logs. AZURE_STORAGE_CONTAINER: # Specify the name of the blob store container which will be used for storing LLM logs. AZURE_AUTH_MODE: managed AZURE_MANAGED_CLIENT_ID: ``` ```yaml theme={"system"} ## To enable Entra ID update values.yaml with following details:- environment: data: LOG_STORE: azure AZURE_STORAGE_ACCOUNT: # Specify the name of the storage account which will be used for storing LLM logs. AZURE_STORAGE_CONTAINER: # Specify the name of the blob store container which will be used for storing LLM logs. AZURE_AUTH_MODE: entra AZURE_ENTRA_CLIENT_ID: # Specify client id of the app created during set up of Entra ID access for blob store. AZURE_ENTRA_CLIENT_SECRET: # Specify client secret of the app created during set up of Entra ID access for blob store. AZURE_ENTRA_TENANT_ID: # Specify tenant id obtained during set up of Entra ID access for blob store. ``` 3. (Optional) Configure log path format using `LOG_STORE_FILE_PATH_FORMAT`. See [Log Object Path Format](/product/enterprise-offering/components#log-object-path-format) for details. ## Network Configuration ### Set Up External Access To make the Gateway service accessible externally, you can set up either of the following: * **Azure Load Balancer** with NGINX `Ingress` controller * **Azure Load Balancer** with Kubernetes `Service` #### Azure Load Balancer with NGINX Ingress Controller Recommended if `SERVER_MODE` is set to `all` 1. Enable application routing add-on. ```sh theme={"system"} RESOURCE_GROUP= # Provide name of resource group in which AKS cluster exists CLUSTER_NAME= # Provide name of AKS cluster az aks approuting enable --resource-group ${RESOURCE_GROUP} --name ${CLUSTER_NAME} --nginx None ``` 2. Create NGINX Ingress Controller **External LB with Static IP** ```sh theme={"system"} STATIC_IP_NAME=portkey-lb-static-ip # Create Static Public IP to associate with LB az network public-ip create \ --resource-group ${RESOURCE_GROUP} \ --name ${STATIC_IP_NAME} \ --sku Standard \ --allocation-method static # Delegate AKS cluster permission to created Public IP CLIENT_ID=$(az aks show --name ${CLUSTER_NAME} --resource-group ${RESOURCE_GROUP} --query identity.principalId -o tsv) RG_SCOPE=$(az group show --name ${RESOURCE_GROUP} --query id -o tsv) az role assignment create \ --assignee ${CLIENT_ID} \ --role "Network Contributor" \ --scope ${RG_SCOPE} # Create External NGINX Ingress Controller kubectl apply -f - <" # Set 'nginx-static' for External LB and 'nginx-internal' for Internal LB # hostname: "" # hostBased: true # mcpHostname: "" annotations: service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/v1/health" ``` 4. Retrieve IP address of Azure Load Balancer. ```sh theme={"system"} IP=$(kubectl get ingress portkey-ai-gateway -n portkeyai -o jsonpath='{.status.loadBalancer.ingress[0].ip}') echo ${IP} ``` **Notes:** * If `SERVER_MODE` is set to `all` (i.e., both AI Gateway and MCP Gateway are enabled), you must enable host-based routing by setting `hostBased: true` and specify the hostname through which the AI Gateway and MCP Gateway will be accessed. * After the Load Balancer is provisioned, create a DNS record in your public or private hosted zone that points to the Load Balancer’s static public or private IPv4 address. If `SERVER_MODE` is set to `all`, you must create two separate DNS records—one for the AI Gateway hostname and one for the MCP Gateway hostname—each pointing to the Load Balancer IP. * The Azure Load Balancer NGINX Controller supports additional annotations—such as TLS configuration, custom health checks, and more—for managing the Ingress load balancer. For a complete list of supported annotations, refer to the [Azure NGINX Load Balancer ingress](https://learn.microsoft.com/en-us/azure/aks/app-routing). #### Azure Load Balancer with Kubernetes Service ```yaml theme={"system"} service: type: LoadBalancer port: 80 # LB listener port containerPort: 8787 annotations: # Specify the type of Load Balancer to create - internal or internet-facing service.beta.kubernetes.io/azure-load-balancer-internal: "false" service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol: "http" service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/v1/health" # Replace the cidr ranges to make it more restrictive service.beta.kubernetes.io/azure-allowed-ip-ranges: 0.0.0.0/0 ``` Azure Load Balancer supports various annotations to fine-tune its configuration. For a complete list of supported annotations, see the Azure Load Balancer [annotations](https://cloud-provider-azure.sigs.k8s.io/topics/loadbalancer/#loadbalancer-annotations). **Note:**`service.containerPort` must be same as `environment.data.PORT`. ## Deploying Portkey Gateway ```sh theme={"system"} # Add the Portkey AI Gateway helm repository helm repo add portkey-ai https://portkey-ai.github.io/helm helm repo update # Install the chart helm upgrade --install portkey-ai portkey-ai/gateway -f ./values.yaml -n $namespace --create-namespace ``` ## Verify the deployment To confirm that the deployment was successful, follow these steps: * Verify that all pods are running correctly. ```sh theme={"system"} # kubectl get pods -n $namespace # You should see all pods with a 'STATUS' of 'Running'. ``` **Note:** If pods are in a Pending, CrashLoopBackOff, or other error state, inspect the [pod logs](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_logs/) and [events](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_events/) to diagnose potential issues. * Test Gateway by sending a cURL request. 1. Port-forward the Gateway pod ```sh theme={"system"} kubectl port-forward -n $namespace 9000:8787 # Replace with your Gateway pod's actual name. ``` 2. Once port forwarding is active, open a new terminal window or tab and send a test request by running: ```sh theme={"system"} # Specify LLM provider and Portkey API keys OPENAI_API_KEY= # Replace with an actual API key PORTKEY_API_KEY= # Replace with Portkey API key which can be created from Portkey website(https://app.portkey.ai/api-keys). # Configure and send the curl request curl 'http://localhost:9000/v1/chat/completions' \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -H "x-portkey-provider: openai" \ -H "x-portkey-api-key: $PORTKEY_API_KEY" \ -d '{ "model": "gpt-4o-mini", "messages": [{"role": "user","content": "What is a fractal?"}] }' ``` 3. Test gateway service integration with Load Balancer. ```sh theme={"system"} # Replace and with the Load Balancer's IP/DNS and listener port respectively. curl 'http://:/v1/chat/completions' \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -H "x-portkey-provider: openai" \ -H "x-portkey-api-key: $PORTKEY_API_KEY" \ -d '{ "model": "gpt-4o-mini", "messages": [{"role": "user","content": "What is a fractal?"}] }' ``` ## Integrating Gateway with Control Plane **Outbound Connectivity (Data Plane to Control Plane)** Portkey supports the following methods for integrating the Data Plane with the Control Plane for outbound connectivity: * Azure Private Link * Over the Internet **Ensure Outbound Network Access** By default, Kubernetes allows full outbound access, but if your cluster has NetworkPolicies that restrict egress, configure them to allow outbound traffic. Example NetworkPolicy for Outbound Access: ```yaml theme={"system"} apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-all-egress namespace: portkeyai spec: podSelector: {} policyTypes: - Egress egress: - to: - ipBlock: cidr: 0.0.0.0/0 ``` This allows the gateway to access LLMs hosted both within your VPC and externally. This also enables connection for the sync service to the Portkey Control Plane. #### Azure Private Link Establishes a secure, private connection between the Control Plane and Data Plane within the Azure network. **Steps to establish Azure Private Link connectivity:** 1. Contact Portkey and provide your Azure Subscription Id so it can be whitelisted in Portkey's Control Plane. 2. Once you get confirmation from Portkey that your Azure account is whitelisted, create Azure Private Endpoint connection to Portkey Control Plane. ```sh theme={"system"} # Replace with Subnet ID (not Subnet Name) in which Portkey gateway is deployed. az network private-endpoint create \ --name portkey-cp-pvt-endpoint \ --resource-group ${RESOURCE_GROUP} \ --subnet \ --private-connection-resource-alias "portkey-privatelink-pls.d17828ab-c5d7-4f67-aae6-1f5b68ae0565.eastus2.azure.privatelinkservice" \ --connection-name portkey-cp-pvt-connection ``` 3. Retrieve Private IP of created endpoint. ```sh theme={"system"} PE_IP=$(az network nic show --ids $(az network private-endpoint show --name portkey-cp-pvt-endpoint --resource-group ${RESOURCE_GROUP} --query "networkInterfaces[0].id" -o tsv) --query "ipConfigurations[0].privateIPAddress" -o tsv) ``` 4. Contact the Portkey team and share the endpoint name with them. 5. Once the connection request is approved, create Private Hosted Zone and link it to Gateway VNet. ```sh theme={"system"} # Create PHZ az network private-dns zone create \ --resource-group ${RESOURCE_GROUP} \ --name privatelink-az.portkey.ai # Create VNet Link # Replace with the VNet ID (not VNet name) of AKS Node Group. # IMPORTANT!! must be full Resource ID of AKS's node group's VNet. # You can find it by navigating to AKS Cluster > Settings > Networking > VNet integration > Click on VNet. # On Vnet console, click on JSON View and copy Resource ID # e.g, az network private-dns link vnet create \ --resource-group ${RESOURCE_GROUP} \ --zone-name privatelink-az.portkey.ai \ --name portkey-gateway-vnet-link \ --virtual-network "" \ --registration-enabled false ``` 6. Create a record in the PHZ pointing to Portkey's Private Endpoint IP. ```sh theme={"system"} az network private-dns record-set a add-record \ --resource-group ${RESOURCE_GROUP} \ --zone-name privatelink-az.portkey.ai \ --record-set-name azure-cp \ --ipv4-address ${PE_IP} ``` 7. Update the `values.yaml` file with the following environment variables. ```yaml theme={"system"} environment: create: true secret: true data: ALBUS_BASEPATH: "https://azure-cp.privatelink-az.portkey.ai/albus" CONTROL_PLANE_BASEPATH: "https://azure-cp.privatelink-az.portkey.ai/api/v1" SOURCE_SYNC_API_BASEPATH: "https://azure-cp.privatelink-az.portkey.ai/api/v1/sync" CONFIG_READER_PATH: "https://azure-cp.privatelink-az.portkey.ai/api/model-configs" ``` 8. Re-deploy the gateway. ```sh theme={"system"} helm upgrade --install portkey-ai portkey-ai/gateway -f ./values.yaml -n ${namespace} --create-namespace ``` #### Over the Internet Ensure Gateway has access to the following endpoints over the internet. * `https://api.portkey.ai` * `https://albus.portkey.ai` ### Inbound Connectivity (Control Plane to Data Plane) * Azure Private Link * IP Whitelisting #### Azure Private Link Establishes a secure, private connection between the Control Plane and Data Plane within the Azure network. **Steps to establish Azure Private Link connectivity:** 1. Create a dedicated subnet for Azure Private Link Service. ```sh theme={"system"} # Replace with CIDR you want to allocate to this subnet. CIDR must be at least /27, must fall within the AKS VNet CIDR range, and must not overlap with existing subnet CIDRs. # IMPORTANT!! must be the Resource Group in which AKS Node Group exists. It starts with MC_*. az network vnet subnet create \ --resource-group ${VNET_RESOURCE_GROUP} \ --vnet-name \ --name private-endpoints-subnet \ --address-prefixes --private-link-service-network-policies Disabled ``` 2. On the Azure Portal, navigate to `Private Link services` and click **+ Create** to create a new Private Link Service (PLS). 3. Select the **Subscription** and **Resource Group**, provide a name for the PLS, and click **Next**. 4. Under **Outbound settings**, select the Load Balancer created for the gateway. Then choose the appropriate **frontend IP configuration** of that Load Balancer. 5. In the `Source NAT subnet` field, select the dedicated PLS subnet created earlier, and click **Next**. 6. Under **Access security**, select **Restricted by subscription** and whitelist the Portkey Control Plane subscription ID (`4bec865f-23ea-4d04-be20-3e883cbb3eb1`). 7. Leave all other settings as default and click **Create** to provision the Private Link Service. 8. Contact the Portkey team and share the **alias** of the created Private Link Service, along with the **Gateway URL**. 9. Once the Portkey team has raised the connection request from their end, approve it by navigating to the PLS in the Azure Portal, then go to **Private Endpoint connections** and click **Approve**. #### IP Whitelisting Allows control plane to access the Data Plane over the internet by restricting inbound traffic to specific IP address of Control Plane. This method requires the Data Plane to have a publicly accessible endpoint. To whitelist, add an inbound rule to the Azure NSG or Firewall allowing connections from the Portkey Control Plane's IPs (`54.81.226.149`, `34.200.113.35`, `44.221.117.129`) on required port. To integrate the Control Plane with the Data Plane, contact the Portkey team and provide the **Public Endpoint** of the Data Plane. ## Verifying Gateway Integration with the Control Plane * Send a test request to Gateway using `curl`. * Go to [Portkey website](https://app.portkey.ai/) -> **Logs**. * Verify that the test request appears in the logs and that you can view its full details by selecting the log entry. ## Uninstalling Portkey Gateway ```sh theme={"system"} helm uninstall portkey-ai --namespace $namespace ``` ## Setting up Permission #### Azure Blob Storage To allow the Portkey Gateway to access Azure Blob Storage for log storage, permissions must be granted. Follow the steps below to set up these permissions according to your selected access method. **Prerequisites:** Ensure [OIDC issuer](https://learn.microsoft.com/en-us/azure/aks/use-oidc-issuer) and [Workload Identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster#update-an-existing-aks-cluster) are enabled on your AKS cluster. 1. Specify the details: ```sh theme={"system"} CLUSTER_NAME= # Specify name of AKS cluster RESOURCE_GROUP= # Specify the name of the resource group STORAGE_ACCOUNT_NAME= # Specify the name of the storage account which will be used for storing LLM logs. CONTAINER_NAME= # Specify the name of the blob store container which will be used for storing LLM logs. MANAGED_IDENTITY_NAME= # Specify a name for the User-Assigned Managed Identity (e.g., portkey-gateway-identity) SERVICE_ACCOUNT_NAME= # Must match the service account name used in values.yaml NAMESPACE= # Namespace where the gateway is deployed ``` 2. Create a User-Assigned Managed Identity. ```sh theme={"system"} az identity create \ --name ${MANAGED_IDENTITY_NAME} \ --resource-group ${RESOURCE_GROUP} MANAGED_IDENTITY_CLIENT_ID=$(az identity show \ --name ${MANAGED_IDENTITY_NAME} \ --resource-group ${RESOURCE_GROUP} \ --query clientId -o tsv) echo $MANAGED_IDENTITY_CLIENT_ID ``` Make a note of `MANAGED_IDENTITY_CLIENT_ID`, as it is required in `values.yaml`. 3. Create a federated identity credential to link the Kubernetes service account to the managed identity. ```sh theme={"system"} OIDC_ISSUER=$(az aks show \ --name ${CLUSTER_NAME} \ --resource-group ${RESOURCE_GROUP} \ --query "oidcIssuerProfile.issuerUrl" -o tsv) az identity federated-credential create \ --name portkey-gateway-federated-cred \ --identity-name ${MANAGED_IDENTITY_NAME} \ --resource-group ${RESOURCE_GROUP} \ --issuer ${OIDC_ISSUER} \ --subject system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT_NAME} \ --audiences api://AzureADTokenExchange ``` 4. Grant the managed identity a role that allows it to access Blob Storage. ```sh theme={"system"} STORAGE_ID=$(az storage account show \ --name ${STORAGE_ACCOUNT_NAME} \ --resource-group ${RESOURCE_GROUP} --query id -o tsv) az role assignment create \ --assignee ${MANAGED_IDENTITY_CLIENT_ID} \ --role "Storage Blob Data Contributor" \ --scope "${STORAGE_ID}/blobServices/default/containers/${CONTAINER_NAME}" ``` 1. Specify the details: ```sh theme={"system"} RESOURCE_GROUP= # Specify the name of the resource group STORAGE_ACCOUNT_NAME= # Specify the name of the storage account which will be used for storing LLM logs. CONTAINER_NAME= # Specify the name of the blob store container which will be used for storing LLM logs. MANAGED_IDENTITY_NAME= # Specify a name for the User-Assigned Managed Identity (e.g., portkey-gateway-identity) ``` 2. Create a User-Assigned Managed Identity. ```sh theme={"system"} az identity create \ --name ${MANAGED_IDENTITY_NAME} \ --resource-group ${RESOURCE_GROUP} MANAGED_IDENTITY_CLIENT_ID=$(az identity show \ --name ${MANAGED_IDENTITY_NAME} \ --resource-group ${RESOURCE_GROUP} \ --query clientId -o tsv) ``` Make a note of `MANAGED_IDENTITY_CLIENT_ID`, as it is required in `values.yaml`. 3. [Assign the managed identity to the AKS cluster VMSS](https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/qs-configure-cli-windows-vmss#user-assigned-managed-identity) so pods can use it. ```sh theme={"system"} VMSS_RESOURCE_GROUP=$(az aks show \ --resource-group ${RESOURCE_GROUP} \ --name ${CLUSTER_NAME} \ --query nodeResourceGroup -o tsv) VMSS_NAME=$(az vmss list \ --resource-group ${VMSS_RESOURCE_GROUP} \ --query "[0].name" -o tsv) az vmss identity assign \ --resource-group ${VMSS_RESOURCE_GROUP} \ --name ${VMSS_NAME} \ --identities $(az identity show --name ${MANAGED_IDENTITY_NAME} --resource-group ${RESOURCE_GROUP} --query id -o tsv) ``` 4. Grant the managed identity a role that allows it to access Blob Storage. ```sh theme={"system"} STORAGE_ID=$(az storage account show \ --name ${STORAGE_ACCOUNT_NAME} \ --resource-group ${RESOURCE_GROUP} --query id -o tsv) az role assignment create \ --assignee ${MANAGED_IDENTITY_CLIENT_ID} \ --role "Storage Blob Data Contributor" \ --scope "${STORAGE_ID}/blobServices/default/containers/${CONTAINER_NAME}" ``` 1. Register an Entra Identity application. ```sh theme={"system"} AZURE_ENTRA_CLIENT_ID=$(az ad app create \ --display-name portkey-gateway-logstore-app \ --sign-in-audience "AzureADMyOrg" \ --query appId -o tsv) echo $AZURE_ENTRA_CLIENT_ID ``` Make a note of the value of `AZURE_ENTRA_CLIENT_ID`, as it will be required in defining `values.yaml`. 2. Create a client secret for registered app. ```sh theme={"system"} AZURE_ENTRA_CLIENT_SECRET=$(az ad app credential reset \ --id $AZURE_ENTRA_CLIENT_ID \ --display-name "entra-secret" \ --query password -o tsv) echo $AZURE_ENTRA_CLIENT_SECRET ``` Make a note of the value of `AZURE_ENTRA_CLIENT_SECRET`, as it will be required in defining `values.yaml`. 3. Retrieve your Tenant ID. ```sh theme={"system"} AZURE_ENTRA_TENANT_ID=$(az account show \ --query tenantId -o tsv) echo $AZURE_ENTRA_TENANT_ID ``` Make a note of the value of `AZURE_ENTRA_TENANT_ID`, as it will be required in defining `values.yaml`. 4. Grant the app a role that allows it to access Blob Storage. ```sh theme={"system"} RESOURCE_GROUP= # Specify the name of the resource group that contains the storage account. STORAGE_ACCOUNT_NAME= # Specify the name of the storage account which will be used for storing LLM logs. CONTAINER_NAME= # Specify the name of the blob store container which will be used for storing LLM logs. # Create a Service Principal for the app. az ad sp create --id $AZURE_ENTRA_CLIENT_ID # Fetch storage id of storage account STORAGE_ID=$(az storage account show \ --name $STORAGE_ACCOUNT_NAME \ --resource-group $RESOURCE_GROUP \ --query id -o tsv) # Assign the role to the app. az role assignment create \ --assignee $AZURE_ENTRA_CLIENT_ID \ --role "Storage Blob Data Contributor" \ --scope "$STORAGE_ID/blobServices/default/containers/$CONTAINER_NAME" ``` #### Azure Managed Redis To allow the Portkey Gateway to authenticate with Azure Managed Redis using Microsoft Entra ID, permissions must be granted. **Prerequisites:** [Microsoft Entra Authentication](https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/cache-azure-active-directory-for-authentication) must be enabled on the Azure Managed Redis instance. Follow the steps below to set up these permissions according to your selected access method. Complete steps 1 to 3 from the [Azure Blob Storage > Workload Identity](#azure-blob-storage) section if not already done. 1. Assign a data access policy to the managed identity on the Redis instance. ```sh theme={"system"} REDIS_NAME= # Name of the Azure Managed Redis instance RESOURCE_GROUP= MANAGED_IDENTITY_NAME= # Same identity created in Blob Storage setup MANAGED_IDENTITY_OBJECT_ID=$(az identity show \ --name ${MANAGED_IDENTITY_NAME} \ --resource-group ${RESOURCE_GROUP} \ --query principalId -o tsv) az redisenterprise database access-policy-assignment create \ --access-policy-assignment-name portkeyGatewayRedisAccess \ --cluster-name ${REDIS_NAME} \ --database-name default \ --resource-group ${RESOURCE_GROUP} \ --access-policy-name default \ --object-id ${MANAGED_IDENTITY_OBJECT_ID} ``` Complete steps 1 to 3 from the [Azure Blob Storage > Managed Identity](#azure-blob-storage) section if not already done. 1. Assign a data access policy to the managed identity on the Redis instance. ```sh theme={"system"} REDIS_NAME= # Name of the Azure Managed Redis instance RESOURCE_GROUP= MANAGED_IDENTITY_NAME= # Same identity created in Blob Storage setup MANAGED_IDENTITY_OBJECT_ID=$(az identity show \ --name ${MANAGED_IDENTITY_NAME} \ --resource-group ${RESOURCE_GROUP} \ --query principalId -o tsv) az redisenterprise database access-policy-assignment create \ --access-policy-assignment-name portkeyGatewayRedisAccess \ --cluster-name ${REDIS_NAME} \ --database-name default \ --resource-group ${RESOURCE_GROUP} \ --access-policy-name default \ --object-id ${MANAGED_IDENTITY_OBJECT_ID} ``` Complete steps 1 to 3 from the [Azure Blob Storage > Entra Identity](#azure-blob-storage) section if not already done. 1. Assign a data access policy to the service principal on the Redis instance. ```sh theme={"system"} REDIS_NAME= # Name of the Azure Managed Redis instance RESOURCE_GROUP= ENTRA_OBJECT_ID=$(az ad sp show \ --id ${AZURE_ENTRA_CLIENT_ID} \ --query id -o tsv) az redisenterprise database access-policy-assignment create \ --access-policy-assignment-name portkeyGatewayRedisAccess \ --cluster-name ${REDIS_NAME} \ --database-name default \ --resource-group ${RESOURCE_GROUP} \ --access-policy-name default \ --object-id ${ENTRA_OBJECT_ID} ``` #### Azure OpenAI and Azure AI Foundry (Optional) If the Portkey Gateway routes requests to Azure OpenAI or Azure AI Foundry models, the identity used by the gateway must be granted the `Cognitive Services OpenAI User` role on the respective resource. This is only required when using Managed Identity or Workload Identity to authenticate with Azure OpenAI or Azure AI Foundry. If you are using API keys, no additional role assignment is needed. Complete steps 1 to 3 from the [Azure Blob Storage > Workload Identity](#azure-blob-storage) section if not already done. ```sh theme={"system"} MANAGED_IDENTITY_CLIENT_ID= # Replace with the full Resource ID of Azure OpenAI or Azure AI Foundry resource. # e.g., /subscriptions//resourceGroups//providers/Microsoft.CognitiveServices/accounts/ az role assignment create \ --assignee ${MANAGED_IDENTITY_CLIENT_ID} \ --role "Cognitive Services OpenAI User" \ --scope ``` Create a User-Assigned Managed Identity if not already done, and [assign it to the AKS cluster or VMSS](https://learn.microsoft.com/en-us/azure/aks/use-managed-identity#bring-your-own-managed-identity). ```sh theme={"system"} MANAGED_IDENTITY_CLIENT_ID= # Replace with the full Resource ID of Azure OpenAI or Azure AI Foundry resource. az role assignment create \ --assignee ${MANAGED_IDENTITY_CLIENT_ID} \ --role "Cognitive Services OpenAI User" \ --scope ``` Complete steps 1 to 3 from the [Azure Blob Storage > Entra Identity](#azure-blob-storage) section if not already done. ```sh theme={"system"} # Replace with the full Resource ID of Azure OpenAI or Azure AI Foundry resource. az role assignment create \ --assignee ${AZURE_ENTRA_CLIENT_ID} \ --role "Cognitive Services OpenAI User" \ --scope ``` ## Examples **Built-in Redis** The following sample `values.yaml` below shows how to configure the built-in Redis cache and Azure Blob Storage for log store using Entra ID. ```yaml theme={"system"} images: gatewayImage: repository: "docker.io/portkeyai/gateway_enterprise" pullPolicy: Always tag: "latest" dataserviceImage: repository: "docker.io/portkeyai/data-service" pullPolicy: Always tag: "latest" redisImage: repository: "docker.io/redis" pullPolicy: IfNotPresent tag: "7.2-alpine" imageCredentials: - name: portkeyenterpriseregistrycredentials create: true registry: https://index.docker.io/v1/ username: password: environment: create: true secret: true data: ANALYTICS_STORE: control_plane SERVICE_NAME: gateway PORTKEY_CLIENT_AUTH: # REPLACE with client auth shared by Portkey team. ORGANISATIONS_TO_SYNC: # REPLACE with organisation_id of your account. PORT: "8787" # Configuration for using built-in redis CACHE_STORE: redis REDIS_URL: "redis://redis:6379" REDIS_TLS_ENABLED: "false" # Configuration for enabling Entra ID access to Azure Blob Storage. LOG_STORE: azure AZURE_STORAGE_ACCOUNT: # Specify the name of the storage account which will be used for storing LLM logs. AZURE_STORAGE_CONTAINER: # Specify the name of the blob store container which will be used for storing LLM logs. AZURE_AUTH_MODE: entra AZURE_ENTRA_CLIENT_ID: # Specify client id of the app created during set up of Entra ID access for blob store. AZURE_ENTRA_CLIENT_SECRET: # Specify client secret of the app created during set up of Entra ID access for blob store. AZURE_ENTRA_TENANT_ID: # Specify tenant id obtained during set up of Entra ID access for blob store. # Enabling Load Balancer to provide access outside of cluster service: type: LoadBalancer port: 80 # LB listener port containerPort: 8787 annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "false" service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol: "http" service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/v1/health" service.beta.kubernetes.io/azure-allowed-ip-ranges: 0.0.0.0/0 ``` **Azure Managed Redis with Workload Identity** The following sample `values.yaml` shows how to configure Azure Managed Redis (Workload Identity) and Azure Blob Storage (Workload Identity) using a shared User-Assigned Managed Identity, with Private Link for outbound connectivity to the Control Plane. ```yaml theme={"system"} images: gatewayImage: repository: "docker.io/portkeyai/gateway_enterprise" pullPolicy: Always tag: "latest" dataserviceImage: repository: "docker.io/portkeyai/data-service" pullPolicy: Always tag: "latest" imageCredentials: - name: portkeyenterpriseregistrycredentials create: true registry: https://index.docker.io/v1/ username: password: serviceAccount: create: true automount: true name: annotations: azure.workload.identity/client-id: podLabels: azure.workload.identity/use: "true" environment: create: true secret: true data: ANALYTICS_STORE: control_plane SERVICE_NAME: gateway PORTKEY_CLIENT_AUTH: ORGANISATIONS_TO_SYNC: PORT: "8787" # Azure Managed Redis with Workload Identity CACHE_STORE: azure-redis REDIS_URL: "redis://:" REDIS_TLS_ENABLED: "true" AZURE_REDIS_AUTH_MODE: workload # Azure Blob Storage with Workload Identity LOG_STORE: azure AZURE_STORAGE_ACCOUNT: AZURE_STORAGE_CONTAINER: AZURE_AUTH_MODE: workload # Private Link outbound connectivity to Control Plane ALBUS_BASEPATH: "https://azure-cp.privatelink-az.portkey.ai/albus" CONTROL_PLANE_BASEPATH: "https://azure-cp.privatelink-az.portkey.ai/api/v1" SOURCE_SYNC_API_BASEPATH: "https://azure-cp.privatelink-az.portkey.ai/api/v1/sync" CONFIG_READER_PATH: "https://azure-cp.privatelink-az.portkey.ai/api/model-configs" service: type: LoadBalancer port: 80 # LB listener port containerPort: 8787 annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "false" service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol: "http" service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/v1/health" service.beta.kubernetes.io/azure-allowed-ip-ranges: 0.0.0.0/0 ``` **Azure Managed Redis (Access Key) with Managed Identity for Blob Storage** The following sample `values.yaml` shows how to configure Azure Managed Redis with an access key and Azure Blob Storage using the AKS Managed Identity. ```yaml theme={"system"} images: gatewayImage: repository: "docker.io/portkeyai/gateway_enterprise" pullPolicy: Always tag: "latest" dataserviceImage: repository: "docker.io/portkeyai/data-service" pullPolicy: Always tag: "latest" imageCredentials: - name: portkeyenterpriseregistrycredentials create: true registry: https://index.docker.io/v1/ username: password: environment: create: true secret: true data: ANALYTICS_STORE: control_plane SERVICE_NAME: gateway PORTKEY_CLIENT_AUTH: ORGANISATIONS_TO_SYNC: PORT: "8787" # Azure Managed Redis with Access Key CACHE_STORE: azure-redis REDIS_URL: "redis://:" REDIS_TLS_ENABLED: "true" REDIS_PASSWORD: # Azure Blob Storage with Managed Identity LOG_STORE: azure AZURE_STORAGE_ACCOUNT: AZURE_STORAGE_CONTAINER: AZURE_AUTH_MODE: managed service: type: LoadBalancer port: 80 # LB listener port containerPort: 8787 annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true" service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol: "http" service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/v1/health" ``` **AI Gateway + MCP Gateway with NGINX Ingress** The following sample `values.yaml` shows how to deploy both AI Gateway and MCP Gateway with host-based routing using the NGINX Ingress controller, built-in Redis, and Workload Identity for Blob Storage. ```yaml theme={"system"} images: gatewayImage: repository: "docker.io/portkeyai/gateway_enterprise" pullPolicy: Always tag: "latest" dataserviceImage: repository: "docker.io/portkeyai/data-service" pullPolicy: Always tag: "latest" redisImage: repository: "docker.io/redis" pullPolicy: IfNotPresent tag: "7.2-alpine" imageCredentials: - name: portkeyenterpriseregistrycredentials create: true registry: https://index.docker.io/v1/ username: password: serviceAccount: create: true automount: true name: annotations: azure.workload.identity/client-id: podLabels: azure.workload.identity/use: "true" environment: create: true secret: true data: ANALYTICS_STORE: control_plane SERVICE_NAME: gateway PORTKEY_CLIENT_AUTH: ORGANISATIONS_TO_SYNC: PORT: "8787" SERVER_MODE: "all" MCP_PORT: "8788" MCP_GATEWAY_BASE_URL: "https://mcp.example.com" # Built-in Redis CACHE_STORE: redis REDIS_URL: "redis://redis:6379" REDIS_TLS_ENABLED: "false" # Azure Blob Storage with Workload Identity LOG_STORE: azure AZURE_STORAGE_ACCOUNT: AZURE_STORAGE_CONTAINER: AZURE_AUTH_MODE: workload ingress: enabled: true ingressClassName: "nginx-static" hostname: "gateway.example.com" hostBased: true mcpHostname: "mcp.example.com" annotations: service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/v1/health" ```