LLM Integrations

The Integrations page is the central command center for Organization Admins. It’s where you securely manage all third-party LLM provider credentials and govern their use across all workspaces from a single, unified dashboard.

This “create once, provision many” model saves significant time, reduces configuration errors, and gives you complete oversight of your AI stack.

​

Understanding the Integrations Dashboard

The Integrations page is organized into three tabs, each serving a distinct purpose:

• All: This is a comprehensive list of all 50+ providers Portkey supports. This is your starting point for connecting a new provider to your organization.
• Connected: This tab lists all the integrations that you have personally connected at the organization level. It’s your primary view for managing your centrally-governed providers.
• Workspace-Created: This tab gives you complete visibility and governance over any integrations created by Workspace Admins for their specific workspaces. It ensures that even with delegated control, you maintain a full audit trail and can manage these instances if needed.

​

Creating and Provisioning a New Integration

This guide walks you through connecting a new provider and making it available to your workspaces.

​

Step 1: Connect the Provider

1. From the All tab, find the provider you want to connect (e.g., OpenAI, Azure OpenAI, AWS Bedrock) and click Connect.
2. Fill in the details:
   • Integration Name: A friendly name for you to identify this connection (e.g., “Azure Production - US East”).
   • Slug: A unique, URL-friendly identifier. This will be used by developers to call models (e.g., azure-prod-useast).
   • Credentials: Securely enter your API keys or other authentication details. These are encrypted and will not be visible after saving.
3. Click Next.

Special Case: Connecting to Azure OpenAI

Portkey allows you to manage multiple Azure deployments under a single integration. This is useful when different models or versions are in separate deployments.

1. When connecting an Azure integration, you can add multiple deployments.
2. Assign a unique alias to each deployment (e.g., gpt-4o-deploy, gpt-35-deploy).
3. Set one deployment as the default.

To use a specific deployment, developers will pass its alias as the model in their LLM request. If no alias is provided, the default deployment is used.

Special Case: Connecting to Self-Hosted LLMs

You can manage your privately hosted models alongside commercial providers.

1. When creating a new integration, enable the “Local/Privately hosted provider” toggle.
2. Select the API specification your LLM implements (e.g., OpenAI).
3. Enter your model’s base URL in the “Custom Host” field and add any required authentication headers.

This allows you to use your self-hosted models with all Portkey features. For more details, see Bring Your Own LLM.

​

Step 2: Provision to Workspaces

Here, you decide which teams get access to this provider and under what conditions.

1. You will see a list of all workspaces within your organization.
2. Use the toggle next to a workspace name to enable or disable access.
3. For each enabled workspace, you can optionally click Edit Budget & Rate Limits to set specific spending caps or request limits that apply only to that workspace for this integration.
4. (Optional) For Provisioning to New Workspaces: Toggle on “Automatically provision this integration for new workspaces” to ensure any future teams automatically get access with a default budget/rate limit you define.
5. Click Next.

​

Step 3: Provision Specific Models

This is where you enforce model governance and control costs.

1. You will see a list of all models available from the provider you’re connecting.
2. You can Clear all and then select only the models you wish to approve for use.
3. (Optional) For Dynamic Models: If you’re using a provider like Fireworks AI with many community models, you can toggle on “Automatically enable new models”. For providers like OpenAI or Azure, we recommend an explicit allow-list for better cost control.
4. Click Create Integration.

That’s it! You have successfully created and provisioned a centrally managed integration. It will now appear in your Connected tab. The workspaces you provisioned will see this as an available “AI Provider” in their Model Catalog, with access only to the models you specified and constrained by the budgets you set.

​

Setting Budgets and Rate Limits

When provisioning an integration to a workspace, you can enforce powerful governance rules by setting budget and rate limits. This gives you granular control over costs and usage patterns.

Click on Edit Budget & Rate Limits for any workspace to configure the following:

• Cost-Based Budgets: Set a maximum budget in USD. Once the limit is reached, the provider will be disabled for that workspace to prevent overspending.
• Token-Based Budgets: Set a maximum number of tokens that can be consumed.
• Request-Based Rate Limits: Control the number of requests allowed per minute, hour, or day.
• Token-Based Rate Limits: Control the number of tokens consumed per minute, hour, or day.

You can also configure periodic resets (weekly or monthly) for these limits, which is perfect for managing recurring team budgets.

​

FAQs