Portkey’s Bring Your Own LLM feature allows you to seamlessly integrate privately hosted language models into your AI infrastructure. This powerful capability enables unified management of both private and commercial LLMs through a consistent interface while leveraging Portkey’s comprehensive suite of observability and reliability features.

​
Key Benefits

  • Unified API Access: Manage private and commercial LLMs through a single, consistent interface
  • Enhanced Reliability: Leverage Portkey’s fallbacks, retries, and load balancing for your private deployments
  • Comprehensive Monitoring: Track performance, usage, and costs alongside your commercial LLM usage
  • Simplified Access Control: Manage team-specific permissions and usage limits
  • Secure Credential Management: Protect sensitive authentication details through Portkey’s secure vault

​
Integration Options

Prerequisites

Your private LLM must implement an API specification compatible with one of Portkey’s supported providers (e.g., OpenAI’s /chat/completions, Anthropic’s /messages, etc.).

Portkey offers two primary methods to integrate your private LLMs:

  1. Using Virtual Keys: Store your deployment details securely in Portkey’s vault
  2. Direct Integration: Pass deployment details in your requests without storing them

​
Option 1: Using Virtual Keys

​
Step 1: Add Your Deployment Details

Navigate to the Virtual Keys section in your Portkey dashboard and create a new Virtual Key.

Adding a private LLM as a Virtual Key

  1. Click “Add Key” and enable the “Local/Privately hosted provider” toggle
  2. Configure your deployment:
    • Select the matching provider API specification (typically OpenAI)
    • Enter your model’s base URL in the Custom Host field
    • Add required authentication headers and their values
  3. Click “Create” to generate your virtual key

​
Step 2: Use Your Virtual Key in Requests

After creating your virtual key, you can use it in your applications:

import Portkey from 'portkey-ai'

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    virtualKey: "YOUR_PRIVATE_LLM_VIRTUAL_KEY"
})

async function main() {
  const response = await portkey.chat.completions.create({
    messages: [{ role: "user", content: "Explain quantum computing in simple terms" }],
    model: "YOUR_MODEL_NAME",  // The model name your private deployment expects
  });

  console.log(response.choices[0].message.content);
}

main();

​
Option 2: Direct Integration Without Virtual Keys

If you prefer not to store your private LLM details in Portkey’s vault, you can pass them directly in your API requests:

import Portkey from 'portkey-ai'

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    provider: "openai",  // The API spec your LLM implements
    customHost: "https://your-llm-server.com/v1/",  // Include the API version
    Authorization: "Bearer YOUR_AUTH_TOKEN",  // Optional: Any auth headers needed
    forwardHeaders: ["Authorization"]  // Headers to forward without processing
})

async function main() {
  const response = await portkey.chat.completions.create({
    messages: [{ role: "user", content: "Explain quantum computing in simple terms" }],
    model: "YOUR_MODEL_NAME",
  });

  console.log(response.choices[0].message.content);
}

main();

The custom_host must include the API version path (e.g., /v1/). Portkey will automatically append the endpoint path (/chat/completions, /completions, or /embeddings).

​
Securely Forwarding Sensitive Headers

For headers containing sensitive information that shouldn’t be logged or processed by Portkey, use the forward_headers parameter to pass them directly to your private LLM:

import Portkey from 'portkey-ai'

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    provider: "openai",
    customHost: "https://your-llm-server.com/v1/",
    // Headers to include with requests
    Authorization: "Bearer sk_live_xxxxx",
    xApiKey: "sensitive-key-value",
    xOrgId: "org-12345",
    // Headers to forward without processing
    forwardHeaders: ["Authorization", "xApiKey", "xOrgId"]
})

In the JavaScript SDK, convert header names to camelCase. For example, X-My-Custom-Header becomes xMyCustomHeader.

​
Using Forward Headers in Gateway Configs

You can also specify forward_headers in your Gateway Config for consistent header forwarding:

{
  "strategy": { "mode": "fallback" },
  "targets": [
    {
      "provider": "openai",
      "custom_host": "https://your-private-llm.com/v1",
      "forward_headers": ["Authorization", "x-api-key", "x-custom-token"]
    },
    {
      "provider": "openai",
      "api_key": "sk-xxxxx"  // Fallback to commercial provider
    }
  ]
}

​
Advanced Features

​
Using Private LLMs with Gateway Configs

Private LLMs work seamlessly with all Portkey Gateway features. Some common use cases:

  • Load Balancing: Distribute traffic across multiple private LLM instances
  • Fallbacks: Set up automatic failover between private and commercial LLMs
  • Conditional Routing: Route requests to different LLMs based on metadata
{
    "strategy": {
        "mode": "fallback"
    },
    "targets": [
        {
            "provider": "openai",
            "custom_host": "http://PRIVATE_LLM/v1",
            "forward_headers": ["Authorization"]
        },
        {
            "virtual_key": "openai-key"
        }
    ]
}

Learn more about Gateway Configs.

​
Monitoring and Analytics

Portkey provides comprehensive observability for your private LLM deployments, just like it does for commercial providers:

  • Log Analysis: View detailed request and response logs
  • Performance Metrics: Track latency, token usage, and error rates
  • User Attribution: Associate requests with specific users via metadata

Portkey Analytics Dashboard for Private LLMs

​
Troubleshooting

IssuePossible CausesSolutions
Connection ErrorsIncorrect URL, network issues, firewall rulesVerify URL format, check network connectivity, confirm firewall allows traffic
Authentication FailuresInvalid credentials, incorrect header formatCheck credentials, ensure headers are correctly formatted and forwarded
Timeout ErrorsLLM server overloaded, request too complexAdjust timeout settings, implement load balancing, simplify requests
Inconsistent ResponsesDifferent model versions, configuration differencesStandardize model versions, document expected behavior differences

​
FAQs

​
Next Steps

Explore these related resources to get the most out of your private LLM integration:

Universal API

Adding Metadata

Gateway Configs

Request Tracing

Was this page helpful?

Suggest editsRaise issue