Bring Your Own LLM

The AI Gateway’s Bring Your Own LLM feature allows you to seamlessly integrate privately hosted language models into your AI infrastructure. This powerful capability enables unified management of both private and commercial LLMs through a consistent interface while leveraging the AI Gateway’s comprehensive suite of observability and reliability features.

Key Benefits

Unified API Access: Manage private and commercial LLMs through a single, consistent interface
Enhanced Reliability: Leverage the AI Gateway’s fallbacks, retries, and load balancing for your private deployments
Comprehensive Monitoring: Track performance, usage, and costs alongside your commercial LLM usage
Simplified Access Control: Manage team-specific permissions and usage limits
Secure Credential Management: Protect sensitive authentication details through the AI Gateway’s secure vault

Integration Options

PrerequisitesYour private LLM must implement an API specification compatible with one of the AI Gateway’s supported providers (e.g., OpenAI’s /chat/completions, Anthropic’s /messages, etc.).

The AI Gateway offers two primary methods to integrate your private LLMs:

Using Model Catalog: Store your deployment details securely in the AI Gateway’s Model Catalog
Direct Integration: Pass deployment details in your requests without storing them

Option 1: Using Model Catalog

Step 1: Add Your Private LLM to Model Catalog

Navigate to Model Catalog → Add Provider in Strata Cloud Manager.

Click “Add Provider” and enable the “Local/Privately hosted provider” toggle
Configure your deployment:
- Select the matching provider API specification (typically OpenAI)
- Enter your model’s base URL in the Custom Host field
- Add required authentication headers and their values
Name your provider (e.g., my-private-llm)
Click “Create” to save your configuration

Step 2: Use Your Provider in Requests

After adding your provider to the Model Catalog, you can use it in your applications:

curl https://aigw.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $PORTKEY_API_KEY" \
  -d '{
    "model": "@my-private-llm/YOUR_MODEL_NAME",
    "messages": [
      { "role": "user", "content": "Explain quantum computing in simple terms" }
    ]
  }'

Option 2: Direct Integration Without Model Catalog

If you prefer not to store your private LLM details in the AI Gateway’s Model Catalog, you can pass them directly in your API requests:

curl https://aigw.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-provider: openai" \
  -H "x-portkey-custom-host: https://your-llm-server.com/v1" \
  -H "Authorization: Bearer YOUR_AUTH_TOKEN" \
  -H "x-portkey-forward-headers: Authorization" \
  -d '{
    "model": "YOUR_MODEL_NAME",
    "messages": [
      { "role": "user", "content": "Explain quantum computing in simple terms" }
    ]
  }'

The custom_host must include the API version path (e.g., /v1/). The AI Gateway will automatically append the endpoint path (/chat/completions, /completions, or /embeddings).

The AI Gateway blocks requests to private and reserved IP ranges by default. If your private LLM runs on an internal network IP, see Custom hosts for blocked patterns and how to allowlist specific hosts.

Securely Forwarding Sensitive Headers

For headers containing sensitive information that shouldn’t be logged or processed by the AI Gateway, use the forward_headers parameter to pass them directly to your private LLM:

cURL

curl https://aigw.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-provider: openai" \
  -H "x-portkey-custom-host: https://your-llm-server.com/v1" \
  -H "Authorization: Bearer sk_live_xxxxx" \
  -H "x-api-key: sensitive-key-value" \
  -H "x-org-id: org-12345" \
  -H "x-portkey-forward-headers: Authorization, x-api-key, x-org-id" \
  -d '{
    "model": "YOUR_MODEL_NAME",
    "messages": [{ "role": "user", "content": "Hello!" }]
  }'

Using Forward Headers in Gateway Configs

You can also specify forward_headers in your Gateway Config for consistent header forwarding:

{
  "strategy": { "mode": "fallback" },
  "targets": [
    {
      "provider": "openai",
      "custom_host": "https://your-private-llm.com/v1",
      "forward_headers": ["Authorization", "x-api-key", "x-custom-token"]
    },
    {
      "provider": "openai",
      "api_key": "sk-xxxxx"  // Fallback to commercial provider
    }
  ]
}

Advanced Features

Using Private LLMs with Gateway Configs

Private LLMs work seamlessly with all AI Gateway features. Some common use cases:

Load Balancing: Distribute traffic across multiple private LLM instances
Fallbacks: Set up automatic failover between private and commercial LLMs
Conditional Routing: Route requests to different LLMs based on metadata

{
    "strategy": {
        "mode": "fallback"
    },
    "targets": [
        {
            "provider": "openai",
            "custom_host": "http://PRIVATE_LLM/v1",
            "forward_headers": ["Authorization"]
        },
        {
            "provider":"@openai-key"
        }
    ]
}

Learn more about Gateway Configs.

Monitoring and Analytics

The AI Gateway provides comprehensive observability for your private LLM deployments, just like it does for commercial providers:

Log Analysis: View detailed request and response logs
Performance Metrics: Track latency, token usage, and error rates
User Attribution: Associate requests with specific users via metadata

Troubleshooting

Issue	Possible Causes	Solutions
Connection Errors	Incorrect URL, network issues, firewall rules	Verify URL format, check network connectivity, confirm firewall allows traffic
Authentication Failures	Invalid credentials, incorrect header format	Check credentials, ensure headers are correctly formatted and forwarded
Timeout Errors	LLM server overloaded, request too complex	Adjust timeout settings, implement load balancing, simplify requests
Inconsistent Responses	Different model versions, configuration differences	Standardize model versions, document expected behavior differences

FAQs

Can I use any private LLM with the AI Gateway?

Yes, as long as it implements an API specification compatible with one of the AI Gateway’s supported providers (OpenAI, Anthropic, etc.). The model should accept requests and return responses in the format expected by that provider.

How do I handle multiple deployment endpoints?

You have two options:

Create separate integration for each endpoint
Use Gateway Configs with load balancing to distribute traffic across multiple endpoints

Are there any request volume limitations?

The AI Gateway itself doesn’t impose specific request volume limitations for private LLMs. Your throughput will be limited only by your private LLM deployment’s capabilities and any rate limits you configure in the AI Gateway.

Can I use different models with the same private deployment?

Yes, you can specify different model names in your requests as long as your private LLM deployment supports them. The model name is passed through to your deployment.

Can I mix private and commercial LLMs in the same application?

Absolutely! One of the AI Gateway’s key benefits is the ability to manage both private and commercial LLMs through a unified interface. You can even set up fallbacks between them or route requests conditionally.

Next Steps

Explore these related resources to get the most out of your private LLM integration:

Integrations

LLM Integrations

Cloud Platforms

Guardrails

Plugins

Vector Databases

Agents

AI Apps

Libraries

Tracing Providers

MCP Clients

MCP Servers

Key Benefits

Integration Options

Option 1: Using Model Catalog

Step 1: Add Your Private LLM to Model Catalog

Step 2: Use Your Provider in Requests

Option 2: Direct Integration Without Model Catalog

Securely Forwarding Sensitive Headers

Using Forward Headers in Gateway Configs

Advanced Features

Using Private LLMs with Gateway Configs

Monitoring and Analytics

Troubleshooting

FAQs

Next Steps

Universal API

Adding Metadata

Gateway Configs

Request Tracing

​Key Benefits

​Integration Options

​Option 1: Using Model Catalog

​Step 1: Add Your Private LLM to Model Catalog

​Step 2: Use Your Provider in Requests

​Option 2: Direct Integration Without Model Catalog

​Securely Forwarding Sensitive Headers

​Using Forward Headers in Gateway Configs

​Advanced Features

​Using Private LLMs with Gateway Configs

​Monitoring and Analytics

​Troubleshooting

​FAQs

​Next Steps

Universal API

Adding Metadata

Gateway Configs

Request Tracing

Key Benefits

Integration Options

Option 1: Using Model Catalog

Step 1: Add Your Private LLM to Model Catalog

Step 2: Use Your Provider in Requests

Option 2: Direct Integration Without Model Catalog

Securely Forwarding Sensitive Headers

Using Forward Headers in Gateway Configs

Advanced Features

Using Private LLMs with Gateway Configs

Monitoring and Analytics

Troubleshooting

FAQs

Next Steps