Bring Your Own LLM

Portkey provides a robust and secure platform to observe, integrate, and manage your locally or privately hosted custom models.

Integrating Custom Models with Portkey SDK

You can integrate any custom LLM with Portkey as long as it's API is compliant with any of the 15+ providers Portkey already supports.

1. Install the Portkey SDK

npm install --save portkey-ai

pip install portkey-ai

2. Initialize Portkey with your Custom URL

Instead of using a provider + authorization pair or a virtualKey referring to the provider, you can specify a provider + custom_host pair while instantiating the Portkey client.

custom_host here refers to the URL where your custom model is hosted, including the API version identifier.

import Portkey from 'portkey-ai'
 
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    provider: "PROVIDER_NAME", // This can be mistral-ai, openai, or anything else
    customHost: "http://MODEL_URL/v1/", // Your custom URL with version identifier
    authorization: "AUTH_KEY", // If you need to pass auth
})

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    provider="PROVIDER_NAME", # This can be mistral-ai, openai, or anything else
    custom_host="http://MODEL_URL/v1/", # Your custom URL with version identifier
    authorization="AUTH_KEY", # If you need to pass auth
)

3. Invoke Chat Completions

Use the Portkey SDK to invoke chat completions from your model, just as you would with any other provider.

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }]
});

console.log(chatCompletion.choices);

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }]
)

print(completion)

Forward Sensitive Headers Securely

When integrating custom LLMs with Portkey, you may have sensitive information in your request headers that you don't want Portkey to track or log. Portkey provides a secure way to forward specific headers directly to your model's API without any processing.

Just specify an array of header names using the forward_headers property when initializing the Portkey client. Portkey will then forward these headers directly to your custom host URL without logging or tracking them.

Here's an example:

import Portkey from 'portkey-ai'
 
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    provider: "PROVIDER_NAME", // This can be mistral-ai, openai, or anything else
    customHost: "http://MODEL_URL/v1/", // Your custom URL with version identifier
    authorization: "AUTH_KEY", // If you need to pass auth
    forwardHeaders: [ "authorization" ]
})

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    provider="PROVIDER_NAME", # This can be mistral-ai, openai, or anything else
    custom_host="http://MODEL_URL/v1/", # Your custom URL with version identifier
    authorization="AUTH_KEY", # If you need to pass auth
    forward_headers= [ "authorization" ]
)

x-portkey-forward-headers accepts comma separated header names

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-provider: $PROVIDER_NAME" \
  -H "x-portkey-custom-host: https://MODEL_URL/v1" \
  -H "x-api-key: $API_KEY" \
  -H "x-secret-access-key: $ACCESS_KEY" \
  -H "x-key-id: $KEY_ID" \
  -H "x-portkey-forward-headers: x-api-key, x-secret-access-key, x-key-id" \
  -d '{
    "model": "llama2",
    "messages": [{ "role": "user", "content": "Say this is a test" }]
  }'

Forward Headers in the Config Object

You can also define forward_headers in your Config object and then pass the headers directly while making a request.

{
    "strategy": {
        "mode": "loadbalance"
    },
    "targets": [
        {
            "provider": "openai",
            "api_key": "<api-key>"
        },
        {
            "strategy": {
                "mode": "fallback"
            },
            "targets": [
                {
                    "provider": "azure-openai",
                    "custom_host": "http://MODEL_URL/v1",
                    "forward_headers": ["my-auth-header-1", "my-auth-header-2"]
                },
                {
                    "provider": "openai",
                    "api_key": "sk-***"
                }
            ]
        }
    ]
}

Next Steps

Explore the complete list of features supported in the SDK:

pagePortkey SDK Client

You'll find more information in the relevant sections:

PreviousOllama NextSegmind

Last updated 24 days ago