Portkey provides a robust and secure gateway to seamlessly integrate open-source and fine-tuned LLMs from Predibase into your applications. With Portkey, you can leverage powerful features like fast AI gateway, caching, observability, prompt management, and more, while securely managing your LLM API keys through a virtual key system.
Portkey SDK Integration with Predibase
Using Portkey, you can call your Predibase models in the familar OpenAI-spec and try out your existing pipelines on Predibase fine-tuned models with 2 LOC change.
1. Install the Portkey SDK
Install the Portkey SDK in your project using npm or pip:
npm install --save portkey-ai
2. Initialize Portkey with the Virtual Key
To use Predibase with Portkey, get your API key from here, then add it to Portkey to create the virtual key.
NodeJS SDK
Python SDK
OpenAI Node SDK
OpenAI Python SDK
import Portkey from 'portkey-ai'
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
provider:"@PROVIDER" // Your Predibase Virtual Key
})
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY", # Replace with your Portkey API key
provider="@PROVIDER" # Replace with your virtual key for Predibase
)
import OpenAI from "openai";
import { PORTKEY_GATEWAY_URL, createHeaders } from "portkey-ai";
const portkey = new OpenAI({
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
apiKey: "PORTKEY_API_KEY",
provider:"@PREDIBASE_PROVIDER",
}),
});
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
portkey = OpenAI(
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
api_key="PORTKEY_API_KEY",
provider="@PREDIBASE_PROVIDER"
)
)
3. Invoke Chat Completions on Predibase Serverless Endpoints
Predibase offers LLMs like Llama 3, Mistral, Gemma, etc. on its serverless infra that you can query instantly.
Sending Predibase Tenand ID
Predibase expects your account tenant ID along with the API key in each request. With Portkey, you can send your Tenand ID with the user param while making your request.const chatCompletion = await portkey.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'llama-3-8b ',
user: 'PREDIBASE_TENANT_ID'
});
console.log(chatCompletion.choices);
completion = portkey.chat.completions.create(
messages= [{ "role": 'user', "content": 'Say this is a test' }],
model= 'llama-3-8b',
user= "PREDIBASE_TENANT_ID"
)
print(completion)
curl https://api.portkey.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-provider: $PREDIBASE_PROVIDER" \
-d '{
"messages": [{"role": "user","content": "Hello!"}],
"model": "llama-3-8b",
"user": "PREDIBASE_TENANT_ID"
}'
4. Invoke Predibase Fine-Tuned Models
With Portkey, you can send your fine-tune model & adapter details directly with the model param while making a request.
The format is:model = <base_model>:<adapter-repo-name/adapter-version-number>
llama-3-8b and the adapter repo name is sentiment-analysis, you can make a request like this:
const chatCompletion = await portkey.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'llama-3-8b:sentiment-analysis/1',
user: 'PREDIBASE_TENANT_ID'
});
console.log(chatCompletion.choices);
completion = portkey.chat.completions.create(
messages= [{ "role": 'user', "content": 'Say this is a test' }],
model= 'llama-3-8b:sentiment-analysis/1',
user= "PREDIBASE_TENANT_ID"
)
print(completion)
curl https://api.portkey.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-provider: $PREDIBASE_PROVIDER" \
-d '{
"messages": [{"role": "user","content": "Hello!"}],
"model": "llama-3-8b:sentiment-analysis/1",
"user": "PREDIBASE_TENANT_ID"
}'
Routing to Dedicated Deployments
Using Portkey, you can easily route to your dedicatedly deployed models as well. Just pass the dedicated deployment name in the model param:
model = "my-dedicated-mistral-deployment-name"
JSON Schema Mode
You can enforce JSON schema for all Predibase models - just set the response_format to json_object and pass the relevant schema while making your request. Portkey logs will show your JSON output separately
const chatCompletion = await portkey.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'llama-3-8b ',
user: 'PREDIBASE_TENANT_ID',
response_format: {
"type": "json_object",
"schema": {"properties": {
"name": {"maxLength": 10, "title": "Name", "type": "string"},
"age": {"title": "Age", "type": "integer"},
"required": ["name", "age", "strength"],
"title": "Character",
"type": "object"
}
}
});
console.log(chatCompletion.choices);
# Using Pydantic to define the schema
from pydantic import BaseModel, constr
# Define JSON Schema
class Character(BaseModel):
name: constr(max_length=10)
age: int
strength: int
completion = portkey.chat.completions.create(
messages= [{ "role": 'user', "content": 'Say this is a test' }],
model= 'llama-3-8b',
user= "PREDIBASE_TENANT_ID",
response_format={
"type": "json_object",
"schema": Character.schema(),
},
)
print(completion)
curl https://api.portkey.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-provider: $PREDIBASE_PROVIDER" \
-d '{
"messages": [{"role": "user","content": "Hello!"}],
"model": "llama-3-8b",
"user": "PREDIBASE_TENANT_ID",
"response_format": {
"type": "json_object",
"schema": {"properties": {
"name": {"maxLength": 10, "title": "Name", "type": "string"},
"age": {"title": "Age", "type": "integer"},
"required": ["name", "age", "strength"],
"title": "Character",
"type": "object"
}
}
}'
Next Steps
The complete list of features supported in the SDK are available on the link below.
You’ll find more information in the relevant sections:
- Add metadata to your requests
- Add gateway configs to your Predibase requests
- Tracing Predibase requests
- Setup a fallback from OpenAI to Predibase
