> ## Documentation Index
> Fetch the complete documentation index at: https://docs.portkey.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Predibase
Portkey provides a robust and secure gateway to seamlessly integrate **open-source** and **fine-tuned** LLMs from Predibase into your applications. With Portkey, you can leverage powerful features like fast AI gateway, caching, observability, prompt management, and more, while securely managing your LLM API keys through a virtual key system.
Provider Slug. `predibase`
## Portkey SDK Integration with Predibase
Using Portkey, you can call your Predibase models in the familar **OpenAI-spec** and try out your existing pipelines on Predibase fine-tuned models with 2 LOC change.
### 1. Install the Portkey SDK
Install the Portkey SDK in your project using npm or pip:
```sh theme={"system"}
npm install --save portkey-ai
```
```sh theme={"system"}
pip install portkey-ai
```
### 2. Initialize Portkey with the Virtual Key
To use Predibase with Portkey, [get your API key from here](https://app.predibase.com/settings), then add it to Portkey to create the virtual key.
```js theme={"system"}
import Portkey from 'portkey-ai'
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
virtualKey: "VIRTUAL_KEY" // Your Predibase Virtual Key
})
```
```python theme={"system"}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY", # Replace with your Portkey API key
virtual_key="VIRTUAL_KEY" # Replace with your virtual key for Predibase
)
```
```js theme={"system"}
import OpenAI from "openai";
import { PORTKEY_GATEWAY_URL, createHeaders } from "portkey-ai";
const portkey = new OpenAI({
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
apiKey: "PORTKEY_API_KEY",
virtualKey: "PREDIBASE_VIRTUAL_KEY",
}),
});
```
```python theme={"system"}
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
portkey = OpenAI(
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
api_key="PORTKEY_API_KEY",
virtual_key="PREDIBASE_VIRTUAL_KEY"
)
)
```
### 3. Invoke Chat Completions on Predibase Serverless Endpoints
Predibase offers LLMs like **Llama 3**, **Mistral**, **Gemma**, etc. on its [serverless infra](https://docs.predibase.com/user-guide/inference/models#serverless-endpoints) that you can query instantly.
#### Sending Predibase Tenand ID
Predibase expects your **account tenant ID** along with the API key in each request. With Portkey, you can send [**your Tenand ID**](https://app.predibase.com/settings) with the `user` param while making your request.
```js theme={"system"}
const chatCompletion = await portkey.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'llama-3-8b ',
user: 'PREDIBASE_TENANT_ID'
});
console.log(chatCompletion.choices);
```
```python theme={"system"}
completion = portkey.chat.completions.create(
messages= [{ "role": 'user', "content": 'Say this is a test' }],
model= 'llama-3-8b',
user= "PREDIBASE_TENANT_ID"
)
print(completion)
```
```sh theme={"system"}
curl https://api.portkey.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-virtual-key: $PREDIBASE_VIRTUAL_KEY" \
-d '{
"messages": [{"role": "user","content": "Hello!"}],
"model": "llama-3-8b",
"user": "PREDIBASE_TENANT_ID"
}'
```
### 4. Invoke Predibase Fine-Tuned Models
With Portkey, you can send your fine-tune model & adapter details directly with the `model` param while making a request.
The format is:
`model = :`
For example, if your base model is `llama-3-8b` and the adapter repo name is `sentiment-analysis`, you can make a request like this:
```js theme={"system"}
const chatCompletion = await portkey.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'llama-3-8b:sentiment-analysis/1',
user: 'PREDIBASE_TENANT_ID'
});
console.log(chatCompletion.choices);
```
```python theme={"system"}
completion = portkey.chat.completions.create(
messages= [{ "role": 'user', "content": 'Say this is a test' }],
model= 'llama-3-8b:sentiment-analysis/1',
user= "PREDIBASE_TENANT_ID"
)
print(completion)
```
```sh theme={"system"}
curl https://api.portkey.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-virtual-key: $PREDIBASE_VIRTUAL_KEY" \
-d '{
"messages": [{"role": "user","content": "Hello!"}],
"model": "llama-3-8b:sentiment-analysis/1",
"user": "PREDIBASE_TENANT_ID"
}'
```
***
### Routing to Dedicated Deployments
Using Portkey, you can easily route to your dedicatedly deployed models as well. Just pass the dedicated deployment name in the `model` param:
`model = "my-dedicated-mistral-deployment-name"`
### JSON Schema Mode
You can enforce JSON schema for all Predibase models - just set the `response_format` to `json_object` and pass the relevant schema while making your request. Portkey logs will show your JSON output separately
```js theme={"system"}
const chatCompletion = await portkey.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'llama-3-8b ',
user: 'PREDIBASE_TENANT_ID',
response_format: {
"type": "json_object",
"schema": {"properties": {
"name": {"maxLength": 10, "title": "Name", "type": "string"},
"age": {"title": "Age", "type": "integer"},
"required": ["name", "age", "strength"],
"title": "Character",
"type": "object"
}
}
});
console.log(chatCompletion.choices);
```
```python theme={"system"}
# Using Pydantic to define the schema
from pydantic import BaseModel, constr
# Define JSON Schema
class Character(BaseModel):
name: constr(max_length=10)
age: int
strength: int
completion = portkey.chat.completions.create(
messages= [{ "role": 'user', "content": 'Say this is a test' }],
model= 'llama-3-8b',
user= "PREDIBASE_TENANT_ID",
response_format={
"type": "json_object",
"schema": Character.schema(),
},
)
print(completion)
```
```sh theme={"system"}
curl https://api.portkey.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-virtual-key: $PREDIBASE_VIRTUAL_KEY" \
-d '{
"messages": [{"role": "user","content": "Hello!"}],
"model": "llama-3-8b",
"user": "PREDIBASE_TENANT_ID",
"response_format": {
"type": "json_object",
"schema": {"properties": {
"name": {"maxLength": 10, "title": "Name", "type": "string"},
"age": {"title": "Age", "type": "integer"},
"required": ["name", "age", "strength"],
"title": "Character",
"type": "object"
}
}
}'
```
***
## Next Steps
The complete list of features supported in the SDK are available on the link below.
You'll find more information in the relevant sections:
1. [Add metadata to your requests](/product/observability/metadata)
2. [Add gateway configs to your Predibase requests](/product/ai-gateway/configs)
3. [Tracing Predibase requests](/product/observability/traces)
4. [Setup a fallback from OpenAI to Predibase](/product/ai-gateway/fallbacks)