Ollama

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including your locally hosted models through Ollama.

​

Portkey SDK Integration with Ollama Models

Portkey provides a consistent API to interact with models from various providers.

​

1. Expose your Ollama API

Expose your Ollama API by using a tunneling service like ngrok or any other way you prefer.

You can skip this step if you’re self-hosting the Gateway.

For using Ollama with ngrok, here’s a useful guide

ngrok http 11434 --host-header="localhost:11434"

​

2. Install the Portkey SDK

Install the Portkey SDK in your application to interact with your Ollama API through Portkey.

• NodeJS
• Python

npm install --save portkey-ai

​

3. Initialize Portkey with Ollama URL

Instantiate the Portkey client by adding your Ollama publicly-exposed URL to the customHost property.

• NodeJS SDK
• Python SDK

import Portkey from 'portkey-ai'

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    provider: "ollama",
    customHost: "https://7cc4-3-235-157-146.ngrok-free.app" // Your Ollama ngrok URL
})

​

4. Invoke Chat Completions with Ollama

Use the Portkey SDK to invoke chat completions from your Ollama model, just as you would with any other provider.

• NodeJS SDK
• Python SDK
• cURL

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'llama3',
});

console.log(chatCompletion.choices);

​

Local Setup (npm or docker)

First, install the Gateway locally:

• Install the Gateway
• Docker Image

npx @portkey-ai/gateway

Then, just change the baseURL to the Gateway URL, customHost to the Ollam URL, and make requests.

​

Next Steps

Explore the complete list of features supported in the SDK:

You’ll find more information in the relevant sections:

1. Add metadata to your requests
2. Add gateway configs to your Ollama requests
3. Tracing Ollama requests
4. Setup a fallback from OpenAI to Ollama APIs