> ## Documentation Index
> Fetch the complete documentation index at: https://docs.portkey.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# LocalAI

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including your **locally hosted models through** [**LocalAI**](https://localai.io/).

## Portkey SDK Integration with LocalAI

### 1. Install the Portkey SDK

<Tabs>
  <Tab title="NodeJS">
    ```sh theme={"system"}
    npm install --save portkey-ai
    ```
  </Tab>

  <Tab title="Python">
    ```sh theme={"system"}
    pip install portkey-ai
    ```
  </Tab>
</Tabs>

### 2. Initialize Portkey with LocalAI URL

First, ensure that your API is externally accessible. If you're running the API on `http://localhost`, consider using a tool like `ngrok` to create a public URL. Then, instantiate the Portkey client by adding your LocalAI URL (along with the version identifier) to the `customHost` property, and add the provider name as `openai`.

<Note>
  **Note:** Don't forget to include the version identifier (e.g., `/v1`) in the `customHost` URL
</Note>

<Tabs>
  <Tab title="NodeJS SDK">
    ```js theme={"system"}
    import Portkey from 'portkey-ai'

    const portkey = new Portkey({
        apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
        provider: "openai",
        customHost: "https://7cc4-3-235-157-146.ngrok-free.app/v1" // Your LocalAI ngrok URL
    })
    ```
  </Tab>

  <Tab title="Python SDK">
    ```python theme={"system"}
    from portkey_ai import Portkey

    portkey = Portkey(
        api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
        provider="openai",
        custom_host="https://7cc4-3-235-157-146.ngrok-free.app/v1" # Your LocalAI ngrok URL
    )
    ```
  </Tab>
</Tabs>

<Note>
  Portkey currently supports all endpoints that adhere to the OpenAI specification. This means, you can access and observe any of your LocalAI models that are exposed through OpenAI-compliant routes.
</Note>

[List of supported endpoints here](/integrations/llms/local-ai#localai-endpoints-supported).

### 3. Invoke Chat Completions

Use the Portkey SDK to invoke chat completions from your LocalAI model, just as you would with any other provider.

<Tabs>
  <Tab title="NodeJS SDK">
    ```js theme={"system"}
    const chatCompletion = await portkey.chat.completions.create({
        messages: [{ role: 'user', content: 'Say this is a test' }],
        model: 'ggml-koala-7b-model-q4_0-r2.bin',
    });

    console.log(chatCompletion.choices);
    ```
  </Tab>

  <Tab title="Python SDK">
    ```python theme={"system"}
    completion = portkey.chat.completions.create(
        messages= [{ "role": 'user', "content": 'Say this is a test' }],
        model= 'ggml-koala-7b-model-q4_0-r2.bin'
    )

    print(completion)
    ```
  </Tab>
</Tabs>

## [Using Virtual Keys](https://app.portkey.ai/virtual-keys)

Virtual Keys serve as Portkey's unified authentication system for all LLM interactions, simplifying the use of multiple providers and Portkey features within your application. For self-hosted LLMs, you can configure custom authentication requirements including authorization keys, bearer tokens, or any other headers needed to access your model:

<Frame>
  <img src="https://mintcdn.com/portkey-docs/wAHXB_jjwLt8bYcN/images/private-llm.png?fit=max&auto=format&n=wAHXB_jjwLt8bYcN&q=85&s=b8aa18e1d761c2dd1dac020d0d5727fb" width="1174" height="914" data-path="images/private-llm.png" />
</Frame>

1. Navigate to [Virtual Keys](https://app.portkey.ai/virtual-keys) in your Portkey dashboard
2. Click **"Add Key"** and enable the **"Local/Privately hosted provider"** toggle
3. Configure your deployment:
   * Select the matching provider API specification (typically `OpenAI`)
   * Enter your model's base URL in the `Custom Host` field
   * Add required authentication headers and their values
4. Click **"Create"** to generate your virtual key

You can now use this virtual key in your requests:

<Tabs>
  <Tab title="NodeJS">
    ```js theme={"system"}
    const portkey = new Portkey({
        apiKey: "PORTKEY_API_KEY",
        virtualKey: "YOUR_SELF_HOSTED_LLM_VIRTUAL_KEY"

    async function main() {
      const response = await client.chat.completions.create({
        messages: [{ role: "user", content: "Bob the builder.." }],
        model: "your-self-hosted-model-name",
      });

    console.log(response.choices[0].message.content);
    })
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={"system"}
    portkey = Portkey(
        api_key="PORTKEY_API_KEY",
        virtual_key="YOUR_SELF_HOSTED_LLM_VIRTUAL_KEY"
    )

    response = portkey.chat.completions.create(
      model="your-self-hosted-model-name",
      messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"}
      ]

    print(response)
    )
    ```
  </Tab>
</Tabs>

For more information about managing self-hosted LLMs with Portkey, see [Bring Your Own LLM](/integrations/llms/byollm).

## LocalAI Endpoints Supported

| Endpoint                                        | Resource                                                          |
| :---------------------------------------------- | :---------------------------------------------------------------- |
| /chat/completions (Chat, Vision, Tools support) | [Doc](/provider-endpoints/chat)                                   |
| /images/generations                             | [Doc](/provider-endpoints/images/create-image)                    |
| /embeddings                                     | [Doc](/provider-endpoints/embeddings)                             |
| /audio/transcriptions                           | [Doc](/product/ai-gateway/multimodal-capabilities/speech-to-text) |

## Next Steps

Explore the complete list of features supported in the SDK:

<Card title="SDK" href="/api-reference/portkey-sdk-client" />

***

You'll find more information in the relevant sections:

1. [Add metadata to your requests](/product/observability/metadata)
2. [Add gateway configs to your Ollama requests](/product/ai-gateway/universal-api#ollama-in-configs)
3. [Tracing Ollama requests](/product/observability/traces)
4. [Setup a fallback from OpenAI to Ollama APIs](/product/ai-gateway/fallbacks)