Fireworks

Portkey provides a robust and secure gateway to facilitate the integration of various models into your apps, including chat, vision, image generation, and embedding models hosted on the Fireworks platform.

With Portkey, you can take advantage of features like fast AI gateway access, observability, prompt management, and more, all while ensuring the secure management of your LLM API keys through a virtual key system.

Portkey SDK Integration with Fireworks Models

Portkey provides a consistent API to interact with models from various providers. To integrate Fireworks with Portkey:

1. Install the Portkey SDK

NodeJS

npm install --save portkey-ai

Python

pip install portkey-ai

2. Initialize Portkey with the Virtual Key

Set up Portkey with your virtual key as part of the initialization configuration. You can create a virtual key in the UI.

NodeJS SDK

import Portkey from 'portkey-ai'
 
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // Defaults to process.env["PORTKEY_API_KEY"]
    virtualKey: "FIREWORKS_VIRTUAL_KEY" // Your Virtual Key
})

Python SDK

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Defaults to os.env("PORTKEY_API_KEY")
    virtual_key="FIREWORKS_VIRTUAL_KEY"   # Your Virtual Key
)

3. Invoke Chat Completions with Fireworks

You can use the Portkey instance now to send requests to Fireworks API.

NodeJS SDK

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'accounts/fireworks/models/llama-v3-70b-instruct',
});

console.log(chatCompletion.choices);

Python SDK

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'accounts/fireworks/models/llama-v3-70b-instruct'
)

print(completion)

Now, let's explore how you can use Portkey to call other models (vision, embedding, image) on the Fireworks API:

Using Embeddings Models

Call any embedding model hosted on Fireworks with the familiar OpenAI embeddings signature:

NodeJS SDK

const embeddings = await portkey.embeddings.create({
    input: "create vector representation on this sentence",
    model: "thenlper/gte-large",
});

console.log(embeddings);

Python SDK

embeddings = portkey.embeddings.create(
    input='create vector representation on this sentence',
    model='thenlper/gte-large'
)
print(embeddings)

Using Vision Models

Portkey natively supports vision models hosted on Fireworks:

NodeJS SDK

const completion = await portkey.chat.completions.create(
    messages: [
        { "role": "user", "content": [
            { "type": "text","text": "Can you describe this image?" },
                { "type": "image_url", "image_url":
                    { "url": "https://images.unsplash.com/photo-1582538885592-e70a5d7ab3d3?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=1770&q=80" }
                }
            ]
        }
    ],
    model: 'accounts/fireworks/models/firellava-13b'
)

console.log(completion);

Python SDK

completion = portkey.chat.completions.create(
    messages= [
        { "role": "user", "content": [
            { "type": "text","text": "Can you describe this image?" },
                { "type": "image_url", "image_url":
                    { "url": "https://images.unsplash.com/photo-1582538885592-e70a5d7ab3d3?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=1770&q=80" }
                }
            ]
        }
    ],
    model= 'accounts/fireworks/models/firellava-13b'
)

print(completion)

Using Image Generation Models

Portkey also supports calling image generation models hosted on Fireworks in the familiar OpenAI signature:

NodeJS SDK

import Portkey from 'portkey-ai';
import fs from 'fs';

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    virtualKey: "FIREWORKS_VIRTUAL_KEY"
});

async function main(){
    const image = await portkey.images.generate({
        model: "accounts/fireworks/models/stable-diffusion-xl-1024-v1-0",
        prompt: "An orange elephant in a purple pond"
    });

    const imageData = image.data[0].b64_json as string;
    fs.writeFileSync("fireworks-image-gen.png", Buffer.from(imageData, 'base64'));
}

main()

Python SDK

from portkey_ai import Portkey
import base64
from io import BytesIO
from PIL import Image

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    virtual_key="FIREWORKS_VIRTUAL_KEY"
)

image = portkey.images.generate(
  model="accounts/fireworks/models/stable-diffusion-xl-1024-v1-0",
  prompt="An orange elephant in a purple pond"
)

Image.open(BytesIO(base64.b64decode(image.data[0].b64_json))).save("fireworks-image-gen.png")

---

Fireworks Grammar Mode

Fireworks lets you define formal grammars to constrain model outputs. You can use it to force the model to generate valid JSON, speak only in emojis, or anything else. (Originally created by GGML)

Grammar mode is set with the response_format param. Just pass your grammar definition with {"type": "grammar", "grammar": grammar_definition}

Let's say you want to classify patient requests into 3 pre-defined classes:

Python

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Defaults to os.env("PORTKEY_API_KEY")
    virtual_key="FIREWORKS_VIRTUAL_KEY"   # Your Virtual Key
)

patient_classification = """
root      ::= diagnosis
diagnosis ::= "flu" | "dengue" | "malaria"
"""

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    response_format={"type": "grammar", "grammar": patient_classification},
    model= 'accounts/fireworks/models/llama-v3-70b-instruct'
)

print(completion)

NodeJS

import Portkey from 'portkey-ai'
 
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // Defaults to process.env["PORTKEY_API_KEY"]
    virtualKey: "FIREWORKS_VIRTUAL_KEY" // Your Virtual Key
})

const patient_classification = `
root ::= diagnosis
diagnosis ::= "flu" | "dengue" | "malaria"
`;

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    response_format: {"type": "grammar", "grammar": patient_classification},
    model: 'accounts/fireworks/models/llama-v3-70b-instruct',
});

console.log(chatCompletion.choices);

NOTE: Fireworks Grammer Mode is not supported on Portkey prompts playground

Explore the Fireworks guide for more examples and a deeper dive on Grammer node.

Fireworks JSON Mode

You can force the model to return (1) An arbitrary JSON, or (2) JSON with given schema with Fireworks' JSON mode.

Python

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Defaults to os.env("PORTKEY_API_KEY")
    virtual_key="FIREWORKS_VIRTUAL_KEY"   # Your Virtual Key
)

class Recipe(BaseModel):
    title: str
    description: str
    steps: List[str]
    
json_response = portkey.chat.completions.create(
    messages = [{ "role": 'user', "content": 'Give me a recipe for making Ramen, in JSON format' }],
    model = 'accounts/fireworks/models/llama-v3-70b-instruct',
    response_format = {
        "type":"json_object",
        "schema": Recipe.schema_json()
    }
)

print(json_response.choices[0].message.content)

NodeJS

import Portkey from 'portkey-ai'
 
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // Defaults to process.env["PORTKEY_API_KEY"]
    virtualKey: "FIREWORKS_VIRTUAL_KEY" // Your Virtual Key
})

asyn function main(){
  const json_response = await portkey.chat.completions.create({
    messages: [{role: "user",content: `Give me a recipe for making Ramen, in JSON format`}],
    model: "accounts/fireworks/models/llama-v3-70b-instruct",
    response_format: {
      type: "json_object",
      schema: {
        type: "object",
        properties: {
          title: { type: "string" },
          description: { type: "string" },
          steps: { type: "array" }
        }
      }
    }
  });
}

console.log(json_response.choices[0].message.content);

main()

Explore Fireworks docs for JSON mode for more examples.

Fireworks Function Calling

Portkey also supports function calling mode on Fireworks. Explore this cookbook for a deep dive and examples.

Managing Fireworks Prompts

You can manage all Fireworks prompts in the Prompt Library. All the current 49+ language models available on Fireworks are supported and you can easily start testing different prompts.

Once you're ready with your prompt, you can use the portkey.prompts.completions.create interface to use the prompt in your application.

Next Steps

The complete list of features supported in the SDK are available on the link below.

pagePortkey SDK Client

You'll find more information in the relevant sections:

1. Add metadata to your requests

2. Add gateway configs to your requests

3. Tracing requests

4. Setup a fallback from OpenAI to Firework APIs