7 Ways to Make Your Vercel AI SDK App Production-Ready

TL;DR
Learn how to leverage Portkey with Vercel AI SDK to create robust, scalable AI applications. We'll cover implementing guardrails, conditional routing, interoperability, reliability features, and observability to accelerate your journey from proof-of-concept to production.

In the fast-paced world of AI development, creating a proof-of-concept is just the beginning. The real challenge lies in building AI applications that are robust, scalable, and responsible enough for production environments. This guide will walk you through integrating Portkey with the Vercel AI SDK, empowering you to overcome common hurdles and accelerate your path to production.

Tools of Choice:

1. Portkey is an end-to-end LLM Ops platform that accelerates teams from proof-of-concept to production 10x faster. It offers:

• An AI Gateway for managing LLM requests
• A comprehensive LLM observability suite
• Robust guardrails for responsible AI development
• Interoperability across 250+ LLMs
• Built-in caching to save costs and time
• Advanced routing and fallback mechanisms

2. Vercel AI SDK: is a powerful open-source library that helps developers build conversational streaming user interfaces in JavaScript and TypeScript. Vercel AI SDK simplifies AI integration into web applications. It provides various tools and pre-built components that streamline the development process, reducing the time and effort needed to implement advanced features.

Why Use Portkey with Vercel AI SDK?

Combining Portkey with Vercel AI SDK allows developers to leverage the best of both worlds:

1. Vercel AI SDK's simplicity in building AI-powered interfaces
2. Portkey's advanced features for production-readiness, including observability, reliability, and responsible AI development

This combination empowers you to create AI applications that are not just functional, but also scalable, reliable, and compliant with best practices in AI development.

Getting Started

Prerequisites

Before we begin, ensure you have:

1. A Vercel project set up
2. Node.js and npm (or yarn) installed
3. Basic familiarity with Next.js and the Vercel AI SDK

Step 1: Installation

Start by installing the Portkey Vercel provider package:

npm install @portkey-ai/vercel-provider

Step 2: Configure Portkey

Sign up for a Portkey account to get your API key. Then, create a configuration file for Portkey:

import { createPortkey } from '@portkey-ai/vercel-provider';

const portkeyConfig = {
  "provider": "openai",
  "api_key": "YOUR_OPENAI_API_KEY",
  "override_params": {
    "model": "gpt-4o"
  }
};

const portkey = createPortkey({
  apiKey: 'YOUR_PORTKEY_API_KEY',
  config: portkeyConfig,
});

This configuration sets up Portkey to use OpenAI as the provider and specifies the GPT-4 model.

Step 3: Integrating with Vercel AI SDK Functions

Let's explore how to use Portkey with Vercel's generateText and streamText functions. Link to docs

Using generateText

Create a new file, e.g., generateTextAction.ts:

"use server";

import { generateText } from "ai";
import { createPortkey } from "@portkey-ai/vercel-provider";

export const generateTextAction = async () => {
  const llmClient = createPortkey({
    apiKey: "YOUR_PORTKEY_API_KEY",
    virtualKey: "YOUR_VIRTUAL_KEY"
  });

  const result = await generateText({
    model: llmClient.completionModel('gpt-3.5-turbo-instruct'),
    prompt: 'Tell me a joke',
  });

  return result.text;
};

Using streamText

For streaming responses, create a file named streamTextAction.ts:

"use server";

import { streamText } from "ai";
import { createPortkey } from "@portkey-ai/vercel-provider";
import { createStreamableValue } from "ai/rsc";

export const streamTextAction = async () => {
  const llmClient = createPortkey({
    apiKey: "YOUR_PORTKEY_API_KEY",
    virtualKey: "YOUR_VIRTUAL_KEY"
  });

  const result = await streamText({
    model: llmClient.completionModel('gpt-3.5-turbo-instruct'),
    temperature: 0.5,
    prompt: "Tell me a joke.",
  });
  return createStreamableValue(result.textStream).value;
};

7 Ways to Make Your App Production-Ready

Building a Gen AI app is simple but getting it into production is a different ball game. Portkey's AI Gateway and observability suite helps you go-to prod 10x faster with your vercel AI SDK app. Here are 5 techniques you can use in your app to make it productionready today

1. Implementing Guardrails

Portkey allows you to implement guardrails using its configuration system. Guardrails help ensure the safety and quality of your AI outputs. Here's how you can set up guardrails:

const portkeyConfig = {
  "retry": {
    "attempts": 3
  },
  "cache": {
    "mode": "simple"
  },
  "virtual_key": "openai-xxx",
  "before_request_hooks": [{
    "id": "input-guardrail-id-xx"
  }],
  "after_request_hooks": [{
    "id": "output-guardrail-id-xx"
  }]
};

In this configuration:

• before_request_hooks apply guardrails to the input before sending it to the LLM.
• after_request_hooks apply guardrails to the output received from the LLM.

You can obtain the guardrail IDs from the Portkey app, where you can create and manage your custom guardrails.

Real-world scenario: Imagine you're building a chatbot for a health institution. You could use input guardrails to ensure no sensitive information (like patient details) is sent to the LLM, and output guardrails to verify that the AI's responses don't contain any potentially harmful advice.

2. Conditional Routing

Conditional routing allows you to direct requests to different models or configurations based on specific conditions.  In this example we will route user's requests to different LLMs based on user metadata- free or paid:

const portkeyConfig = {
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.user_plan": { "$eq": "paid" } },
        "then": "finetuned-gpt4"
      },
      {
        "query": { "metadata.user_plan": { "$eq": "free" } },
        "then": "base-gpt4"
      }     
    ],
    "default": "base-gpt4"
  },
  "targets": [
    {
      "name": "finetuned-gpt4",
      "virtual_key":"xx"
    },
    {
      "name": "base-gpt4",
      "virtual_key":"yy"
    }
  ]
};

This configuration routes requests to different models based on the user's plan, demonstrating how you can tailor your AI responses to different user tiers or conditions.
Common use cases of Conditional routing:
- If this user is on the paid plan, route their request to a custom fine-tuned model
- If this user is an EU resident, call an EU hosted model- If this user is a beta tester, send their request to the preview model
- If the request is coming from testing environment with a llm-pass-through flag, route it to the cheapest model..and more!

3. Interoperability

Easily switch between different AI models by updating your Portkey configuration:

const portkeyConfig = {
  "provider": "anthropic",
  "api_key": "YOUR_ANTHROPIC_API_KEY",
  "override_params": {
    "model": "claude-3-5-sonnet-20240620"
  }
};

Pro Tip: When switching models, always test thoroughly to ensure consistent performance and output quality across different providers.

4. Managing Auth for Hyperscaler Integrations

Portkey's virtual key system simplifies authentication management for cloud providers like AWS, Azure, GCP and many more

• Store LLM API keys securely in Portkey's vault
• Use unique virtual identifiers for streamlined API key management
• Benefit from easier key rotation and multiple virtual keys per API key
• Impose restrictions based on cost, request volume, and user access

5. Reliability Features

Portkey offers various reliability features like fallbacks, caching, and conditional routing. We've already seen examples of caching and conditional routing. Here's an example of setting up a fallback strategy:

const portkeyConfig = {
  "strategy": {
    "mode": "fallback"
  },
  "targets": [
    { 
      "provider": "anthropic",
      "api_key": "YOUR_ANTHROPIC_API_KEY",
      "override_params": {
        "model": "claude-3-5-sonnet-20240620"
      }
    },
    {
      "provider": "openai",
      "api_key": "YOUR_OPENAI_API_KEY",
      "override_params": {
        "model": "gpt-4o"
      }
    }
  ]
};

Real-world scenario: In a critical application like a customer service chatbot, this fallback strategy ensures that the service remains operational even if one or two providers experience downtime.

6. Observability

Portkey provides a comprehensive observability suite. Once integrated, you can access detailed analytics and insights through the Portkey dashboard, including request costs, token usage, latency, and more. Boost your observability capabilities by sending custom metadata with your requests:

PORTKEY_METADATA = {    "_user": "userid123", "environment": "production"}

This metadata can be used for auditing or filtering logs, allowing for more granular insights into your AI application's performance and usage patterns.

Best Practice: Regularly review your observability metrics to identify opportunities for optimization, such as frequently asked questions that could be cached or potential issues with specific types of requests.

7. Calling Local Models

Portkey allows you to seamlessly integrate with local models, giving you more flexibility in your AI infrastructure:

import Portkey from 'portkey-ai'

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    provider: "PROVIDER_NAME", // This can be mistral-ai, openai, or anything else
    customHost: "http://MODEL_URL/v1/", // Your custom URL with version identifier
    authorization: "AUTH_KEY", // If you need to pass auth
})

Conclusion

By integrating Portkey with your Vercel AI apps, you've unlocked a powerful set of tools for managing, monitoring, and optimizing your AI implementations. From seamless model switching to advanced reliability features, comprehensive observability, and robust guardrails, Portkey empowers you to build more robust, efficient, and responsible AI applications.

Resources

• GitHub Repository: Clone this repo to get started with a sample Portkey-Vercel integration.
• Portkey with Vercel Native Integration
• Portkey Documentation for Vercel Integration
• Join our Discord Community for support, discussions, and updates.

Remember to explore the Portkey documentation for more advanced features and best practices. We're excited to see what you'll build with Portkey and Vercel!

Happy coding!