Promptfoo

Promptfoo is an open source library for evaluating LLM output quality. With the AI Gateway, you can:

Manage prompts with versioning and call them in Promptfoo
Run evals on 3,000+ LLMs including private/local models
Track costs and metrics for all eval runs
Avoid rate limits with load balancing and caching

1. Reference AI Gateway Prompts

Use prompts stored in the AI Gateway directly in Promptfoo:

Set PORTKEY_API_KEY environment variable
Use portkey:// prefix with your prompt ID:

prompts:
  - portkey://pp-test-prompt-669f48
providers:
  - openai:gpt-3.5-turbo-0613
tests:
  - vars:
      topic: ...

Variables from test cases are automatically plugged into the AI Gateway prompt.

Promptfoo doesn’t follow temperature, model, and other parameters set in the AI Gateway. Set them in the providers configuration.

2. Route to Any Provider

Set PORTKEY_API_KEY and configure providers with the portkey: prefix:

providers:
  id: portkey:gpt-4o
  config:
    portkeyProvider: openai

providers:
  id: portkey:claude-3-opus-20240229
  config:
    portkeyProvider: anthropic
    portkeyVirtualKey: "@anthropic-prod"

providers:
  id: portkey:gemini-1.5-flash-latest
  config:
    portkeyProvider: google
    portkeyVirtualKey: "@google-prod"

providers:
  id: portkey:llama3-8b-8192
  config:
    portkeyProvider: groq
    portkeyVirtualKey: "@groq-prod"

providers:
  id: portkey:llama3
  config:
    portkeyProvider: ollama
    portkeyCustomHost: YOUR_OLLAMA_NGROK_URL

Cloud Providers (Azure, Bedrock, Vertex)

providers:
  id: portkey:xxx
  config:
    portkeyVirtualKey: "@azure-openai-prod"

providers:
  id: portkey:anthropic.claude-3-sonnet-20240229-v1:0
  config:
    portkeyVirtualKey: "@bedrock-prod"

providers:
  id: portkey:gemini-1.5-flash-latest
  config:
    portkeyVirtualKey: "@vertex-prod"

3. Track Costs & Metrics

Add metadata to segment requests and track costs per team/project:

providers:
  id: portkey:claude-3-opus-20240229
  config:
    portkeyVirtualKey: "@anthropic-prod"
    portkeyMetadata:
      team: alpha9
      prompt: classification
    portkeyTraceId: run_1

Filter and group by these keys in Strata Cloud Manager.

4. Avoid Rate Limits with Caching

Create a config with load balancing and caching for high-volume evals:

{
  "cache": { "mode": "simple" },
  "strategy": { "mode": "loadbalance" },
  "targets": [
    { "provider": "@openai-key-1" },
    { "provider": "@openai-key-2" },
    { "provider": "@openai-key-3" }
  ]
}

Reference the config in your YAML:

providers:
  id: portkey:claude-3-opus-20240229
  config:
    portkeyConfig: pc-your-config-id

Next Steps

Model Catalog

Set up providers

Last modified on September 15, 2026

Integrations

LLM Integrations

Cloud Platforms

Guardrails

Plugins

Vector Databases

Agents

AI Apps

Libraries

Tracing Providers

MCP Clients

MCP Servers

1. Reference AI Gateway Prompts

2. Route to Any Provider

Cloud Providers (Azure, Bedrock, Vertex)

3. Track Costs & Metrics

4. Avoid Rate Limits with Caching

Next Steps

Model Catalog

​1. Reference AI Gateway Prompts

​2. Route to Any Provider

​Cloud Providers (Azure, Bedrock, Vertex)

​3. Track Costs & Metrics

​4. Avoid Rate Limits with Caching

​Next Steps

Model Catalog

1. Reference AI Gateway Prompts

2. Route to Any Provider

Cloud Providers (Azure, Bedrock, Vertex)

3. Track Costs & Metrics

4. Avoid Rate Limits with Caching

Next Steps