Cache (Simple & Semantic)

Speed up and save money on your LLM requests by storing past responses in the Portkey cache. There are 2 cache modes:

• Simple: Matches requests verbatim. Perfect for repeated, identical prompts. Works on all models including image generation models.
• Semantic: Matches responses for requests that are semantically similar. Ideal for denoising requests with extra prepositions, pronouns, etc. Works on any model available on /chat/completionsor /completions routes.

Portkey cache serves requests upto 20x times faster and cheaper.

​

Enable Cache in the Config

To enable Portkey cache, just add the cache params to your config object.

​

Simple Cache

"cache": { "mode": "simple" }

​

How it Works

Simple cache performs an exact match on the input prompts. If the exact same request is received again, Portkey retrieves the response directly from the cache, bypassing the model execution.

​

Semantic Cache

"cache": { "mode": "semantic" }

​

How it Works

Semantic cache considers the contextual similarity between input requests. It uses cosine similarity to ascertain if the similarity between the input and a cached request exceeds a specific threshold. If the similarity threshold is met, Portkey retrieves the response from the cache, saving model execution time. Check out this blog for more details on how we do this.

​

Ignoring the First Message in Semantic Cache

When using the /chat/completions endpoint, Portkey requires at least two message objects in the messages array. The first message object, typically used for the system message, is not considered when determining semantic similarity for caching purposes.

For example:

messages = [
        { "role": "system", "content": "You are a helpful assistant" },
        { "role": "user", "content": "Who is the president of the US?" }
]

In this case, only the content of the user message (“Who is the president of the US?”) is used for finding semantic matches in the cache. The system message (“You are a helpful assistant”) is ignored.

This means that even if you change the system message while keeping the user message semantically similar, Portkey will still return a semantic cache hit.

This allows you to modify the behavior or context of the assistant without affecting the cache hits for similar user queries.

​

Read more how to set cache in Configs.

​

Setting Cache Age

You can set the age (or “ttl”) of your cached response with this setting. Cache age is also set in your Config object:

"cache": {
    "mode": "semantic",
    "max_age": 60
}

In this example, your cache will automatically expire after 60 seconds. Cache age is set in seconds.

​

Force Refresh Cache

Ensure that a new response is fetched and stored in the cache even when there is an existing cached response for your request. Cache force refresh can only be done at the time of making a request, and it is not a part of your Config.

You can enable cache force refresh with this header:

"x-portkey-cache-force-refresh": "True"

• NodeJS
• Python
• Node

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-virtual-key: open-ai-xxx" \
  -H "x-portkey-config: cache-config-xxx" \
  -H "x-portkey-cache-force-refresh: true" \
  -d '{
    "messages": [{"role": "user","content": "Hello!"}]
  }'

​

Cache Namespace: Simplified Cache Partitioning

Portkey generally partitions the cache along all the values passed in your request header. With a custom cache namespace, you can now ignore metadata and other headers, and only partition the cache based on the custom strings that you send.

This allows you to have finer control over your cached data and optimize your cache hit ratio.

​

How It Works

To use Cache Namespaces, simply include the x-portkey-cache-namespace header in your API requests, followed by any custom string value. Portkey will then use this namespace string as the sole basis for partitioning the cache, disregarding all other headers, including metadata.

For example, if you send the following header:

"x-portkey-cache-namespace: user-123"

Portkey will cache the response under the namespace user-123, ignoring any other headers or metadata associated with the request.

• NodeJS
• Python
• cURL

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    config: "pc-cache-xxx",
    virtualKey: "open-ai-xxx"
})

async function main(){
    const response = await portkey.chat.completions.create({
        messages: [{ role: 'user', content: 'Hello' }],
        model: 'gpt-4',
    }, {
        cacheNamespace: 'user-123'
    });
}

main()

In this example, the response will be cached under the namespace user-123, ignoring any other headers or metadata.

​

Cache in Analytics

Portkey shows you powerful stats on cache usage on the Analytics page. Just head over to the Cache tab, and you will see:

• Your raw number of cache hits as well as daily cache hit rate
• Your average latency for delivering results from cache and how much time it saves you
• How much money the cache saves you

​

Cache in Logs

On the Logs page, the cache status is updated on the Status column. You will see Cache Disabled when you are not using the cache, and any of Cache Miss, Cache Refreshed, Cache Hit, Cache Semantic Hit based on the cache hit status. Read more here.

For each request we also calculate and show the cache response time and how much money you saved with each hit.

​

How Cache works with Configs

You can set cache at two levels:

• Top-level that works across all the targets.
• Target-level that works when that specific target is triggered.

• Setting Top-Level Cache
• Setting Target-Level Cache

{
  "cache": {"mode": "semantic", "max_age": 60},
  "strategy": {"mode": "fallback"},
  "targets": [
    {"virtual_key": "openai-key-1"},
    {"virtual_key": "openai-key-2"}
  ]
}