Conditional Routing

Using Portkey Gateway, you can route your requests to different provider targets based on custom conditions you define. These can be conditions like:

• If this user is on the paid plan, route their request to a custom fine-tuned model
• If this user is an EU resident, call an EU hosted model
• If this user is a beta tester, send their request to the preview model
• If the request is coming from testing environment with a llm-pass-through flag, route it to the cheapest model
• ..and more!

Using this strategy, you can set up various conditional checks on the metadata keys you’re passing with your requests and route requests to the appropriate target — all happening very fast on the gateway, on edge.

​

Enabling Conditional Routing

Conditional routing is one of the strategies in Portkey’s Gateway Configs. (others being fallback and loadbalance). To use it in your app,

1. You need to create a conditional config in Portkey UI.
2. Save the Config and get an associated Config ID.
3. And just pass the Config ID along with your requests using the config param.

​

1. Creating the conditional Config

Here’s how a sample conditional config looks (along with its simpler, tree view).

• Sample Config
• Tree View

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      ...conditions
    ],
    "default": "target_1"
  },
  "targets": [
    {
      "name": "target_1",
      "virtual_key":"xx"
    },
    {
      "name": "target_2",
      "virtual_key":"yy"
    }
  ]
}

• strategy.mode: Set to conditional
• strategy.conditions: Query conditions with rules applied on metadata values along with which target to call when the condition passes
• strategy.default: The default target name to call when none of the conditions pass
• targets: Array of target objects with unique names and provider details. These target names are referenced in the conditions objects above.

​

Structure of conditions Object

conditions are where you will actually write the routing rules. Here’s a sample condition object:

{
  "query": { "metadata.user_plan": { "$eq": "paid" } },
  "then": "finetuned-gpt4"
}

query: Write the exact rule for checking metadata values

then: Define which target to call if the query PASSES

​

List of Condition Query Operators

​

Logical Query Operators

• $and: All conditions must be true
• $or: At least one condition must be true

​

Example Condition objects with Logical Operators

• AND
• OR
• NESTING AND, OR

{
  "$and": [
    { "metadata.user_type": { "$eq": "pro" } },
    { "metadata.model": { "$eq": "gpt-4" } }
  ]
}

​

2. Getting Config ID

Based on the conditions and the Config structure described above, you can create your Config in Portkey UI, and save it to get Config ID. The UI also helps you autofill and autoformat your Config.

​

Adding the above sample condition to our final Config:

• Simple Conditional Config
• Nested Conditional Config

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.user_plan": { "$eq": "paid" } },
        "then": "finetuned-gpt4"
      },
      {
        "query": { "metadata.user_plan": { "$eq": "free" } },
        "then": "base-gpt4"
      }
    ],
    "default": "base-gpt4"
  },
  "targets": [
    {
      "name": "finetuned-gpt4",
      "virtual_key":"xx"
    },
    {
      "name": "base-gpt4",
      "virtual_key":"yy"
    }
  ]
}

​

3. Using the Config ID in Requests

Now, while instantiating your Portkey client or while sending headers, you just need to pass the Config ID and all your requests will start getting routed according to your conditions.

Conditional routing happens on Portkey’s on-the-edge stateless AI Gateway. We scan for the given query field in your request body, apply the query condition, and route to the specified target based on it.

Currently, we support Metadata based routing — i.e. routing your requests based on the metadata values you’re sending along with your request.

​

Applying Conditional Routing Based on Metadata

• Sample Full Request

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    config="my-conditional-router-config"
)

response = portkey.with_options(
    metadata = {
        "user_plan": "free",
        "environment": "production",
        "session_id": "1729"
}).chat.completions.create(
    messages = [{ "role": 'user', "content": 'What is 1729' }]
)

​

More Examples Using Conditional Routing

Here are some examples on how you can leverage conditional routing to handle real-world scenarios like:

• Data sensititvity or data residency requirements
• Calling a model based on user’s input lanuage
• Handling feature flags for your app
• Managing traffic better at peak usage times
• ..and many more

• Data Sensitivity
• Feature Flags
• Traffic Management
• Input Language

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "metadata.data_sensitivity": "high"
        },
        "then": "on-premises-model"
      },
      {
        "query": {
          "metadata.data_sensitivity": {
            "$in": ["medium", "low"]
          }
        },
        "then": "cloud-model"
      }
    ],
    "default": "public-model"
  },
  "targets": [
    {
      "name": "public-model",
      "virtual_key": "..."
    },
    {
      "name": "on-premises-model",
      "virtual_key": "..."
    },
    {
      "name": "cloud-model",
      "virtual_key": "..."
    }
  ]
}