Conditional Routing

This feature is available on all Portkey plans.

Using Portkey Gateway, you can route your requests to different provider targets based on custom conditions you define. These can be conditions like:

If this user is on the paid plan, route their request to a custom fine-tuned model
If the model parameter is set to fastest, route to gpt-4o-mini, if smartest, route to openai o1
If this user is an EU resident, call an EU hosted model
If the temperature parameter is above 0.7, route to a more creative model
If the request is coming from testing environment with a llm-pass-through flag, route it to the cheapest model
..and more!

Using this strategy, you can set up sophisticated routing logic based on:

Metadata - Custom key-value pairs you pass with your requests
Request Parameters - Any parameter that you send in your original LLM request (like model, temperature, max_tokens)

All routing happens very fast on the gateway, on edge.

Enabling Conditional Routing

Conditional routing is one of the strategies in Portkey’s Gateway Configs. (others being fallback and loadbalance). To use it in your app,

You need to create a conditional config in Portkey UI.
Save the Config and get an associated Config ID.
And just pass the Config ID along with your requests using the config param.

1. Creating the `conditional` Config

Here’s how a sample conditional config looks (along with its simpler, tree view).

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      ...conditions
    ],
    "default": "target_1"
  },
  "targets": [
    {
      "name": "target_1",
      "virtual_key":"xx"
    },
    {
      "name": "target_2",
      "virtual_key":"yy"
    }
  ]
}

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      ...conditions
    ],
    "default": "target_1"
  },
  "targets": [
    {
      "name": "target_1",
      "virtual_key":"xx"
    },
    {
      "name": "target_2",
      "virtual_key":"yy"
    }
  ]
}

│─ config
│  │─ strategy mode:conditional
│  │  │─ conditions
│  │  │  │─ array of conditions
│  │  │─ default
│  │  │  │─ target name
│  │─ targets
│  │  │─ target 1
│  │  │  │─ name
│  │  │  │─ provider details
│  │  │─ target 2
│  │  │  │─ name
│  │  │  │─ provider details

strategy.mode: Set to conditional
strategy.conditions: Query conditions with rules applied on metadata values or request parameters along with which target to call when the condition passes
strategy.default: The default target name to call when none of the conditions pass
targets: Array of target objects with unique names and provider details. These target names are referenced in the conditions objects above.

conditions and default are required params for the conditional strategy.

Structure of `conditions` Object

conditions are where you will actually write the routing rules. Here’s a sample condition object:

  {
    "query": { "metadata.user_plan": { "$eq": "paid" } },
    "then": "finetuned-gpt4"
  }

query: Write the exact rule for checking metadata values or request parameters

then: Define which target to call if the query PASSES

List of Condition Query Operators

Operator	Description
`$eq`	Equals
`$ne`	Not equals
`$in`	In array
`$nin`	Not in array
`$regex`	Match the regex
`$gt`	Greater than
`$gte`	Greater than or equal to
`$lt`	Less than
`$lte`	Less than or equal to

Logical Query Operators

$and: All conditions must be true
$or: At least one condition must be true

** Example Condition objects with Logical Operators**

{
  "$and": [
    { "metadata.user_type": { "$eq": "pro" } },
    { "metadata.model": { "$eq": "gpt-4" } }
  ]
}

You can write nested queries (with $and, $or operators)
When a condition is incorrect or it fails, Portkey moves on to the next condition until it finds a successful condition.
If no conditions pass, then the default target name is called
Since Portkey iterates through the queries sequentially, the order of your conditions is important
When using parameter-based routing, the user must send the exact parameters expected in the conditions, or the request will give an error

2. Getting Config ID

Based on the conditions and the Config structure described above, you can create your Config in Portkey UI, and save it to get Config ID. The UI also helps you autofill and autoformat your Config.

3. Implementing Conditional Routing

Conditional routing is enabled through Portkey’s Gateway Configs by following these steps:

Create a conditional config in Portkey UI (app.portkey.ai/configs)
Save the config to get a Config ID
Use this Config ID in your requests

Parameter-Based Routing

Route requests based on the values of parameters like model, temperature, or max_tokens

Metadata-Based Routing

Route requests based on custom metadata fields you include with your request

You can also combine both approaches for more sophisticated routing logic

Routing Based on Request Parameters

With conditional routing, you can route based on any parameter in your LLM request (model, temperature, max_tokens, etc.).

Portkey only supports params routing for primitive types (string, number, boolean). Complex types like arrays and objects are not supported.

Example: Model Parameter Based Routing

This example routes based on the model parameter, allowing you to use aliases instead of specific model names:

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "params.model": {
            "$eq": "fastest"
          }
        },
        "then": "fastest-model-target"
      },
      {
        "query": {
          "params.model": {
            "$eq": "smartest"
          }
        },
        "then": "smartest-model-target"
      }
    ],
    "default": "fastest-model-target"
  },
  "targets": [
    {
      "name": "smartest-model-target",
      "virtual_key": "anthropic-vk",
      "override_params": {
        "model": "claude-3.5-sonnet"
      }
    },
    {
      "name": "fastest-model-target",
      "virtual_key": "oai-vk",
      "override_params": {
        "model": "gpt-4o-mini"
      }
    }
  ]
}

Using the config:

from portkey_ai import Portkey
client = Portkey(
    api_key="PORTKEY_API_KEY",
    config="model-routing-config-id"
)

# This will use the "smartest" model (claude-3.5-sonnet)
completion = client.chat.completions.create(
  model="smartest",  # This value matches our conditional routing condition
  messages=[
    {"role": "developer", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing to a 5-year-old"}
  ]
)

# This will use the "fastest" model (gpt-4o-mini)
completion = client.chat.completions.create(
  model="fastest",  # This value matches our conditional routing condition
  messages=[
    {"role": "developer", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's 2+2?"}
  ]
)

When using parameter-based routing, make sure to include the parameter specified in your routing condition in your request. In the example above, you need to include model in your request for routing to work properly.

Routing Based on Metadata

Portkey support metadata-based routing, which requires sending custom metadata with your request.

Applying Conditional Routing Based on Metadata

In this example we are routing our request based on user_plan metadata sent along request. If the user is on a paid plan, we route to a finetuned-gpt4 model, otherwise we route to a base-gpt4 model.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.user_plan": { "$eq": "paid" } },
        "then": "finetuned-gpt4"
      },
      {
        "query": { "metadata.user_plan": { "$eq": "free" } },
        "then": "base-gpt4"
      }
    ],
    "default": "base-gpt4"
  },
  "targets": [
    {
      "name": "finetuned-gpt4",
      "virtual_key":"xx",
      "override_params": {
          "model": "ft://gpt4-xxxxx"
      }
    },
    {
      "name": "base-gpt4",
      "virtual_key":"yy",
      "override_params": {
          "model": "gpt-4"
      }
    }
}

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.user_plan": { "$eq": "paid" } },
        "then": "finetuned-gpt4"
      },
      {
        "query": { "metadata.user_plan": { "$eq": "free" } },
        "then": "base-gpt4"
      }
    ],
    "default": "base-gpt4"
  },
  "targets": [
    {
      "name": "finetuned-gpt4",
      "virtual_key":"xx",
      "override_params": {
          "model": "ft://gpt4-xxxxx"
      }
    },
    {
      "name": "base-gpt4",
      "virtual_key":"yy",
      "override_params": {
          "model": "gpt-4"
      }
    }
}

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "$and": [
            { "metadata.user_type": { "$eq": "pro" } },
            { "metadata.user_tier": { "$eq": "tier-1" } }
          ]
        },
        "then": "gpt4_v2_target"
      },
      {
        "query": {
          "$or": [
            { "metadata.client": { "$eq": "UI" } },
            { "metadata.app_name": { "$regex": "my_app" } }
          ]
        },
        "then": "app_target"
      }
    ],
    "default": "default_target"
  },
  "targets": [
    { "name": "gpt4_v2_target", "virtual_key": "openai-xx"},
    { "name": "app_target", "virtual_key": "openai-yy" },
    { "name": "default_target", "virtual_key": "openai-zz" }
  ]
}

Using the config:

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    config="my-conditional-router-config"
)

response = portkey.with_options(
    metadata = {
        "user_plan": "free",
        "environment": "production",
        "session_id": "1729"
}).chat.completions.create(
    messages = [{ "role": 'user', "content": 'What is 1729' }]
)

You can combine both metadata and request parameters in your conditions. For example, you can route based on a combination of metadata.user_plan and params.model.

Combined Routing with Multiple Conditions

You can combine metadata and parameter conditions for more sophisticated routing:

from portkey_ai import Portkey

client = Portkey(
    api_key="PORTKEY_API_KEY",
    config="combined-routing-config-id"
)

# Route based on both metadata and temperature parameter
completion = client.with_options(
    metadata={"user_tier": "enterprise"}
).chat.completions.create(
    temperature=0.9,  # High temperature will route to creative model
    messages=[{"role": "user", "content": "Write a story about dragons"}]
)

Config example for combined conditions:

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "$and": [
            { "metadata.user_tier": { "$eq": "enterprise" } },
            { "params.temperature": { "$gte": 0.7 } }
          ]
        },
        "then": "creative-premium-target"
      }
    ],
    "default": "standard-target"
  },
  "targets": [
    {
      "name": "creative-premium-target",
      "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3-opus" }
    },
    {
      "name": "standard-target",
      "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}

More Examples Using Conditional Routing

Here are practical examples of how you can leverage conditional routing to handle real-world challenges in your LLM applications. These examples showcase common patterns that help optimize cost, performance, compliance, and feature deployment without changing your application code.

Route premium users to advanced models and free users to cost-effective ones.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.user_tier": { "$eq": "premium" } },
        "then": "premium-model"
      },
      {
        "query": { "metadata.user_tier": { "$eq": "free" } },
        "then": "basic-model"
      }
    ],
    "default": "basic-model"
  },
  "targets": [
    {
      "name": "premium-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    },
    {
      "name": "basic-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}

Route premium users to advanced models and free users to cost-effective ones.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.user_tier": { "$eq": "premium" } },
        "then": "premium-model"
      },
      {
        "query": { "metadata.user_tier": { "$eq": "free" } },
        "then": "basic-model"
      }
    ],
    "default": "basic-model"
  },
  "targets": [
    {
      "name": "premium-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    },
    {
      "name": "basic-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}

Ensure GDPR compliance by routing EU users to EU-compliant deployments.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.user_region": { "$eq": "EU" } },
        "then": "eu-compliant"
      }
    ],
    "default": "standard-model"
  },
  "targets": [
    {
      "name": "eu-compliant", "virtual_key": "azure-eu-vk",
      "override_params": { "model": "gpt-4" }
    },
    {
      "name": "standard-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    }
  ]
}

Route requests to different models based on data sensitivity levels.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.data_sensitivity": { "$eq": "high" } },
        "then": "on-premises-model"
      },
      {
        "query": { "metadata.data_sensitivity": { "$in": ["medium", "low"] } },
        "then": "cloud-model"
      }
    ],
    "default": "public-model"
  },
  "targets": [
    {
      "name": "public-model", "virtual_key": "openai-vk" },
    {
      "name": "on-premises-model", "virtual_key": "private-vk" },
    {
      "name": "cloud-model", "virtual_key": "azure-vk" }
  ]
}

Use intuitive aliases instead of remembering specific model names.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "params.model": { "$eq": "fastest" } },
        "then": "fast-model"
      },
      {
        "query": { "params.model": { "$eq": "smartest" } },
        "then": "smart-model"
      }
    ],
    "default": "balanced-model"
  },
  "targets": [
    {
      "name": "smart-model", "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3.5-sonnet" }
    },
    {
      "name": "fast-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o-mini" }
    },
    {
      "name": "balanced-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    }
  ]
}

Use intuitive aliases instead of remembering specific model names.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "params.model": { "$eq": "fastest" } },
        "then": "fast-model"
      },
      {
        "query": { "params.model": { "$eq": "smartest" } },
        "then": "smart-model"
      }
    ],
    "default": "balanced-model"
  },
  "targets": [
    {
      "name": "smart-model", "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3.5-sonnet" }
    },
    {
      "name": "fast-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o-mini" }
    },
    {
      "name": "balanced-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    }
  ]
}

Route creative vs factual requests to different optimized models.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "params.temperature": { "$gte": 0.7 } },
        "then": "creative-model"
      },
      {
        "query": { "params.temperature": { "$lt": 0.3 } },
        "then": "factual-model"
      }
    ],
    "default": "balanced-model"
  },
  "targets": [
    {
      "name": "creative-model", "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3-sonnet" }
    },
    {
      "name": "factual-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    },
    {
      "name": "balanced-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}

Route high-token requests to models with larger context windows.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "params.max_tokens": { "$gt": 4000 } },
        "then": "high-capacity"
      }
    ],
    "default": "standard-capacity"
  },
  "targets": [
    {
      "name": "high-capacity", "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3-opus" }
    },
    {
      "name": "standard-capacity", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o-mini" }
    }
  ]
}

Route different tasks to specialized models for optimal results.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.task": { "$eq": "coding" } },
        "then": "coding-model"
      },
      {
        "query": { "metadata.task": { "$eq": "writing" } },
        "then": "writing-model"
      }
    ],
    "default": "general-model"
  },
  "targets": [
    {
      "name": "coding-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    },
    {
      "name": "writing-model", "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3-opus" }
    },
    {
      "name": "general-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}

Route different tasks to specialized models for optimal results.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.task": { "$eq": "coding" } },
        "then": "coding-model"
      },
      {
        "query": { "metadata.task": { "$eq": "writing" } },
        "then": "writing-model"
      }
    ],
    "default": "general-model"
  },
  "targets": [
    {
      "name": "coding-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    },
    {
      "name": "writing-model", "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3-opus" }
    },
    {
      "name": "general-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}

Test new models with specific user groups before full rollout.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.user_group": { "$eq": "beta" } },
        "then": "new-model"
      }
    ],
    "default": "current-model"
  },
  "targets": [
    {
      "name": "new-model", "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3.5-sonnet" }
    },
    {
      "name": "current-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}

Control feature rollout with feature flags in metadata.

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": { "metadata.features.new_model_enabled": { "$eq": true } },
        "then": "new-model-target"
      }
    ],
    "default": "standard-model"
  },
  "targets": [
    {
      "name": "new-model-target", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-4o" }
    },
    {
      "name": "standard-model", "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}

You can apply input guardrails to specific targets in your conditional routing configuration. This allows you to validate or transform the input before it’s sent to the LLM:

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "metadata.user_plan": { "$eq": "paid" }
        },
        "then": "premium-model"
      },
      {
        "query": {
          "metadata.user_plan": { "$eq": "free" }
        },
        "then": "standard-model"
      }
    ],
    "default": "standard-model"
  },
  "targets": [
    {
      "name": "premium-model",
      "virtual_key": "openai-vk-xxx",
      "input_guardrails": ["pg-pii-detector-xxx", "pg-toxic-content-xxx"]
    },
    {
      "name": "standard-model",
      "virtual_key": "openai-vk-yyy"
    }
  ]
}

In this example, when a user with the “paid” plan sends a request, it’s routed to the premium model but first runs through two input guardrails: one for PII detection and another for toxic content filtering.

You can apply input guardrails to specific targets in your conditional routing configuration. This allows you to validate or transform the input before it’s sent to the LLM:

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "metadata.user_plan": { "$eq": "paid" }
        },
        "then": "premium-model"
      },
      {
        "query": {
          "metadata.user_plan": { "$eq": "free" }
        },
        "then": "standard-model"
      }
    ],
    "default": "standard-model"
  },
  "targets": [
    {
      "name": "premium-model",
      "virtual_key": "openai-vk-xxx",
      "input_guardrails": ["pg-pii-detector-xxx", "pg-toxic-content-xxx"]
    },
    {
      "name": "standard-model",
      "virtual_key": "openai-vk-yyy"
    }
  ]
}

You can also apply output guardrails to validate or transform the LLM’s response:

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "params.temperature": { "$gte": 0.7 }
        },
        "then": "creative-model"
      }
    ],
    "default": "standard-model"
  },
  "targets": [
    {
      "name": "creative-model",
      "virtual_key": "anthropic-vk-xxx",
      "output_guardrails": [
        "pg-fact-checker-xxx",
        "pg-content-moderator-xxx"
      ]
    },
    {
      "name": "standard-model",
      "virtual_key": "openai-vk-xxx"
    }
  ]
}

This configuration applies output guardrails to the “creative-model” target, which is used when the temperature parameter is high. The response from this model will be checked for factual accuracy and moderated for inappropriate content.

For comprehensive protection, you can apply both input and output guardrails to your targets:

{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "metadata.user_plan": { "$eq": "paid" }
        },
        "then": "premium-model"
      },
      {
        "query": {
          "metadata.user_plan": { "$eq": "free" }
        },
        "then": "standard-model"
      }
    ],
    "default": "standard-model"
  },
  "targets": [
    {
      "name": "premium-model",
      "virtual_key": "openai-vk-xxx",
      "input_guardrails": ["pg-pii-detector-xxx", "pg-toxic-content-xxx"],
      "output_guardrails": ["pg-fact-checker-xxx", "pg-compliance-xxx"]
    },
    {
      "name": "standard-model",
      "virtual_key": "openai-vk-yyy",
      "input_guardrails": ["pg-basic-filter-xxx"],
      "output_guardrails": ["pg-basic-moderator-xxx"]
    }
  ]
}

This configuration provides end-to-end protection by:

First checking inputs with guardrails before sending to the LLM
Then validating outputs with guardrails before returning to the user

Each target can have its own set of guardrails, allowing you to apply different levels of protection based on your routing conditions.

Guardrails are referenced by their IDs in the configuration. You can create guardrails in the Portkey UI and then reference them in your conditional routing config.

Learn more about Portkey Guardrails to understand how to create and manage different types of guardrails for your LLM applications.

Important Note

Parameter Presence: When routing based on request parameters, make sure the parameters specified in your conditions are included in your requests.
Primitive Types Only: Parameter-based routing works with primitive data types (strings, numbers, booleans) but not with arrays or nested objects.
Order Matters: Conditions are evaluated in the order they’re defined, so place more specific conditions before more general ones.
Error Handling: If your condition references a parameter that doesn’t exist in the request, the request will fail with an error.
Multiple Parameter Types: Portkey supports routing based on any parameter that can be sent in LLM requests including model, temperature, top_p, frequency_penalty, presence_penalty, max_tokens, and many others.

Join us on Discord to share your thoughts on this feature or to get help with setting up advanced routing scenarios.

Introduction

Product

Support

Enabling Conditional Routing

1. Creating the `conditional` Config

Structure of `conditions` Object

List of Condition Query Operators

Logical Query Operators

2. Getting Config ID

3. Implementing Conditional Routing

Parameter-Based Routing

Metadata-Based Routing

Routing Based on Request Parameters

Routing Based on Metadata

Combined Routing with Multiple Conditions

More Examples Using Conditional Routing

Important Note

Introduction

Product

Support

​Enabling Conditional Routing

​1. Creating the conditional Config

​Structure of conditions Object

​List of Condition Query Operators

​Logical Query Operators

​2. Getting Config ID

​3. Implementing Conditional Routing

Parameter-Based Routing

Metadata-Based Routing

​Routing Based on Request Parameters

​Routing Based on Metadata

​Combined Routing with Multiple Conditions

​More Examples Using Conditional Routing

​Important Note

Enabling Conditional Routing

1. Creating the `conditional` Config

Structure of `conditions` Object

List of Condition Query Operators

Logical Query Operators

2. Getting Config ID

3. Implementing Conditional Routing

Routing Based on Request Parameters

Routing Based on Metadata

Combined Routing with Multiple Conditions

More Examples Using Conditional Routing

Important Note