> ## Documentation Index
> Fetch the complete documentation index at: https://docs.portkey.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Conditional Routing

<Info>
  This feature is available on all Portkey [plans](https://portkey.ai/pricing).
</Info>

Using Portkey Gateway, you can route your requests to different provider targets based on custom conditions you define. These can be conditions like:

* If this user is on the `paid plan`, route their request to a `custom fine-tuned model`
* If the model parameter is set to `fastest`, route to `gpt-4o-mini`, if `smartest`, route to `openai o1`
* If this user is an `EU resident`, call an `EU hosted model`
* If the `temperature` parameter is above `0.7`, route to a more creative model
* If the request is coming from `testing environment` with a `llm-pass-through` flag, route it to the `cheapest model`
* ..and more!

Using this strategy, you can set up sophisticated routing logic based on:

1. **Metadata** - Custom key-value pairs you pass with your requests
2. **Request Parameters** - Any parameter that you send in your original LLM request (like model, temperature, max\_tokens)

All routing happens very fast on the *gateway*, on *edge*.

## Enabling Conditional Routing

Conditional routing is one of the *strategies* in Portkey's [Gateway Configs](/product/ai-gateway/configs). (others being `fallback` and `loadbalance`). To use it in your app,

1. You need to create a `conditional` config in Portkey UI.
2. Save the Config and get an associated Config ID.
3. And just pass the Config ID along with your requests using the `config` param.

## 1. Creating the `conditional` Config

Here's how a sample `conditional` config looks (along with its simpler, tree view).

<Tabs>
  <Tab title="Sample Config">
    ```json theme={"system"}
    {
      "strategy": {
        "mode": "conditional",
        "conditions": [
          ...conditions
        ],
        "default": "target_1"
      },
      "targets": [
        {
          "name": "target_1",
          "virtual_key":"xx"
        },
        {
          "name": "target_2",
          "virtual_key":"yy"
        }
      ]
    }
    ```
  </Tab>

  <Tab title="Tree View">
    ```
    │─ config
    │  │─ strategy mode:conditional
    │  │  │─ conditions
    │  │  │  │─ array of conditions
    │  │  │─ default
    │  │  │  │─ target name
    │  │─ targets
    │  │  │─ target 1
    │  │  │  │─ name
    │  │  │  │─ provider details
    │  │  │─ target 2
    │  │  │  │─ name
    │  │  │  │─ provider details
    ```
  </Tab>
</Tabs>

* `strategy.mode`: Set to `conditional`
* `strategy.conditions`: Query conditions with rules applied on metadata values or request parameters along with which target to call when the condition passes
* `strategy.default`: The default target name to call when none of the conditions pass
* `targets`: Array of target objects with unique `names` and provider details. These target names are referenced in the `conditions` objects above.

<Info>
  `conditions` and `default` are **required params** for the `conditional` strategy.
</Info>

### Structure of `conditions` Object

`conditions` are where you will actually write the routing rules. Here's a sample `condition` object:

<Tabs>
  <Tab title="Metadata Based">
    ```json theme={"system"}
      {
        "query": { "metadata.user_plan": { "$eq": "paid" } },
        "then": "finetuned-gpt4"
      }
    ```
  </Tab>

  <Tab title="Params Based">
    ```json theme={"system"}
    {
      "query": { "params.model": { "$eq": "smartest" } },
      "then": "smartest-model-target"
    }
    ```
  </Tab>
</Tabs>

`query`: Write the exact rule for checking metadata values or request parameters

`then`: Define which target to call if the query `PASSES`

### List of Condition Query Operators

| Operator | Description              |
| -------- | ------------------------ |
| `$eq`    | Equals                   |
| `$ne`    | Not equals               |
| `$in`    | In array                 |
| `$nin`   | Not in array             |
| `$regex` | Match the regex          |
| `$gt`    | Greater than             |
| `$gte`   | Greater than or equal to |
| `$lt`    | Less than                |
| `$lte`   | Less than or equal to    |

### Logical Query Operators

* `$and`: All conditions must be true
* `$or`: At least one condition must be true

\*\* Example Condition objects with Logical Operators\*\*

<Tabs>
  <Tab title="AND">
    ```json theme={"system"}
    {
      "$and": [
        { "metadata.user_type": { "$eq": "pro" } },
        { "metadata.model": { "$eq": "gpt-4" } }
      ]
    }
    ```
  </Tab>

  <Tab title="OR">
    ```json theme={"system"}
    {
      "$or": [
        { "params.temperature": { "$gt": 0.7 } },
        { "params.top_p": { "$lt": 0.8 } }
      ]
    }
    ```
  </Tab>

  <Tab title="MIXED Conditions">
    ```json theme={"system"}
    {
      "$or": [
        {
          "$and": [
            { "metadata.user_type": { "$eq": "pro" } },
            { "params.model": { "$eq": "gpt-4o" } }
          ]
        },
        { "params.max_tokens": { "$gt": 1000 } }
      ]
    }
    ```
  </Tab>
</Tabs>

<Info>
  1. You can write nested queries (with `$and`, `$or` operators)
  2. When a condition is incorrect or it fails, Portkey moves on to the next condition until it finds a successful condition.
  3. If no conditions pass, then the `default` target name is called
  4. Since Portkey iterates through the queries sequentially, the order of your conditions is important
  5. When using parameter-based routing, the user must send the exact parameters expected in the conditions, or the request will give an error
</Info>

## 2. Getting Config ID

Based on the `conditions` and the Config structure described above, you can create your [Config in Portkey UI](https://app.portkey.ai/configs), and save it to get Config ID. The UI also helps you autofill and autoformat your Config.

## 3. Implementing Conditional Routing

Conditional routing is enabled through Portkey's [Gateway Configs](/product/ai-gateway/configs) by following these steps:

1. Create a conditional config in Portkey UI ([app.portkey.ai/configs](https://app.portkey.ai/configs))
2. Save the config to get a Config ID
3. Use this Config ID in your requests

<CardGroup cols={2}>
  <Card title="Parameter-Based Routing" href="#routing-based-on-request-parameters" icon="sliders">
    Route requests based on the values of parameters like `model`, `temperature`, or `max_tokens`
  </Card>

  <Card title="Metadata-Based Routing" href="#routing-based-on-metadata" icon="tags">
    Route requests based on custom metadata fields you include with your request
  </Card>
</CardGroup>

You can also combine both approaches for [more sophisticated routing logic](#combined-routing-with-multiple-conditions)

### Routing Based on Request Parameters

With conditional routing, you can route based on any parameter in your LLM request (`model`, `temperature`, `max_tokens`, etc.).

<Note>
  Portkey only supports params routing for primitive types (string, number, boolean). Complex types like arrays and objects are not supported.
</Note>

**Example: Model Parameter Based Routing**

This example routes based on the `model` parameter, allowing you to use aliases instead of specific model names:

```json theme={"system"}
{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "params.model": {
            "$eq": "fastest"
          }
        },
        "then": "fastest-model-target"
      },
      {
        "query": {
          "params.model": {
            "$eq": "smartest"
          }
        },
        "then": "smartest-model-target"
      }
    ],
    "default": "fastest-model-target"
  },
  "targets": [
    {
      "name": "smartest-model-target",
      "virtual_key": "anthropic-vk",
      "override_params": {
        "model": "claude-3.5-sonnet"
      }
    },
    {
      "name": "fastest-model-target",
      "virtual_key": "oai-vk",
      "override_params": {
        "model": "gpt-4o-mini"
      }
    }
  ]
}
```

Using the config:

```python theme={"system"}
from portkey_ai import Portkey
client = Portkey(
    api_key="PORTKEY_API_KEY",
    config="model-routing-config-id"
)

# This will use the "smartest" model (claude-3.5-sonnet)
completion = client.chat.completions.create(
  model="smartest",  # This value matches our conditional routing condition
  messages=[
    {"role": "developer", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing to a 5-year-old"}
  ]
)

# This will use the "fastest" model (gpt-4o-mini)
completion = client.chat.completions.create(
  model="fastest",  # This value matches our conditional routing condition
  messages=[
    {"role": "developer", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's 2+2?"}
  ]
)
```

<Info>
  When using parameter-based routing, make sure to include the parameter specified in your routing condition in your request. In the example above, you need to include `model` in your request for routing to work properly.
</Info>

### Routing Based on Metadata

Portkey support [metadata](/product/observability/metadata)-based routing, which requires sending custom metadata with your request.

**Applying Conditional Routing Based on Metadata**

<Tabs>
  <Tab title="Simple Conditional Config">
    In this example we are routing our request based on `user_plan` metadata sent along request. If the user is on a `paid` plan, we route to a `finetuned-gpt4` model, otherwise we route to a `base-gpt4` model.

    ```json theme={"system"}
    {
      "strategy": {
        "mode": "conditional",
        "conditions": [
          {
            "query": { "metadata.user_plan": { "$eq": "paid" } },
            "then": "finetuned-gpt4"
          },
          {
            "query": { "metadata.user_plan": { "$eq": "free" } },
            "then": "base-gpt4"
          }
        ],
        "default": "base-gpt4"
      },
      "targets": [
        {
          "name": "finetuned-gpt4",
          "virtual_key":"xx",
          "override_params": {
              "model": "ft://gpt4-xxxxx"
          }
        },
        {
          "name": "base-gpt4",
          "virtual_key":"yy",
          "override_params": {
              "model": "gpt-4"
          }
        }
    }
    ```
  </Tab>

  <Tab title="Nested Conditional Config">
    ```json theme={"system"}
    {
      "strategy": {
        "mode": "conditional",
        "conditions": [
          {
            "query": {
              "$and": [
                { "metadata.user_type": { "$eq": "pro" } },
                { "metadata.user_tier": { "$eq": "tier-1" } }
              ]
            },
            "then": "gpt4_v2_target"
          },
          {
            "query": {
              "$or": [
                { "metadata.client": { "$eq": "UI" } },
                { "metadata.app_name": { "$regex": "my_app" } }
              ]
            },
            "then": "app_target"
          }
        ],
        "default": "default_target"
      },
      "targets": [
        { "name": "gpt4_v2_target", "virtual_key": "openai-xx"},
        { "name": "app_target", "virtual_key": "openai-yy" },
        { "name": "default_target", "virtual_key": "openai-zz" }
      ]
    }
    ```
  </Tab>
</Tabs>

Using the config:

```python theme={"system"}
from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    config="my-conditional-router-config"
)

response = portkey.with_options(
    metadata = {
        "user_plan": "free",
        "environment": "production",
        "session_id": "1729"
}).chat.completions.create(
    messages = [{ "role": 'user', "content": 'What is 1729' }]
)
```

<Info>
  You can combine both metadata and request parameters in your conditions. For example, you can route based on a combination of `metadata.user_plan` and `params.model`.
</Info>

### Combined Routing with Multiple Conditions

You can combine metadata and parameter conditions for more sophisticated routing:

```python theme={"system"}
from portkey_ai import Portkey

client = Portkey(
    api_key="PORTKEY_API_KEY",
    config="combined-routing-config-id"
)

# Route based on both metadata and temperature parameter
completion = client.with_options(
    metadata={"user_tier": "enterprise"}
).chat.completions.create(
    temperature=0.9,  # High temperature will route to creative model
    messages=[{"role": "user", "content": "Write a story about dragons"}]
)
```

Config example for combined conditions:

```json theme={"system"}
{
  "strategy": {
    "mode": "conditional",
    "conditions": [
      {
        "query": {
          "$and": [
            { "metadata.user_tier": { "$eq": "enterprise" } },
            { "params.temperature": { "$gte": 0.7 } }
          ]
        },
        "then": "creative-premium-target"
      }
    ],
    "default": "standard-target"
  },
  "targets": [
    {
      "name": "creative-premium-target",
      "virtual_key": "anthropic-vk",
      "override_params": { "model": "claude-3-opus" }
    },
    {
      "name": "standard-target",
      "virtual_key": "openai-vk",
      "override_params": { "model": "gpt-3.5-turbo" }
    }
  ]
}
```

## More Examples Using Conditional Routing

Here are practical examples of how you can leverage conditional routing to handle real-world challenges in your LLM applications. These examples showcase common patterns that help optimize cost, performance, compliance, and feature deployment without changing your application code.

<AccordionGroup>
  <Accordion title="User-Based Routing">
    <Tabs>
      <Tab title="User Subscription Tier">
        Route premium users to advanced models and free users to cost-effective ones.

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": { "metadata.user_tier": { "$eq": "premium" } },
                "then": "premium-model"
              },
              {
                "query": { "metadata.user_tier": { "$eq": "free" } },
                "then": "basic-model"
              }
            ],
            "default": "basic-model"
          },
          "targets": [
            {
              "name": "premium-model", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-4o" }
            },
            {
              "name": "basic-model", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-3.5-turbo" }
            }
          ]
        }
        ```
      </Tab>

      <Tab title="Regional Compliance">
        Ensure GDPR compliance by routing EU users to EU-compliant deployments.

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": { "metadata.user_region": { "$eq": "EU" } },
                "then": "eu-compliant"
              }
            ],
            "default": "standard-model"
          },
          "targets": [
            {
              "name": "eu-compliant", "virtual_key": "azure-eu-vk",
              "override_params": { "model": "gpt-4" }
            },
            {
              "name": "standard-model", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-4o" }
            }
          ]
        }
        ```
      </Tab>

      <Tab title="Data Sensitivity">
        Route requests to different models based on data sensitivity levels.

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": { "metadata.data_sensitivity": { "$eq": "high" } },
                "then": "on-premises-model"
              },
              {
                "query": { "metadata.data_sensitivity": { "$in": ["medium", "low"] } },
                "then": "cloud-model"
              }
            ],
            "default": "public-model"
          },
          "targets": [
            {
              "name": "public-model", "virtual_key": "openai-vk" },
            {
              "name": "on-premises-model", "virtual_key": "private-vk" },
            {
              "name": "cloud-model", "virtual_key": "azure-vk" }
          ]
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="Model Selection & Parameter Routing">
    <Tabs>
      <Tab title="Routing Based on Model Name Aliases">
        Use intuitive aliases instead of remembering specific model names.

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": { "params.model": { "$eq": "fastest" } },
                "then": "fast-model"
              },
              {
                "query": { "params.model": { "$eq": "smartest" } },
                "then": "smart-model"
              }
            ],
            "default": "balanced-model"
          },
          "targets": [
            {
              "name": "smart-model", "virtual_key": "anthropic-vk",
              "override_params": { "model": "claude-3.5-sonnet" }
            },
            {
              "name": "fast-model", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-4o-mini" }
            },
            {
              "name": "balanced-model", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-4o" }
            }
          ]
        }
        ```
      </Tab>

      <Tab title="Model Hyperparam Based Routing">
        Route creative vs factual requests to different optimized models.

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": { "params.temperature": { "$gte": 0.7 } },
                "then": "creative-model"
              },
              {
                "query": { "params.temperature": { "$lt": 0.3 } },
                "then": "factual-model"
              }
            ],
            "default": "balanced-model"
          },
          "targets": [
            {
              "name": "creative-model", "virtual_key": "anthropic-vk",
              "override_params": { "model": "claude-3-sonnet" }
            },
            {
              "name": "factual-model", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-4o" }
            },
            {
              "name": "balanced-model", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-3.5-turbo" }
            }
          ]
        }
        ```
      </Tab>

      <Tab title="Max Tokens Based Routing">
        Route high-token requests to models with larger context windows.

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": { "params.max_tokens": { "$gt": 4000 } },
                "then": "high-capacity"
              }
            ],
            "default": "standard-capacity"
          },
          "targets": [
            {
              "name": "high-capacity", "virtual_key": "anthropic-vk",
              "override_params": { "model": "claude-3-opus" }
            },
            {
              "name": "standard-capacity", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-4o-mini" }
            }
          ]
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="Application Features & Testing">
    <Tabs>
      <Tab title="Task-Specific Routing">
        Route different tasks to specialized models for optimal results.

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": { "metadata.task": { "$eq": "coding" } },
                "then": "coding-model"
              },
              {
                "query": { "metadata.task": { "$eq": "writing" } },
                "then": "writing-model"
              }
            ],
            "default": "general-model"
          },
          "targets": [
            {
              "name": "coding-model", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-4o" }
            },
            {
              "name": "writing-model", "virtual_key": "anthropic-vk",
              "override_params": { "model": "claude-3-opus" }
            },
            {
              "name": "general-model", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-3.5-turbo" }
            }
          ]
        }
        ```
      </Tab>

      <Tab title="A/B Testing">
        Test new models with specific user groups before full rollout.

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": { "metadata.user_group": { "$eq": "beta" } },
                "then": "new-model"
              }
            ],
            "default": "current-model"
          },
          "targets": [
            {
              "name": "new-model", "virtual_key": "anthropic-vk",
              "override_params": { "model": "claude-3.5-sonnet" }
            },
            {
              "name": "current-model", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-3.5-turbo" }
            }
          ]
        }
        ```
      </Tab>

      <Tab title="Feature Flags">
        Control feature rollout with feature flags in metadata.

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": { "metadata.features.new_model_enabled": { "$eq": true } },
                "then": "new-model-target"
              }
            ],
            "default": "standard-model"
          },
          "targets": [
            {
              "name": "new-model-target", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-4o" }
            },
            {
              "name": "standard-model", "virtual_key": "openai-vk",
              "override_params": { "model": "gpt-3.5-turbo" }
            }
          ]
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="Using Conditional Router with Guardrails">
    <Tabs>
      <Tab title="Input Guardrails">
        You can apply input guardrails to specific targets in your conditional routing configuration. This allows you to validate or transform the input before it's sent to the LLM:

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": {
                  "metadata.user_plan": { "$eq": "paid" }
                },
                "then": "premium-model"
              },
              {
                "query": {
                  "metadata.user_plan": { "$eq": "free" }
                },
                "then": "standard-model"
              }
            ],
            "default": "standard-model"
          },
          "targets": [
            {
              "name": "premium-model",
              "virtual_key": "openai-vk-xxx",
              "input_guardrails": ["pg-pii-detector-xxx", "pg-toxic-content-xxx"]
            },
            {
              "name": "standard-model",
              "virtual_key": "openai-vk-yyy"
            }
          ]
        }
        ```

        In this example, when a user with the "paid" plan sends a request, it's routed to the premium model but first runs through two input guardrails: one for PII detection and another for toxic content filtering.
      </Tab>

      <Tab title="Output Guardrails">
        You can also apply output guardrails to validate or transform the LLM's response:

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": {
                  "params.temperature": { "$gte": 0.7 }
                },
                "then": "creative-model"
              }
            ],
            "default": "standard-model"
          },
          "targets": [
            {
              "name": "creative-model",
              "virtual_key": "anthropic-vk-xxx",
              "output_guardrails": [
                "pg-fact-checker-xxx",
                "pg-content-moderator-xxx"
              ]
            },
            {
              "name": "standard-model",
              "virtual_key": "openai-vk-xxx"
            }
          ]
        }
        ```

        This configuration applies output guardrails to the "creative-model" target, which is used when the temperature parameter is high. The response from this model will be checked for factual accuracy and moderated for inappropriate content.
      </Tab>

      <Tab title="Both Input and Output Guardrails">
        For comprehensive protection, you can apply both input and output guardrails to your targets:

        ```json theme={"system"}
        {
          "strategy": {
            "mode": "conditional",
            "conditions": [
              {
                "query": {
                  "metadata.user_plan": { "$eq": "paid" }
                },
                "then": "premium-model"
              },
              {
                "query": {
                  "metadata.user_plan": { "$eq": "free" }
                },
                "then": "standard-model"
              }
            ],
            "default": "standard-model"
          },
          "targets": [
            {
              "name": "premium-model",
              "virtual_key": "openai-vk-xxx",
              "input_guardrails": ["pg-pii-detector-xxx", "pg-toxic-content-xxx"],
              "output_guardrails": ["pg-fact-checker-xxx", "pg-compliance-xxx"]
            },
            {
              "name": "standard-model",
              "virtual_key": "openai-vk-yyy",
              "input_guardrails": ["pg-basic-filter-xxx"],
              "output_guardrails": ["pg-basic-moderator-xxx"]
            }
          ]
        }
        ```

        This configuration provides end-to-end protection by:

        1. First checking inputs with guardrails before sending to the LLM
        2. Then validating outputs with guardrails before returning to the user

        Each target can have its own set of guardrails, allowing you to apply different levels of protection based on your routing conditions.
      </Tab>
    </Tabs>

    <Info>
      Guardrails are referenced by their IDs in the configuration. You can create guardrails in the Portkey UI and then reference them in your conditional routing config.

      Learn more about [Portkey Guardrails](/product/guardrails) to understand how to create and manage different types of guardrails for your LLM applications.
    </Info>
  </Accordion>
</AccordionGroup>

## Important Note

1. **Parameter Presence**: When routing based on request parameters, make sure the parameters specified in your conditions are included in your requests.

2. **Primitive Types Only**: Parameter-based routing works with primitive data types (strings, numbers, booleans) but not with arrays or nested objects.

3. **Order Matters**: Conditions are evaluated in the order they're defined, so place more specific conditions before more general ones.

4. **Error Handling**: If your condition references a parameter that doesn't exist in the request, the request will fail with an error.

5. **Multiple Parameter Types**: Portkey supports routing based on any parameter that can be sent in LLM requests including `model`, `temperature`, `top_p`, `frequency_penalty`, `presence_penalty`, `max_tokens`, and many others.

<Info>
  [Join us on Discord](https://portkey.wiki/chat) to share your thoughts on this feature or to get help with setting up advanced routing scenarios.
</Info>