Gateway Config Object

The config object is used to configure API interactions with various providers. It supports multiple modes such as single provider access, load balancing between providers, and fallback strategies.

The following JSON schema is used to validate the config object:

{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
  "strategy": {
    "type": "object",
    "properties": {
      "mode": {
        "type": "string",
        "enum": [
          "single",
          "loadbalance",
          "fallback"
        ]
      },
      "on_status_codes": {
        "type": "array",
        "items": {
          "type": "number"
        },
        "optional": true
      }
    }
  },
  "provider": {
    "type": "string",
    "enum": [
      "openai",
      "anthropic",
      "azure-openai",
      "anyscale",
      "cohere",
      "palm"
    ]
  },
  "resource_name": {
    "type": "string",
    "optional": true
  },
  "deployment_id": {
    "type": "string",
    "optional": true
  },
  "api_version": {
    "type": "string",
    "optional": true
  },
  "override_params": {
    "type": "object"
  },
  "api_key": {
    "type": "string"
  },
  "virtual_key": {
    "type": "string"
  },
  "cache": {
    "type": "object",
    "properties": {
      "mode": {
        "type": "string",
        "enum": [
          "simple",
          "semantic"
        ]
      },
      "max_age": {
        "type": "integer",
        "optional": true
      }
    },
    "required": [
      "mode"
    ]
  },
  "retry": {
    "type": "object",
    "properties": {
      "attempts": {
        "type": "integer"
      },
      "on_status_codes": {
        "type": "array",
        "items": {
          "type": "number"
        },
        "optional": true
      }
    },
    "required": [
      "attempts"
    ]
  },
  "weight": {
    "type": "number"
  },
  "on_status_codes": {
    "type": "array",
    "items": {
      "type": "integer"
    }
  },
  "targets": {
    "type": "array",
    "items": {
      "$ref": "#"
    }
  }
},
"anyOf": [
  {
    "required": [
      "provider",
      "api_key"
    ]
  },
  {
    "required": [
      "virtual_key"
    ]
  },
  {
    "required": [
      "strategy",
      "targets"
    ]
  },
  {
    "required": [
      "cache"
    ]
  },
  {
    "required": [
      "retry"
    ]
  }
],
"additionalProperties": false
}

Example Configs

// Simple config with cache and retry
{
  "virtual_key": "***", // Your Virtual Key
  "cache": { // Optional
    "mode": "semantic",
    "max_age": 10000
  },
  "retry": { // Optional
    "attempts": 5,
    "on_status_codes": []
  }
}

// Load balancing with 2 OpenAI keys
{
  "strategy": {
      "mode": "loadbalance"
    },
  "targets": [
    {
      "provider": "openai",
      "api_key": "sk-***"
    },
    {
      "provider": "openai",
      "api_key": "sk-***"
    }
  ]
}

You can find more examples of schemas below.

Schema Details

Key Name	Description	Type	Required	Enum Values	Additional Info
`strategy`	Operational strategy for the config or any individual target	object	Yes (if no `provider` or `virtual_key`)	-	See Strategy Object Details
`provider`	Name of the service provider	string	Yes (if no `mode` or `virtual_key`)	“openai”, “anthropic”, “azure-openai”, “anyscale”, “cohere”	-
`api_key`	API key for the service provider	string	Yes (if `provider` is specified)	-	-
`virtual_key`	Virtual key identifier	string	Yes (if no `mode` or `provider`)	-	-
`cache`	Caching configuration	object	No	-	See Cache Object Details
`retry`	Retry configuration	object	No	-	See Retry Object Details
`weight`	Weight for load balancing	number	No	-	Used in `loadbalance` mode
`on_status_codes`	Status codes triggering fallback	array of strings	No	-	Used in `fallback` mode
`targets`	List of target configurations	array	Yes (if `mode` is specified)	-	Each item follows the config schema
`request_timeout`	Request timeout configuration	number	No	-	-
`custom_host`	Route to privately hosted model	string	No	-	Used in combination with `provider` + `api_key`
`forward_headers`	Forward sensitive headers directly	array of strings	No	-	-
`override_params`	Pass model name and other hyper parameters	object	No	”model”, “temperature”, “frequency_penalty”, “logit_bias”, “logprobs”, “top_logprobs”, “max_tokens”, “n”, “presence_penalty”, “response_format”, “seed”, “stop”, “top_p”, etc.	Pass everything that’s typically part of the payload

Strategy Object Details

Key Name	Description	Type	Required	Enum Values	Additional Info
`mode`	strategy mode for the config	string	Yes	”loadbalance”, “fallback”
`on_status_codes`	status codes to apply the strategy. This field is only used when strategy mode is “fallback”	array of numbers	No		Optional

Cache Object Details

Key Name	Description	Type	Required	Enum Values	Additional Info
`mode`	Cache mode	string	Yes	”simple”, “semantic”	-
`max_age`	Maximum age for cache entries	integer	No	-	Optional

Retry Object Details

Key Name	Description	Type	Required	Enum Values	Additional Info
`attempts`	Number of retry attempts	integer	Yes	-	-
`on_status_codes`	Status codes to trigger retries	array of strings	No	-	Optional

Cloud Provider Params (Azure OpenAI, Google Vertex, AWS Bedrock)

Azure OpenAI

Key Name	Type	Required
`azure_resource_name`	string	No
`azure_deployment_id`	string	No
`azure_api_version`	string	No
`azure_model_name`	string	No
`Authorization`	string (“Bearer $API_KEY”)	No

Google Vertex AI

Key Name	Type	Required
`vertex_project_id`	string	No
`vertex_region`	string	No

AWS Bedrock

Key Name	Type	Required
`aws_access_key_id`	string	No
`aws_secret_access_key`	string	No
`aws_region`	string	No
`aws_session_token`	string	No

Notes

The strategy mode key determines the operational mode of the config. If strategy mode is not specified, a single provider mode is assumed, requiring either provider and api_key or virtual_key.
In loadbalance and fallback modes, the targets array specifies the configurations for each target.
The cache and retry objects provide additional configurations for caching and retry policies, respectively.

Examples

{
  "strategy": {
    "mode": "loadbalance"
  },
  "targets": [
    {
      "provider": "openai",
      "api_key": "sk-***"
    },
    {
      "provider": "openai",
      "api_key": "sk-***"
    }
  ]
}

  {
    "strategy": {
      "mode": "loadbalance"
    },
    "targets": [
      {
        "provider": "openai",
        "api_key": "sk-***"
      },
      {
        "strategy": {
          "mode": "fallback",
          "on_status_codes": [429, 241]
        },
        "targets": [
          {
            "virtual_key": "***"
          },
          {
            "virtual_key": "***"
          }
        ]
      }
    ]
  }

API Reference

Chat Completions

Portkey Prompts

Embeddings

Images

Audio

Other APIs

Completions

Moderations

Fine-tuning

Batch

Files

Assistants

Example Configs

Schema Details

Strategy Object Details

Cache Object Details

Retry Object Details

Cloud Provider Params (Azure OpenAI, Google Vertex, AWS Bedrock)

Azure OpenAI

Google Vertex AI

AWS Bedrock

Notes

Examples

API Reference

Chat Completions

Portkey Prompts

Embeddings

Images

Audio

Other APIs

Completions

Moderations

Fine-tuning

Batch

Files

Assistants

​Example Configs

​Schema Details

​Strategy Object Details

​Cache Object Details

​Retry Object Details

​Cloud Provider Params (Azure OpenAI, Google Vertex, AWS Bedrock)

​Azure OpenAI

​Google Vertex AI

​AWS Bedrock

​Notes

​Examples

Example Configs

Schema Details

Strategy Object Details

Cache Object Details

Retry Object Details

Cloud Provider Params (Azure OpenAI, Google Vertex, AWS Bedrock)

Azure OpenAI

Google Vertex AI

AWS Bedrock

Notes

Examples