Config Object
The config
object is used to configure API interactions with various providers. It supports multiple modes such as single provider access, load balancing between providers, and fallback strategies.
Here are some sample config objects:
You can find more examples of schemas below.
Schema Details
Key Name | Description | Type | Required | Enum Values | Additional Info |
---|---|---|---|---|---|
| Operational strategy for the config or any individual target | object | Yes (if no | - | See Strategy Object Details |
| Name of the service provider | string | Yes (if no | "openai", "anthropic", "azure-openai", "anyscale", "cohere" | - |
| API key for the service provider | string | Yes (if | - | - |
| Virtual key identifier | string | Yes (if no | - | - |
| Caching configuration | object | No | - | See Cache Object Details |
| Retry configuration | object | No | - | See Retry Object Details |
| Weight for load balancing | number | No | - | Used in |
| Status codes triggering fallback | array of strings | No | - | Used in |
| List of target configurations | array | Yes (if | - | Each item follows the config schema |
| Request timeout configuration | number | No | - | - |
| Route to privately hosted model | string | No | - | Used in combination with |
| Forward sensitive headers directly | array of strings | No | - | - |
| Pass model name and other hyper parameters | object | No | "model", "temperature", "frequency_penalty", "logit_bias", "logprobs", "top_logprobs", "max_tokens", "n", "presence_penalty", "response_format", "seed", "stop", "top_p", etc. | Pass everything that's typically part of the payload |
Strategy Object Details
Key Name | Description | Type | Required | Enum Values | Additional Info |
---|---|---|---|---|---|
| strategy mode for the config | string | Yes | "loadbalance", "fallback" | |
| status codes to apply the strategy. This field is only used when strategy mode is "fallback" | array of numbers | No | Optional |
Cache Object Details
Key Name | Description | Type | Required | Enum Values | Additional Info |
---|---|---|---|---|---|
| Cache mode | string | Yes | "simple", "semantic" | - |
| Maximum age for cache entries | integer | No | - | Optional |
Retry Object Details
Key Name | Description | Type | Required | Enum Values | Additional Info |
---|---|---|---|---|---|
| Number of retry attempts | integer | Yes | - | - |
| Status codes to trigger retries | array of strings | No | - | Optional |
Notes
The strategy
mode
key determines the operational mode of the config. If strategymode
is not specified, a single provider mode is assumed, requiring eitherprovider
andapi_key
orvirtual_key
.In
loadbalance
andfallback
modes, thetargets
array specifies the configurations for each target.The
cache
andretry
objects provide additional configurations for caching and retry policies, respectively.
Examples
These examples demonstrate different use cases of the configuration object, helping users to understand how they can structure their own configurations based on the schema. Feel free to add more examples or adjust these as necessary to match your specific use cases and requirements.
JSON Schema
The following JSON schema is used to validate the config object
Last updated