Automatic Retries

LLM APIs often have inexplicable failures. With Portkey, you can rescue a substantial number of your requests with our in-built automatic retries feature.

• Automatic retries are triggered up to 5 times

• Retries can also be triggered only on specific error codes

• And each subsequent retry attempt follows exponential backoff strategy to prevent network overload

Enabling Retries

To enable retry, just add the retry param to your config object.

Retry with 5 attempts

{
    "retry": {
        "attempts": 5
    },
    "virtual_key": "virtual-key-xxx"
}

Retry only on specific error codes

By default, Portkey triggers retries on the following error codes: [429, 500, 502, 503, 504]

You can change this behaviour by setting the optional on_status_codes param in your retry config and manually inputting the error codes on which rety will be triggered.

{
  "retry": {
    "attempts": 3,
    "on_status_codes": [ 408, 429, 401 ]
  },
  "virtual_key": "virtual-key-xxx"
}

If the on_status_codes param is present, retries will be triggered only on the error codes specified in that Config and not on Portkey's default error codes for retries (i.e. [429, 500, 502, 503, 504])

Exponential backoff strategy

Here's how Portkey triggers retries following exponential backoff:

Attempt	Time out between requests
Initial Call	Immediately
Retry 1st attempt	1 second
Retry 2nd attempt	2 seconds
Retry 3rd attempt	4 seconds
Retry 4th attempt	8 seconds
Retry 5th attempt	16 seconds