Automatic Retries
LLM APIs often have inexplicable failures. With Portkey, you can rescue a substantial number of your requests with our in-built automatic retries feature.
This feature is available on all Portkey plans.
- Automatic retries are triggered up to 5 times
- Retries can also be triggered only on specific error codes
- And each subsequent retry attempt follows exponential backoff strategy to prevent network overload
Enabling Retries
To enable retry, just add the retry
param to your config object.
Retry with 5 attempts
Retry only on specific error codes
By default, Portkey triggers retries on the following error codes: [429, 500, 502, 503, 504]
You can change this behaviour by setting the optional on_status_codes
param in your retry config and manually inputting the error codes on which rety will be triggered.
If the on_status_codes
param is present, retries will be triggered only on the error codes specified in that Config and not on Portkey’s default error codes for retries (i.e. [429, 500, 502, 503, 504])
Exponential backoff strategy
Here’s how Portkey triggers retries following exponential backoff:
Attempt | Time out between requests |
---|---|
Initial Call | Immediately |
Retry 1st attempt | 1 second |
Retry 2nd attempt | 2 seconds |
Retry 3rd attempt | 4 seconds |
Retry 4th attempt | 8 seconds |
Retry 5th attempt | 16 seconds |
Was this page helpful?