1. API calls to LLMs with Portkey
Consider a typical API call to GPT4 to get chat completions using OpenAI SDK. It takesmessages
and model
arguments to get us a response. If you have tried one before, the following code snippet should look familiar. That’s because Portkey Client SDK follows the same signature as OpenAI’s.
portkey
variable — virtualKey
. Portkey securely stores API keys of LLM providers in a vault and substitutes them at runtime in your requests. These unique identifiers to your API keys are called Virtual Keys. For more information, see the docs.
With basics out of our way, let’s jump into applying what we set out to do in the first place with the AI gateway — To automatically retry our request when we hit rate-limits (429 status codes).
2. Apply Gateway Configs
The AI gateway requires instructions to automatically retry requests. This involves providing Gateway Configs, which are essentially JSON objects that orchestrate the AI gateway. In our current scenario, we are targeting GPT4 with requests that have automatic retries on 429 status codes.a. Reference Gateway Configs from the UI
Just as the title says — you create them on the UI and use an ID to have Portkey automatically apply in the request headers to instruct the AI gateway. UI builder features lint suggestions, makes it easy to reference (through config ID), eliminates manual management, and allows you to view version history. To create Gateway Configs,- Go to portkey.ai and
- Click on Configs
- Select Create
- Choose any name (such as request_retries)
ID
:
Try it out now!
The Configs saved will appear as a row item on the Configs page. The ID
is important as it is referenced in our calls through the AI gateway.
Portkey SDK
The Portkey SDK accepts the config parameter that takes the created config ID as it’s argument. To ensure all requests have automatic retries enabled on them, pass the config ID as argument whenportkey
is instantiated.
That’s right! One line of code, and all the request from your apps now inherit Gateway Configs and demonstrate automatic retries.
Let’s take a look at the code snippet:
Axios
In the cases, where you are not able to use an SDK, you can pass the same configs as headers with the keyx-portkey-config
.
OpenAI SDK
Portkey can be used with OpenAI SDK. To send a request with using OpenAI SDK client and apply gateway configs to the request pass abaseURL
and necessary headers as follows:
b. Reference Gateway Configs in the Code
Depending on the dynamics of your app, you might want to construct the Gateway Configs at the runtime. All you need to do is to pass the Gateway Configs directly to theconfig
parameter as an argument.
Portkey SDK
Axios
OpenAI SDK
config
argument as seperate objects right at the time of chat completions call instead of Portkey({..})
instantiation.