With Portkey, you can perform OpenAI Batch Inference operations. This is the most efficient way to

  • Test your data with different foundation models
  • Perform A/B testing with different foundation models
  • Perform batch inference with different foundation models

​
Start Batch Inference

curl --location 'https://api.portkey.ai/v1/batches' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-virtual-key: <virtual_key>' \
--header 'Content-Type: application/json' \
--data '{
    "input_file_id": "<file_id>",
    "endpoint": "<endpoint>",
    "completion_window": "<completion_window>",
    "metadata": {},
}'

​
List Batch Inferences

curl --location 'https://api.portkey.ai/v1/batches' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-virtual-key: <virtual_key>'

​
Get Batch Inference

curl --location 'https://api.portkey.ai/v1/batches/<batch_id>' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-virtual-key: <virtual_key>'

​
Cancel Batch Inference

curl --location --request POST 'https://api.portkey.ai/v1/batches/<batch_id>/cancel' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-virtual-key: <virtual_key>'

​
Get Batch Output

This is a Gateway only feature

curl --location 'https://api.portkey.ai/v1/batches/<batch_id>/output' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-virtual-key: <virtual_key>'

Was this page helpful?

Suggest editsRaise issue
FilesAnthropic