To perform batch inference with Bedrock, you need to upload files to S3. This process can be cumbersome and duplicative in nature because you need to transform your data into model specific formats.

With Portkey, you can upload the file in OpenAI format and portkey will handle transforming the file into the format required by Bedrock on the fly!

This is the most efficient way to

  • Test your data with different foundation models
  • Perform A/B testing with different foundation models
  • Perform batch inference with different foundation models

​
Start Batch Inference

curl --location 'https://api.portkey.ai/v1/batches' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-aws-access-key-id: {YOUR_AWS_ACCESS_KEY_ID}' \
--header 'x-portkey-aws-secret-access-key: {YOUR_AWS_SECRET_ACCESS_KEY}' \
--header 'x-portkey-aws-region: {YOUR_AWS_REGION}' \
--header 'Content-Type: application/json' \
--data '{
    "input_file_id": "s3://your-bucket-name/your-file-name.jsonl",
    "endpoint": "<endpoint>",
    "completion_window": "<completion_window>",
    "metadata": {},
    "model": "anthropic.claude-3-5-sonnet-20240620-v1:0",
    "outputDataConfig": {
        "s3OutputDataConfig": {
            "s3Uri": "s3://your-bucket-name"
        }
    },
    "roleArn": "<YOUR_AWS_BEDROCK_ROLE_ARN>",
    "jobName": "<YOUR_JOB_NAME>"
}'

​
List Batch Inferences

curl --location 'https://api.portkey.ai/v1/batches' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-virtual-key: <virtual_key>'

​
Get Batch Inference

curl --location 'https://api.portkey.ai/v1/batches/<batch_id>' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-virtual-key: <virtual_key>'

​
Cancel Batch Inference

curl --location --request POST 'https://api.portkey.ai/v1/batches/<batch_id>/cancel' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-virtual-key: <virtual_key>'

​
Get Batch Output

This is a Gateway only feature

curl --location 'https://api.portkey.ai/v1/batches/<batch_id>/output' \
--header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-virtual-key: <virtual_key>'

Was this page helpful?

Suggest editsRaise issue
FilesAWS SageMaker