Batches

The Prisma AIRS AI Gateway lets you run Bedrock batch jobs without any manual S3 wrangling—simply upload an OpenAI-format .jsonl file and the AI Gateway converts it on-the-fly to the Bedrock format. Supported batch endpoints:

Chat Completions (/v1/chat/completions)
Embeddings (/v1/embeddings)

This is the most efficient way to

Test your data with different foundation models
Perform A/B testing with different foundation models
Perform batch inference with different foundation models

The AI Gateway supports two modes on Bedrock:

Provider Batch API (cheapest, completion_window: 24h, 48h, etc.)
AI Gateway Batch API ⭐️ (fast, provider-agnostic, completion_window: immediate)

Before You Start

AI Gateway API key ($PORTKEY_API_KEY).
Bedrock credentials — either an AI Gateway Provider from Model Catalog or explicit AWS keys (aws_access_key_id, aws_secret_access_key, aws_region, optional aws_session_token).
S3 bucket with read/write access for inputs and outputs.
IAM roles (see Permissions & IAM below).
Optional: a AI Gateway File (input_file_id) — required only when you set completion_window:"immediate" (gateway-batch mode).

Using Bedrock Batch API through the AI Gateway

Create Batch Job

curl --location 'https://aigw.portkey.ai/v1/batches' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-aws-access-key-id: {YOUR_AWS_ACCESS_KEY_ID}' \
--header 'x-portkey-aws-secret-access-key: {YOUR_AWS_SECRET_ACCESS_KEY}' \
--header 'x-portkey-aws-region: {YOUR_AWS_REGION}' \
--header 'Content-Type: application/json' \
--data '{
    "model": "meta.llama3-1-8b-instruct-v1:0",
    "input_file_id": "s3%3A%2F%2Fgenerations-raw-west-2%2Fbatch_files%2Fllama2%2Fbatch_chat_completions_101_requests.jsonl",
    "role_arn": "arn:aws:iam::12312:role/BedrockBatchRole", // the role to use for creating the batch job
    "output_data_config": {            // output_data_config is optional, if you want to specify a different output location for the batch job, default is the same as the input file
        "s3OutputDataConfig": {
            "s3Uri": "s3://generations-raw/",
            "s3EncryptionKeyId": "arn:aws:kms:us-west-2:517194595696:key/89b483cb-130d-497b-aa37-7db177e7cd32" // this is optional, if you want to use a KMS key to encrypt the output data
        }
    },
    "job_name": "anthropi-requests" // optional
}'

import OpenAI from 'openai'; // We're using the v4 SDK

const openai = new OpenAI({
  apiKey: "PORTKEY_API_KEY", // defaults to process.env["OPENAI_API_KEY"],
  baseURL: "https://aigw.portkey.ai/v1",
  defaultHeaders: {
      "x-portkey-provider": "openai",
      // defaults to process.env["PORTKEY_API_KEY"]
      "x-portkey-aws-access-key-id": "YOUR_AWS_ACCESS_KEY_ID",
      "x-portkey-aws-secret-access-key": "YOUR_AWS_SECRET_ACCESS_KEY",
      "x-portkey-aws-region": "YOUR_AWS_REGION",
      "x-portkey-aws-s3-bucket": "YOUR_AWS_S3_BUCKET",
      "x-portkey-aws-s3-object-key": "YOUR_AWS_S3_OBJECT_KEY",
      "x-portkey-aws-bedrock-model": "YOUR_AWS_BEDROCK_MODEL",
  }
});

const startBatch = async () => {
  const startBatchResponse = await openai.batches.create({
    input_file_id: "file_id", // file id of the input file
    endpoint: "endpoint", // ex: /v1/chat/completions
    completion_window: "completion_window", // ex: 24h
    metadata: {}, // metadata for the batch
    role_arn: "arn:aws:iam::12312:role/BedrockBatchRole", // the role to use for creating the batch job
    model: "anthropic.claude-3-5-sonnet-20240620-v1:0", // the model to use for the batch
    output_data_config: {
      s3OutputDataConfig: {
        s3Uri: "s3://generations-raw/",
        s3EncryptionKeyId: "arn:aws:kms:us-west-2:517194595696:key/89b483cb-130d-497b-aa37-7db177e7cd32" // this is optional, if you want to use a KMS key to encrypt the output data
      }
    }, // output_data_config is optional, if you want to specify a different output location for the batch job, default is the same as the input file
    job_name: "anthropi-requests-test" // optional
  });

  console.log(startBatchResponse);
}

await startBatch();

from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "openai", "x-portkey-aws-access-key-id": "YOUR_AWS_ACCESS_KEY_ID", "x-portkey-aws-secret-access-key": "YOUR_AWS_SECRET_ACCESS_KEY", "x-portkey-aws-region": "YOUR_AWS_REGION", "x-portkey-aws-s3-bucket": "YOUR_AWS_S3_BUCKET", "x-portkey-aws-s3-object-key": "YOUR_AWS_S3_OBJECT_KEY", "x-portkey-aws-bedrock-model": "YOUR_AWS_BEDROCK_MODEL"}
)

start_batch_response = openai.batches.create(
  input_file_id="file_id", # file id of the input file
  endpoint="endpoint", # ex: /v1/chat/completions
  completion_window="completion_window", # ex: 24h
  metadata={}, # metadata for the batch
  role_arn="arn:aws:iam::12312:role/BedrockBatchRole", # the role to use for creating the batch job
  model="anthropic.claude-3-5-sonnet-20240620-v1:0", # the model to use for the batch
  output_data_config={
    "s3OutputDataConfig": {
      "s3Uri": "s3://generations-raw/",
      "s3EncryptionKeyId": "arn:aws:kms:us-west-2:517194595696:key/89b483cb-130d-497b-aa37-7db177e7cd32" // this is optional, if you want to use a KMS key to encrypt the output data
    }
  }, # output_data_config is optional, if you want to specify a different output location for the batch job, default is the same as the input file
  job_name="anthropi-requests-test" # optional
)
print(start_batch_response)

List Batch Jobs

curl --location 'https://aigw.portkey.ai/v1/batches' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @provider' \

import OpenAI from 'openai'; // We're using the v4 SDK

const openai = new OpenAI({
  apiKey: "PORTKEY_API_KEY", // defaults to process.env["OPENAI_API_KEY"],
  baseURL: "https://aigw.portkey.ai/v1",
  defaultHeaders: {
      "x-portkey-provider": "openai",
      // defaults to process.env["PORTKEY_API_KEY"]
      "x-portkey-aws-access-key-id": "YOUR_AWS_ACCESS_KEY_ID",
      "x-portkey-aws-secret-access-key": "YOUR_AWS_SECRET_ACCESS_KEY",
      "x-portkey-aws-region": "YOUR_AWS_REGION",
  }
});

const listBatches = async () => {
  const batches = await openai.batches.list();

  console.log(batches);
}

await listBatches();

from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "openai", "x-portkey-aws-access-key-id": "YOUR_AWS_ACCESS_KEY_ID", "x-portkey-aws-secret-access-key": "YOUR_AWS_SECRET_ACCESS_KEY", "x-portkey-aws-region": "YOUR_AWS_REGION"}
)

batches = openai.batches.list()

print(batches)

Get Batch Job Details

curl --location 'https://aigw.portkey.ai/v1/batches/<batch_id>' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @provider' \

import OpenAI from 'openai'; // We're using the v4 SDK

const openai = new OpenAI({
  apiKey: "PORTKEY_API_KEY", // defaults to process.env["OPENAI_API_KEY"],
  baseURL: "https://aigw.portkey.ai/v1",
  defaultHeaders: {
      // defaults to process.env["PORTKEY_API_KEY"]
      "x-portkey-provider": "@BEDROCK_PROVIDER",
  }
});

const getBatch = async () => {
  const batch = await openai.batches.retrieve(batch_id="batch_id");

  console.log(batch);
}

await getBatch();

from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "@BEDROCK_PROVIDER"}
)

batch = openai.batches.retrieve(batch_id="batch_id")

print(batch)

Get Batch Output

curl --location 'https://aigw.portkey.ai/v1/batches/<batch_id>/output' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @provider' \

List Batch Jobs

curl --location 'https://aigw.portkey.ai/v1/batches' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @provider' \

import OpenAI from 'openai'; // We're using the v4 SDK

const openai = new OpenAI({
  apiKey: "PORTKEY_API_KEY", // defaults to process.env["OPENAI_API_KEY"],
  baseURL: "https://aigw.portkey.ai/v1",
  defaultHeaders: {
      // defaults to process.env["PORTKEY_API_KEY"]
      "x-portkey-provider": "@BEDROCK_PROVIDER",
  }
});

const listBatchingJobs = async () => {
  const batching_jobs = await openai.batches.list();

  console.log(batching_jobs);
}

await listBatchingJobs();

from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "@BEDROCK_PROVIDER"}
)

batching_jobs = openai.batches.list()

print(batching_jobs)

Cancel Batch Job

curl --request POST --location 'https://aigw.portkey.ai/v1/batches/<batch_id>/cancel' \
--header 'Authorization: Bearer <portkey_api_key>' \
--header 'x-portkey-provider: @provider' \

import OpenAI from 'openai'; // We're using the v4 SDK

const openai = new OpenAI({
  apiKey: "PORTKEY_API_KEY", // defaults to process.env["OPENAI_API_KEY"],
  baseURL: "https://aigw.portkey.ai/v1",
  defaultHeaders: {
      // defaults to process.env["PORTKEY_API_KEY"]
      "x-portkey-provider": "@BEDROCK_PROVIDER",
  }
});

const cancelBatch = async () => {
  const cancel_batch_response = await openai.batches.cancel(batch_id="batch_id");

  console.log(cancel_batch_response);
}

await cancelBatch();

from openai import OpenAI

openai = OpenAI(
    api_key="PORTKEY_API_KEY",
    base_url="https://aigw.portkey.ai/v1",
    default_headers={"x-portkey-provider": "@BEDROCK_PROVIDER"}
)

cancel_batch_response = openai.batches.cancel(batch_id="batch_id")

print(cancel_batch_response)

Information about Permissions and IAM Roles

For Principal Role (Used in Provider)

These are the minimum permissions required to use the Bedrock Batch APIs. (Bedrock Docs](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference-permissions.html))

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "bedrock:ListFoundationModels",
              "bedrock:GetFoundationModel",
              "bedrock:ListInferenceProfiles",
              "bedrock:GetInferenceProfile",
              "bedrock:ListCustomModels",
              "bedrock:GetCustomModel",
              "bedrock:TagResource",
              "bedrock:UntagResource",
              "bedrock:ListTagsForResource",
              "bedrock:CreateModelInvocationJob",
              "bedrock:GetModelInvocationJob",
              "bedrock:ListModelInvocationJobs",
              "bedrock:StopModelInvocationJob"
          ],
          "Resource": [
              "arn:aws:bedrock:<region>:<account_id>:model-invocation-job/*",
              "arn:aws:bedrock:<region>:<account_id>:custom-model/*", // Can be a custom model available for batching (optional)
              "arn:aws:bedrock:<region>::foundation-model/*"
          ]
      },
      {
          "Effect": "Allow",
          "Action": [
              "s3:ListBucket",
              "s3:PutObject",
              "s3:GetObject",
              "s3:GetObjectAttributes"
          ],
          "Resource": [
              "arn:aws:s3:::<bucket>",
              "arn:aws:s3:::<bucket>/*"
          ]
      },
      {
          "Action": [
              "iam:PassRole"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:iam::<account_id>:role/<service_role_name>",
          "Condition": {
              "StringEquals": {
                  "iam:PassedToService": [
                      "bedrock.amazonaws.com"
                  ]
              }
          }
      }
  ]
}

For Service Role (role_arn) used for creating and executing the batch job

These are the minimum permissions required to use the Bedrock Batch APIs.Trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Principal": {
              "Service": "bedrock.amazonaws.com"
          },
          "Action": "sts:AssumeRole",
          "Condition": {
              "StringEquals": {
                  "aws:SourceAccount": "<account_id>"
              },
              "ArnEquals": {
                  "aws:SourceArn": "arn:aws:bedrock:<region>:<account_id>:model-invocation-job/*"
              }
          }
  ]
}

Permission Policy:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "s3:GetObject",
              "s3:PutObject",
              "s3:ListBucket"
          ],
          "Resource": [
              "arn:aws:s3:::<bucket>",
              "arn:aws:s3:::<bucket>/*"
          ]
      }
  ]
}

Defaults & Limits (Bedrock Native)

Property	Default	Notes
`completion_window`	`24h`	Fixed by Bedrock.
Job quota	50k/day	Subject to your AWS account.
Max input file size	10 GB	Across all `.jsonl` objects.

Portkey-Batch mode inherits the Gateway defaults (25 req / 5 s) and retries (3×) per request.

Glossary

Term	Meaning
Batch Job	Collection of asynchronous completions.
AI Gateway File (`input_file_id`)	File uploaded to the AI Gateway and re-uploaded to the provider for batch processing.
Provider Slug	Bedrock credential stored in the AI Gateway via Model Catalog; referenced by slug.
Completion Window	`immediate` → gateway batch; `24h` → Bedrock native.

Integrations

LLM Integrations

Cloud Platforms

Guardrails

Plugins

Vector Databases

Agents

AI Apps

Libraries

Tracing Providers

MCP Clients

MCP Servers

Before You Start

Using Bedrock Batch API through the AI Gateway

Create Batch Job

List Batch Jobs

Get Batch Job Details

Get Batch Output

List Batch Jobs

Cancel Batch Job

Information about Permissions and IAM Roles

Defaults & Limits (Bedrock Native)

Glossary

See Also

​Before You Start

​Using Bedrock Batch API through the AI Gateway

​Create Batch Job

​List Batch Jobs

​Get Batch Job Details

​Get Batch Output

​List Batch Jobs

​Cancel Batch Job

​Information about Permissions and IAM Roles

​Defaults & Limits (Bedrock Native)

​Glossary

​See Also

Before You Start

Using Bedrock Batch API through the AI Gateway

Create Batch Job

List Batch Jobs

Get Batch Job Details

Get Batch Output

List Batch Jobs

Cancel Batch Job

Information about Permissions and IAM Roles

Defaults & Limits (Bedrock Native)

Glossary

See Also