Guardrails for Batches

Gateway >= 2.9.0 Data Service >= 1.8.0

Imagine usecases where you want to redact PII, remove some requests and redact responses when making a bulk batch inference call to providers like OpenAI, Azure OpenAI, Bedrock, Vertex AI, etc. Portkey supports running guardrails on provider batch requests. You pass guardrails through a config, and Portkey applies them asynchronously — pre-processing inputs before sending to the provider, and post-processing outputs after the provider completes.

How It Works

Upload a batch input file

Upload a JSONL file in OpenAI chat completions format to Portkey.

curl --location 'https://api.portkey.ai/v1/files' \
  --header 'x-portkey-api-key: {{PORTKEY_API_KEY}}' \
  --header 'x-portkey-provider: portkey' \
  --form 'file=@"/path/to/batch_requests.jsonl"' \
  --form 'purpose="batch"'

Each line in the file should be a valid chat completions request. Download an example JSONL file.The response returns a Portkey file ID (pk_file_*):

{
  "object": "file",
  "id": "pk_file_1db86eb1-22b5-4440-8810-e45ddd1e5aca",
  "purpose": "batch",
  "filename": "d2d44065-0f52-4780-a216-eb0122a25d78/pk_file_1db86eb1-22b5-4440-8810-e45ddd1e5aca.jsonl",
  "bytes": 26204,
  "created_at": 1777975975402,
  "status": "processed",
  "status_details": null
}

Create a config with guardrails

Guardrails are the only supported way to pass guardrail checks to a provider batch request. Define them in a Portkey config here:

{
  "retry": {
    "attempts": 3
  },
  "input_guardrails": ["pg-sync-g-0663db"],
  "output_guardrails": ["pg-mutato-9760bc"]
}

Save this config in the Portkey dashboard and note the config ID (e.g., pc-sync-g-1009a3).

Create a provider batch

Reference your uploaded file ID and config when creating the batch:

curl --location 'https://api.portkey.ai/v1/batches' \
  --header 'x-portkey-api-key: {{PORTKEY_API_KEY}}' \
  --header 'x-portkey-provider: portkey' \
  --header 'Content-Type: application/json' \
  --data '{
      "input_file_id": "pk_file_1db86eb1-22b5-4440-8810-e45ddd1e5aca",
      "endpoint": "/v1/chat/completions",
      "completion_window": "24h",
      "model": "gpt-3.5-turbo",
      "portkey_options": {
          "x-portkey-config": "pc-sync-g-1009a3",
          "x-portkey-virtual-key": "openai-virtual-key"
      }
  }'

The response contains a Portkey batch ID (pk_batch_*) and the initial guardrail results in hook_results:

{
  "id": "pk_batch_cd7f0e97-37b6-48cb-abd7-1bb699300585",
  "object": "batch",
  "status": "validating",
  "input_file_id": "pk_file_1db86eb1-22b5-4440-8810-e45ddd1e5aca",
  "hook_results": {
    "before_request_hooks": [],
    "after_request_hooks": [
      {
        "verdict": true,
        "id": "portkey-dataservice",
        "type": "guardrail",
        "deny": false
      }
    ]
  }
}

Poll for batch status

Poll the batch to track progress. Once input guardrails are applied, the input_file_id is replaced with a new processed file. Use include_provider_details=true to see full upstream details:

curl --location 'https://api.portkey.ai/v1/batches/pk_batch_cd7f0e97-37b6-48cb-abd7-1bb699300585?include_provider_details=true' \
  --header 'x-portkey-api-key: {{PORTKEY_API_KEY}}'

If your config includes output_guardrails, the batch enters an intermediate processing_post_request_guardrails state after the upstream provider finishes. Once complete, output_file_id reflects the final guardrail-processed output file.

Download the output file

Once the batch status is completed, retrieve the output using the Portkey-exclusive file content endpoint:

curl --location 'https://api.portkey.ai/v1/files/{{OUTPUT_FILE_ID}}/content' \
  --header 'x-portkey-api-key: {{PORTKEY_API_KEY}}' \
  --header 'x-portkey-provider: portkey'

Batch Lifecycle with Guardrails

State	Description
`validating`	Batch created; input guardrails being evaluated
`in_progress`	Input guardrails passed; provider is processing requests
`processing_post_request_guardrails`	Provider finished; output guardrails being evaluated
`completed`	All guardrails applied; output file ready

The input_file_id in the batch response is updated after input guardrails process the file — the new ID points to the transformed input sent to the upstream provider.

Introduction

Product

Self-Hosting

Support

How It Works

Batch Lifecycle with Guardrails

Portkey is now PRISMA AIRS AI Gateway. See it in action.

​How It Works

​Batch Lifecycle with Guardrails

Portkey is now PRISMA AIRS AI Gateway. See it in action.

How It Works

Batch Lifecycle with Guardrails