October '24

​

1.8.0

​

Bedrock Converse API integration

• Bedrock’s /chat/completions have been updated to use Bedrock converse API.
• This enables features like tool calls, vision, etc. for many bedrock models.
• This also removes the hassle of maintaining chat templating logic for llama and mistral models.

​

Vertex Image Generation

• Added support for Vertex Imagen models.

​

Stable Diffusion v2 Models

• StabilityAI introduced v2 models with a new API signature. Gateway now supports both v1 and v2 models, with internal transformations for different API signatures.
• Supported for both stability-ai and bedrock providers.
• New models: Stable Image Ultra, Core, 3.0 and 3.5.

​

Pydantic SDK Integration for Structured Outputs

• Done for GoogleAI and VertexAI (follows OpenAI)
• We previously added support for structured outputs through REST API. However, SDKs using Pydantic were not supported due to extra fields in the JSON schema.
• Added a dereferencing function that converts JSON schemas from the library to Google-compatible schemas.

​

OpenAI and AzureOpenAI Prompt Cache Pricing

• Added support for handling prompt caching pricing for required models.

​

New Providers

• Lambda (lambda): Supports chat completions and completions.

​

Fixes & Enhancements:

• Exclude files, batches, threads, etc. from llm_cost_sum prometheus metric. Apart from the unified routes, all other routes will be excluded from llm_cost_sum metric to avoid unnecessary labels.
• PerplexityAI: Added the missing [DONE] chunk for stream calls to comply with OpenAI’s spec.
• VertexAI: Fixed provider name extraction logic for meta models, so users can send it like other partner models (e.g., meta.<model-name>).
• GoogleAI: Added structured outputs support (similar to Vertex-ai).
• Updated/Added pricing for new models.

​

Block api.portkey.ai

• We now block Gateway routes for Enterprise Organisations (Configurable)

​

1.7.5