> ## Documentation Index
> Fetch the complete documentation index at: https://docs.portkey.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Google Gemini
Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including [Google Gemini APIs](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini).
With Portkey, you can take advantage of features like fast AI gateway access, observability, prompt management, and more, all while ensuring the secure management of your LLM API keys through [Model Catalog](/product/model-catalog).
## Quick Start
```python Python icon="python" theme={"system"}
from portkey_ai import Portkey
# 1. Install: pip install portkey-ai
# 2. Add @google provider in Model Catalog
# 3. Use it:
portkey = Portkey(api_key="PORTKEY_API_KEY")
response = portkey.chat.completions.create(
model="@google/gemini-1.5-pro",
messages=[{"role": "user", "content": "Say this is a test"}]
)
print(response.choices[0].message.content)
```
```js Javascript icon="square-js" theme={"system"}
import Portkey from 'portkey-ai'
// 1. Install: npm install portkey-ai
// 2. Add @google provider in Model Catalog
// 3. Use it:
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY"
})
const response = await portkey.chat.completions.create({
model: "@google/gemini-1.5-pro",
messages: [{ role: "user", content: "Say this is a test" }]
})
console.log(response.choices[0].message.content)
```
```python OpenAI Py icon="python" theme={"system"}
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL
# 1. Install: pip install openai portkey-ai
# 2. Add @google provider in Model Catalog
# 3. Use it:
client = OpenAI(
api_key="PORTKEY_API_KEY", # Portkey API key
base_url=PORTKEY_GATEWAY_URL
)
response = client.chat.completions.create(
model="@google/gemini-1.5-pro",
messages=[{"role": "user", "content": "Say this is a test"}]
)
print(response.choices[0].message.content)
```
```js OpenAI JS icon="square-js" theme={"system"}
import OpenAI from "openai"
import { PORTKEY_GATEWAY_URL } from "portkey-ai"
// 1. Install: npm install openai portkey-ai
// 2. Add @google provider in Model Catalog
// 3. Use it:
const client = new OpenAI({
apiKey: "PORTKEY_API_KEY", // Portkey API key
baseURL: PORTKEY_GATEWAY_URL
})
const response = await client.chat.completions.create({
model: "@google/gemini-1.5-pro",
messages: [{ role: "user", content: "Say this is a test" }]
})
console.log(response.choices[0].message.content)
```
```sh cURL icon="square-terminal" theme={"system"}
# 1. Add @google provider in Model Catalog
# 2. Use it:
curl https://api.portkey.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-d '{
"model": "@google/gemini-1.5-pro",
"messages": [{"role": "user", "content": "Say this is a test"}]
}'
```
***
## Add Provider in Model Catalog
Go to [**Model Catalog → Add Provider**](https://app.portkey.ai/model-catalog/providers) in your Portkey dashboard.
Find and select **Google** from the provider list.
Get your API key from [Google AI Studio](https://aistudio.google.com/app/apikey) and enter it in Model Catalog.
Save your configuration. Your provider slug will be `@google` (or a custom name you specify).
Portkey supports the `system_instructions` parameter for Google Gemini 1.5 - allowing you to control the behavior and output of your Gemini-powered applications with ease.
Simply include your Gemini system prompt as part of the `{"role":"system"}` message within the `messages` array of your request body. Portkey Gateway will automatically transform your message to ensure seamless compatibility with the Google Gemini API.
***
## Gemini Capabilities
### Function Calling
Portkey supports function calling mode on Google's Gemini Models. Explore this cookbook for a deep dive and examples:
[Function Calling](/guides/getting-started/function-calling)
***
## Advanced Multimodal Capabilities with Gemini
Gemini models are inherently multimodal, capable of processing and understanding content from a wide array of file types. Portkey streamlines the integration of these powerful features by providing a unified, OpenAI-compatible API.
**The Portkey Advantage: A Unified Format for All Media**
To simplify development, Portkey uses a consistent format for all multimodal requests. Whether you're sending an image, audio, video, or document, you will use an object with `type: 'image_url'` within the user message's `content` array.
Portkey's AI Gateway intelligently interprets your request—based on the URL or data URI you provide—and translates it into the precise format required by the Google Gemini API. This means you only need to learn one structure for all your media processing needs.
### Image Processing
Gemini can analyze images to describe their content, answer visual questions, or identify objects.
**Method 1: Sending an Image via Google Files URL**
Use the Google Files API to upload your image and get a URL. This is the recommended approach for larger files or when you need persistent storage.
To upload files and get Google Files URLs, use the [Files API](https://ai.google.dev/gemini-api/docs/files). The URL format will be similar to: `https://generativelanguage.googleapis.com/v1beta/files/[FILE_ID]`
```javascript NodeJS theme={"system"}
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{
role: 'user',
content: [
{
type: 'image_url',
image_url: {
url: 'https://generativelanguage.googleapis.com/v1beta/files/your-file-id'
}
},
{
type: 'text',
text: 'Describe this image in detail.'
}
]
}],
});
console.log(chatCompletion.choices[0].message.content);
```
```python Python theme={"system"}
completion = portkey.chat.completions.create(
model='gemini-1.5-pro',
messages=[{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id"
}
},
{
"type": "text",
"text": "Describe this image in detail."
}
]
}],
)
print(completion.choices[0].message.content)
```
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-1.5-pro",
"messages": [{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this image in detail."
},
{
"type": "image_url",
"image_url": { "url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id" }
}
]
}]
}'
```
**Method 2: Sending a Local Image as Base64 Data**
Use this method for local image files. The file is encoded into a Base64 string and sent as a data URI. This is ideal for smaller files when you don't want to use the Files API.
The data URI format is: `data:;base64,`
```javascript NodeJS theme={"system"}
import fs from 'fs';
const imageBytes = fs.readFileSync('local-image.png');
const base64Image = imageBytes.toString('base64');
const imageUri = `data:image/png;base64,${base64Image}`;
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{
role: 'user',
content: [
{ type: 'image_url', image_url: { url: imageUri }},
{ type: 'text', text: 'What is in this picture?' }
]
}],
});
console.log(chatCompletion.choices[0].message.content);
```
```python Python theme={"system"}
import base64
with open("local-image.png", "rb") as image_file:
image_bytes = image_file.read()
base64_image = base64.b64encode(image_bytes).decode('utf-8')
image_uri = f"data:image/png;base64,{base64_image}"
completion = portkey.chat.completions.create(
model='gemini-1.5-pro',
messages=[{
"role": "user",
"content": [
{ "type": "image_url", "image_url": { "url": image_uri }},
{ "type": "text", "text": "What is in this picture?" }
]
}],
)
print(completion.choices[0].message.content)
```
```sh cURL theme={"system"}
# First, encode your image file to base64
# For example: base64 -i local-image.png -o image.b64
# Then use the encoded content in the request
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-1.5-pro",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "What is in this picture?"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,YOUR_BASE64_IMAGE_DATA"}}
]
}]
}'
```
Supported Image MIME types: `image/png`, `image/jpeg`, `image/webp`, `image/heic`, `image/heif`
***
### Audio Processing
Gemini can transcribe speech, summarize audio content, or answer questions about sounds.
**Method 1: Sending Audio via Google Files URL**
Upload your audio file using the Files API to get a Google Files URL.
```javascript NodeJS theme={"system"}
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{
role: 'user',
content: [
{
type: 'image_url',
image_url: {
url: 'https://generativelanguage.googleapis.com/v1beta/files/your-audio-file-id'
}
},
{ type: 'text', text: 'Please transcribe the speech in this audio.' }
]
}],
});
console.log(chatCompletion.choices[0].message.content);
```
```python Python theme={"system"}
completion = portkey.chat.completions.create(
model='gemini-1.5-pro',
messages=[{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://generativelanguage.googleapis.com/v1beta/files/your-audio-file-id"
}
},
{ "type": "text", "text": "Please transcribe the speech in this audio." }
]
}],
)
print(completion.choices[0].message.content)
```
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-1.5-pro",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Please transcribe the speech in this audio."},
{"type": "image_url", "image_url": {"url": "https://generativelanguage.googleapis.com/v1beta/files/your-audio-file-id"}}
]
}]
}'
```
**Method 2: Sending Local Audio as Base64 Data**
This is the standard way to process local audio files directly through the API.
```javascript NodeJS theme={"system"}
import fs from 'fs';
const audioBytes = fs.readFileSync('audio-example.mp3');
const base64Audio = audioBytes.toString('base64');
const audioUri = `data:audio/mp3;base64,${base64Audio}`;
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{
role: 'user',
content: [
{ type: 'image_url', image_url: { url: audioUri }},
{ type: 'text', text: 'Describe this audio' }
]
}],
});
console.log(chatCompletion.choices[0].message.content);
```
```python Python theme={"system"}
import base64
with open('audio-example.mp3', 'rb') as audio_file:
audio_bytes = audio_file.read()
base64_audio = base64.b64encode(audio_bytes).decode('utf-8')
audio_uri = f"data:audio/mp3;base64,{base64_audio}"
completion = portkey.chat.completions.create(
model='gemini-1.5-pro',
messages=[{
"role": "user",
"content": [
{ "type": "image_url", "image_url": { "url": audio_uri }},
{ "type": "text", "text": "Describe this audio" }
]
}],
)
print(completion.choices[0].message.content)
```
```sh cURL theme={"system"}
# First, encode your audio file to base64
# For example: base64 -i audio-example.mp3 -o audio.b64
# Then use the encoded content in the request
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-1.5-pro",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Describe this audio"},
{"type": "image_url", "image_url": {"url": "data:audio/mp3;base64,YOUR_BASE64_AUDIO_DATA"}}
]
}]
}'
```
Supported Audio MIME types: `audio/wav`, `audio/mp3`, `audio/aiff`, `audio/aac`, `audio/ogg`, `audio/flac`, `audio/pcm`, `audio/m4a`, `audio/mpeg`, `audio/mpga`, `audio/mp4`, `audio/webm`
***
### Video Processing
Gemini can summarize videos, answer questions about specific events, and describe scenes.
**Method 1: Sending a Video via YouTube URL**
YouTube is the only supported public URL source for videos. Simply provide the YouTube video URL.
```javascript NodeJS theme={"system"}
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{
role: 'user',
content: [
{
type: 'text',
text: 'Describe this video in 3 sentences.'
},
{
type: 'image_url',
image_url: {
url: 'https://www.youtube.com/watch?v=9hE5-98ZeCg'
}
}
]
}],
});
console.log(chatCompletion.choices[0].message.content);
```
```python Python theme={"system"}
completion = portkey.chat.completions.create(
model='gemini-1.5-pro',
messages=[{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this video in 3 sentences."
},
{
"type": "image_url",
"image_url": {
"url": "https://www.youtube.com/watch?v=9hE5-98ZeCg"
}
}
]
}],
)
print(completion.choices[0].message.content)
```
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-1.5-pro",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Describe this video in 3 sentences."},
{"type": "image_url", "image_url": {"url": "https://www.youtube.com/watch?v=9hE5-98ZeCg"}}
]
}]
}'
```
**Method 2: Sending Local Video as Base64 Data**
For smaller video files, you can encode them as base64. Note that this method has size limitations.
```javascript NodeJS theme={"system"}
import fs from 'fs';
const videoBytes = fs.readFileSync('video-example.mp4');
const base64Video = videoBytes.toString('base64');
const videoUri = `data:video/mp4;base64,${base64Video}`;
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{
role: 'user',
content: [
{ type: 'image_url', image_url: { url: videoUri }},
{ type: 'text', text: 'Describe this video' }
]
}],
});
console.log(chatCompletion.choices[0].message.content);
```
```python Python theme={"system"}
import base64
with open('video-example.mp4', 'rb') as video_file:
video_bytes = video_file.read()
base64_video = base64.b64encode(video_bytes).decode('utf-8')
video_uri = f"data:video/mp4;base64,{base64_video}"
completion = portkey.chat.completions.create(
model='gemini-1.5-pro',
messages=[{
"role": "user",
"content": [
{ "type": "image_url", "image_url": { "url": video_uri }},
{ "type": "text", "text": "Describe this video" }
]
}],
)
print(completion.choices[0].message.content)
```
```sh cURL theme={"system"}
# First, encode your video file to base64
# For example: base64 -i video-example.mp4 -o video.b64
# Then use the encoded content in the request
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-1.5-pro",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Describe this video"},
{"type": "image_url", "image_url": {"url": "data:video/mp4;base64,YOUR_BASE64_VIDEO_DATA"}}
]
}]
}'
```
**Method 3: Sending Video via Google Files URL**
For larger video files, upload them using the Files API to get a Google Files URL.
```javascript NodeJS theme={"system"}
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{
role: 'user',
content: [
{
type: 'image_url',
image_url: {
url: 'https://generativelanguage.googleapis.com/v1beta/files/your-video-file-id'
}
},
{ type: 'text', text: 'Please describe the main events in this video.' }
]
}],
});
console.log(chatCompletion.choices[0].message.content);
```
```python Python theme={"system"}
completion = portkey.chat.completions.create(
model='gemini-1.5-pro',
messages=[{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://generativelanguage.googleapis.com/v1beta/files/your-video-file-id"
}
},
{ "type": "text", "text": "Please describe the main events in this video." }
]
}],
)
print(completion.choices[0].message.content)
```
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-1.5-pro",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Please describe the main events in this video."},
{"type": "image_url", "image_url": {"url": "https://generativelanguage.googleapis.com/v1beta/files/your-video-file-id"}}
]
}]
}'
```
Supported Video MIME types: `video/mp4`, `video/mpeg`, `video/mov`, `video/avi`, `video/webm`, `video/wmv`
***
### Document Processing (PDF)
Gemini's vision capabilities excel at understanding the content of PDF documents, including text, tables, and images.
**Method 1: Sending a Document via Google Files URL**
Upload your PDF using the Files API to get a Google Files URL.
```javascript NodeJS theme={"system"}
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{
role: 'user',
content: [
{
type: 'image_url',
image_url: {
url: 'https://generativelanguage.googleapis.com/v1beta/files/your-pdf-file-id'
}
},
{ type: 'text', text: 'Summarize the key findings of this research paper.' }
]
}],
});
console.log(chatCompletion.choices[0].message.content);
```
```python Python theme={"system"}
completion = portkey.chat.completions.create(
model='gemini-1.5-pro',
messages=[{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://generativelanguage.googleapis.com/v1beta/files/your-pdf-file-id"
}
},
{ "type": "text", "text": "Summarize the key findings of this research paper." }
]
}],
)
print(completion.choices[0].message.content)
```
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-1.5-pro",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Summarize the key findings of this research paper."},
{"type": "image_url", "image_url": {"url": "https://generativelanguage.googleapis.com/v1beta/files/your-pdf-file-id"}}
]
}]
}'
```
**Method 2: Sending a Local Document as Base64 Data**
This is suitable for smaller, local PDF files.
```javascript NodeJS theme={"system"}
import fs from 'fs';
const pdfBytes = fs.readFileSync('whitepaper.pdf');
const base64Pdf = pdfBytes.toString('base64');
const pdfUri = `data:application/pdf;base64,${base64Pdf}`;
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
messages: [{
role: 'user',
content: [
{ type: 'image_url', image_url: { url: pdfUri }},
{ type: 'text', text: 'What is the main conclusion of this document?' }
]
}],
});
console.log(chatCompletion.choices[0].message.content);
```
```python Python theme={"system"}
import base64
with open("whitepaper.pdf", "rb") as pdf_file:
pdf_bytes = pdf_file.read()
base64_pdf = base64.b64encode(pdf_bytes).decode('utf-8')
pdf_uri = f"data:application/pdf;base64,{base64_pdf}"
completion = portkey.chat.completions.create(
model='gemini-1.5-pro',
messages=[{
"role": "user",
"content": [
{ "type": "image_url", "image_url": { "url": pdf_uri }},
{ "type": "text", "text": "What is the main conclusion of this document?" }
]
}],
)
print(completion.choices[0].message.content)
```
```sh cURL theme={"system"}
# First, encode your PDF file to base64
# For example: base64 -i whitepaper.pdf -o pdf.b64
# Then use the encoded content in the request
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-1.5-pro",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "What is the main conclusion of this document?"},
{"type": "image_url", "image_url": {"url": "data:application/pdf;base64,YOUR_BASE64_PDF_DATA"}}
]
}]
}'
```
While you can send other document types like `.txt` or `.html`, they will be treated as plain text. Gemini's native document vision capabilities are optimized for the `application/pdf` MIME type.
**Important:** For all file uploads (except YouTube videos), it's recommended to use the [Google Files API](https://ai.google.dev/gemini-api/docs/files) to upload your files first, then use the returned file URL in your requests. This approach provides better performance and reliability for larger files.
***
## Media Resolution
The `media_resolution` parameter allows you to control token allocation for media inputs (images, videos, PDFs) when using Gemini models. This helps balance between processing detail and cost/speed.
### Supported values
| Value | Description |
| ----------------------------- | --------------------------------------------------------- |
| `MEDIA_RESOLUTION_LOW` | Reduced tokens for faster, cheaper processing |
| `MEDIA_RESOLUTION_MEDIUM` | Balanced approach between detail and cost |
| `MEDIA_RESOLUTION_HIGH` | Maximum tokens for detailed analysis |
| `MEDIA_RESOLUTION_ULTRA_HIGH` | Highest resolution (per-part only, for specialized tasks) |
### Top-level configuration
Apply media resolution globally to all media in the request:
```javascript NodeJS theme={"system"}
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-1.5-pro',
media_resolution: 'MEDIA_RESOLUTION_HIGH',
messages: [{
role: 'user',
content: [
{
type: 'image_url',
image_url: {
url: 'https://generativelanguage.googleapis.com/v1beta/files/your-file-id'
}
},
{ type: 'text', text: 'Analyze this image in detail.' }
]
}]
});
```
```python Python theme={"system"}
completion = portkey.chat.completions.create(
model='gemini-1.5-pro',
media_resolution='MEDIA_RESOLUTION_HIGH',
messages=[{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id"
}
},
{ "type": "text", "text": "Analyze this image in detail." }
]
}]
)
```
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-1.5-pro",
"media_resolution": "MEDIA_RESOLUTION_HIGH",
"messages": [{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id"}},
{"type": "text", "text": "Analyze this image in detail."}
]
}]
}'
```
### Per-part configuration (Gemini 3 only)
For Gemini 3 models, you can specify media resolution on individual media parts. Per-part settings take precedence over global settings when both are specified.
```javascript NodeJS theme={"system"}
const chatCompletion = await portkey.chat.completions.create({
model: 'gemini-3.0-pro',
messages: [{
role: 'user',
content: [
{
type: 'image_url',
image_url: {
url: 'https://generativelanguage.googleapis.com/v1beta/files/your-file-id',
media_resolution: 'MEDIA_RESOLUTION_HIGH'
}
},
{ type: 'text', text: 'Analyze this image in detail.' }
]
}]
});
```
```python Python theme={"system"}
completion = portkey.chat.completions.create(
model='gemini-3.0-pro',
messages=[{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id",
"media_resolution": "MEDIA_RESOLUTION_HIGH"
}
},
{ "type": "text", "text": "Analyze this image in detail." }
]
}]
)
```
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-3.0-pro",
"messages": [{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id",
"media_resolution": "MEDIA_RESOLUTION_HIGH"
}
},
{"type": "text", "text": "Analyze this image in detail."}
]
}]
}'
```
***
## Code Execution Tool
Gemini can use a built-in code interpreter tool to solve complex computational problems, perform calculations, and generate code. To enable this, simply include the `code_execution` tool in your request. The model will automatically decide when to invoke it.
```javascript NodeJS theme={"system"}
const response = await portkey.chat.completions.create({
model: "gemini-1.5-pro",
messages: [{
"role": "user",
"content": "Calculate the 20th Fibonacci number. Then find the nearest palindrome to it."
}],
tools: [{ "type": "code_execution" }]
});
console.log(response.choices[0].message.content);
```
```python Python theme={"system"}
response = portkey.chat.completions.create(
model="gemini-1.5-pro",
messages=[{
"role": "user",
"content": "Calculate the 20th Fibonacci number. Then find the nearest palindrome to it."
}],
tools=[{ "type": "code_execution" }]
)
print(response.choices[0].message.content)
```
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-1.5-pro",
"messages": [
{
"role": "user",
"content": "Calculate the 20th Fibonacci number. Then find the nearest palindrome to it."
}
],
"tools": [{ "type": "code_execution" }]
}'
```
***
## Thought Signatures (Tool Calling Verification)
Set `x-portkey-strict-open-ai-compliance` to `false` to receive Gemini-native thinking and thought fields (including `thought_signature`) in responses. These are not OpenAI-compatible response fields, so Portkey omits them when strict OpenAI compliance is enabled. This header must be included in all requests when using thought signatures.
Google's Gemini 3 Pro model requires passing a `thought_signature` parameter in tool calling conversations for verifying the payload. This signature is returned by the model in the assistant's tool call response and must be included when continuing multi-turn conversations.
### Single turn conversation
In a single-turn conversation, you make a request with tools defined, and the model returns tool calls with thought signatures.
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'x-portkey-strict-open-ai-compliance: false' \
--data '{
"model": "gemini-3-pro-preview",
"max_tokens": 1000,
"stream": true,
"messages": [
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful assistant"
}
]
},
{
"role": "user",
"content": "What is the current time in Bombay?"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_current_time",
"description": "Get the current time for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g., San Francisco, CA"
}
},
"required": [
"location"
]
}
}
}
]
}'
```
```py Python theme={"system"}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="@PROVIDER",
strict_open_ai_compliance=False
)
response = portkey.chat.completions.create(
model="gemini-3-pro-preview",
max_tokens=1000,
stream=True,
messages=[
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful assistant"
}
]
},
{
"role": "user",
"content": "What is the current time in Bombay?"
}
],
tools=[
{
"type": "function",
"function": {
"name": "get_current_time",
"description": "Get the current time for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g., San Francisco, CA"
}
},
"required": [
"location"
]
}
}
}
]
)
print(response)
```
```ts NodeJS theme={"system"}
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider: '@PROVIDER',
strictOpenAiCompliance: false
});
const response = await portkey.chat.completions.create({
model: 'gemini-3-pro-preview',
max_tokens: 1000,
stream: true,
messages: [
{
role: 'system',
content: [
{
type: 'text',
text: 'You are a helpful assistant'
}
]
},
{
role: 'user',
content: 'What is the current time in Bombay?'
}
],
tools: [
{
type: 'function',
function: {
name: 'get_current_time',
description: 'Get the current time for a specific location',
parameters: {
type: 'object',
properties: {
location: {
type: 'string',
description: 'The city and state, e.g., San Francisco, CA'
}
},
required: [
'location'
]
}
}
}
]
});
console.log(response);
```
```py OpenAI Python theme={"system"}
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
openai = OpenAI(
api_key='GEMINI_API_KEY',
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider='google',
api_key='PORTKEY_API_KEY',
strict_open_ai_compliance=False
)
)
response = openai.chat.completions.create(
model='gemini-3-pro-preview',
max_tokens=1000,
stream=True,
messages=[
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful assistant"
}
]
},
{
"role": "user",
"content": "What is the current time in Bombay?"
}
],
tools=[
{
"type": "function",
"function": {
"name": "get_current_time",
"description": "Get the current time for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g., San Francisco, CA"
}
},
"required": [
"location"
]
}
}
}
]
)
print(response)
```
```js OpenAI NodeJS theme={"system"}
import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';
const openai = new OpenAI({
apiKey: 'GEMINI_API_KEY',
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
provider: 'google',
apiKey: 'PORTKEY_API_KEY',
strictOpenAiCompliance: false
})
});
const response = await openai.chat.completions.create({
model: 'gemini-3-pro-preview',
max_tokens: 1000,
stream: true,
messages: [
{
role: 'system',
content: [
{
type: 'text',
text: 'You are a helpful assistant'
}
]
},
{
role: 'user',
content: 'What is the current time in Bombay?'
}
],
tools: [
{
type: 'function',
function: {
name: 'get_current_time',
description: 'Get the current time for a specific location',
parameters: {
type: 'object',
properties: {
location: {
type: 'string',
description: 'The city and state, e.g., San Francisco, CA'
}
},
required: [
'location'
]
}
}
}
]
});
console.log(response);
```
### Multi turn conversation
In multi-turn conversations, you must include the `thought_signature` field in the assistant's tool call when continuing the conversation.
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'x-portkey-strict-open-ai-compliance: false' \
--data '{
"model": "gemini-3-pro-preview",
"max_tokens": 1000,
"stream": true,
"messages": [
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful assistant"
}
]
},
{
"role": "user",
"content": "Check the time in Chennai and if it is later than 9Pm get the temperature"
},
{
"role": "assistant",
"tool_calls": [
{
"id": "portkey-1dcd51a0-a20a-482d-b244-2d4aff5aebdb",
"type": "function",
"function": {
"name": "get_current_time",
"arguments": "{\"location\":\"Chennai, India\"}",
"thought_signature": "CtQBAePx/17ARdotHH1RN31zOtCF+YpuOFTpU//tJRF4dEvegfDKLUaZnuG38II1POmVFdzBbzt87cTDr0TsEKHyHScN9PURHrhRer7liusjRrLR5QF4n1ZYJJYF3C+3bgC9YJsJyQhY/HAgVZQ53gq7n4I63CgXhYA+tzNN3CnHqdStgY0wLK0mCu/tb1kReSrXYMbre27SB5t2eRA7Wl+OKasKCOk7sYCJ8VkT+NaD+s6+NVTX2Au3RmUGVxYdjapo0vc7nnjvfmpTJHviyGJZIGIdXWw="
}
}
]
},
{
"role": "tool",
"content": "{ '\''time'\'': '\''10PM'\'' }",
"tool_call_id": "toolu_014jEfKqGbfFvRaKfiauxgPv"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_current_time",
"description": "Get the current time for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g., San Francisco, CA"
}
},
"required": [
"location"
]
}
}
},
{
"type": "function",
"function": {
"name": "get_current_temperature",
"description": "Get the current temperature for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g., San Francisco, CA"
},
"unit": {
"type": "string",
"enum": [
"Celsius",
"Fahrenheit"
],
"description": "The temperature unit to use. Infer this from the user'\''s location."
}
},
"required": [
"location",
"unit"
]
}
}
}
]
}'
```
```py Python theme={"system"}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="@PROVIDER",
strict_open_ai_compliance=False
)
response = portkey.chat.completions.create(
model="gemini-3-pro-preview",
max_tokens=1000,
stream=True,
messages=[
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful assistant"
}
]
},
{
"role": "user",
"content": "Check the time in Chennai and if it is later than 9Pm get the temperature"
},
{
"role": "assistant",
"tool_calls": [
{
"id": "portkey-1dcd51a0-a20a-482d-b244-2d4aff5aebdb",
"type": "function",
"function": {
"name": "get_current_time",
"arguments": "{\"location\":\"Chennai, India\"}",
"thought_signature": "CtQBAePx/17ARdotHH1RN31zOtCF+YpuOFTpU//tJRF4dEvegfDKLUaZnuG38II1POmVFdzBbzt87cTDr0TsEKHyHScN9PURHrhRer7liusjRrLR5QF4n1ZYJJYF3C+3bgC9YJsJyQhY/HAgVZQ53gq7n4I63CgXhYA+tzNN3CnHqdStgY0wLK0mCu/tb1kReSrXYMbre27SB5t2eRA7Wl+OKasKCOk7sYCJ8VkT+NaD+s6+NVTX2Au3RmUGVxYdjapo0vc7nnjvfmpTJHviyGJZIGIdXWw="
}
}
]
},
{
"role": "tool",
"content": "{ 'time': '10PM' }",
"tool_call_id": "toolu_014jEfKqGbfFvRaKfiauxgPv"
}
],
tools=[
{
"type": "function",
"function": {
"name": "get_current_time",
"description": "Get the current time for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g., San Francisco, CA"
}
},
"required": [
"location"
]
}
}
},
{
"type": "function",
"function": {
"name": "get_current_temperature",
"description": "Get the current temperature for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g., San Francisco, CA"
},
"unit": {
"type": "string",
"enum": [
"Celsius",
"Fahrenheit"
],
"description": "The temperature unit to use. Infer this from the user's location."
}
},
"required": [
"location",
"unit"
]
}
}
}
]
)
print(response)
```
```ts NodeJS theme={"system"}
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider: '@PROVIDER'
});
const response = await portkey.chat.completions.create({
model: 'gemini-3-pro-preview',
max_tokens: 1000,
stream: true,
messages: [
{
role: 'system',
content: [
{
type: 'text',
text: 'You are a helpful assistant'
}
]
},
{
role: 'user',
content: 'Check the time in Chennai and if it is later than 9Pm get the temperature'
},
{
role: 'assistant',
tool_calls: [
{
id: 'portkey-1dcd51a0-a20a-482d-b244-2d4aff5aebdb',
type: 'function',
function: {
name: 'get_current_time',
arguments: '{"location":"Chennai, India"}',
thought_signature: 'CtQBAePx/17ARdotHH1RN31zOtCF+YpuOFTpU//tJRF4dEvegfDKLUaZnuG38II1POmVFdzBbzt87cTDr0TsEKHyHScN9PURHrhRer7liusjRrLR5QF4n1ZYJJYF3C+3bgC9YJsJyQhY/HAgVZQ53gq7n4I63CgXhYA+tzNN3CnHqdStgY0wLK0mCu/tb1kReSrXYMbre27SB5t2eRA7Wl+OKasKCOk7sYCJ8VkT+NaD+s6+NVTX2Au3RmUGVxYdjapo0vc7nnjvfmpTJHviyGJZIGIdXWw='
}
}
]
},
{
role: 'tool',
content: "{ 'time': '10PM' }",
tool_call_id: 'toolu_014jEfKqGbfFvRaKfiauxgPv'
}
],
tools: [
{
type: 'function',
function: {
name: 'get_current_time',
description: 'Get the current time for a specific location',
parameters: {
type: 'object',
properties: {
location: {
type: 'string',
description: 'The city and state, e.g., San Francisco, CA'
}
},
required: [
'location'
]
}
}
},
{
type: 'function',
function: {
name: 'get_current_temperature',
description: 'Get the current temperature for a specific location',
parameters: {
type: 'object',
properties: {
location: {
type: 'string',
description: 'The city and state, e.g., San Francisco, CA'
},
unit: {
type: 'string',
enum: [
'Celsius',
'Fahrenheit'
],
description: 'The temperature unit to use. Infer this from the user\'s location.'
}
},
required: [
'location',
'unit'
]
}
}
}
]
});
console.log(response);
```
```py OpenAI Python theme={"system"}
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
openai = OpenAI(
api_key='GEMINI_API_KEY',
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider='google',
api_key='PORTKEY_API_KEY',
strict_open_ai_compliance=False
)
)
response = openai.chat.completions.create(
model='gemini-3-pro-preview',
max_tokens=1000,
stream=True,
messages=[
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful assistant"
}
]
},
{
"role": "user",
"content": "Check the time in Chennai and if it is later than 9Pm get the temperature"
},
{
"role": "assistant",
"tool_calls": [
{
"id": "portkey-1dcd51a0-a20a-482d-b244-2d4aff5aebdb",
"type": "function",
"function": {
"name": "get_current_time",
"arguments": "{\"location\":\"Chennai, India\"}",
"thought_signature": "CtQBAePx/17ARdotHH1RN31zOtCF+YpuOFTpU//tJRF4dEvegfDKLUaZnuG38II1POmVFdzBbzt87cTDr0TsEKHyHScN9PURHrhRer7liusjRrLR5QF4n1ZYJJYF3C+3bgC9YJsJyQhY/HAgVZQ53gq7n4I63CgXhYA+tzNN3CnHqdStgY0wLK0mCu/tb1kReSrXYMbre27SB5t2eRA7Wl+OKasKCOk7sYCJ8VkT+NaD+s6+NVTX2Au3RmUGVxYdjapo0vc7nnjvfmpTJHviyGJZIGIdXWw="
}
}
]
},
{
"role": "tool",
"content": "{ 'time': '10PM' }",
"tool_call_id": "toolu_014jEfKqGbfFvRaKfiauxgPv"
}
],
tools=[
{
"type": "function",
"function": {
"name": "get_current_time",
"description": "Get the current time for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g., San Francisco, CA"
}
},
"required": [
"location"
]
}
}
},
{
"type": "function",
"function": {
"name": "get_current_temperature",
"description": "Get the current temperature for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g., San Francisco, CA"
},
"unit": {
"type": "string",
"enum": [
"Celsius",
"Fahrenheit"
],
"description": "The temperature unit to use. Infer this from the user's location."
}
},
"required": [
"location",
"unit"
]
}
}
}
]
)
print(response)
```
```js OpenAI NodeJS theme={"system"}
import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';
const openai = new OpenAI({
apiKey: 'GEMINI_API_KEY',
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
provider: 'google',
apiKey: 'PORTKEY_API_KEY'
})
});
const response = await openai.chat.completions.create({
model: 'gemini-3-pro-preview',
max_tokens: 1000,
stream: true,
messages: [
{
role: 'system',
content: [
{
type: 'text',
text: 'You are a helpful assistant'
}
]
},
{
role: 'user',
content: 'Check the time in Chennai and if it is later than 9Pm get the temperature'
},
{
role: 'assistant',
tool_calls: [
{
id: 'portkey-1dcd51a0-a20a-482d-b244-2d4aff5aebdb',
type: 'function',
function: {
name: 'get_current_time',
arguments: '{"location":"Chennai, India"}',
thought_signature: 'CtQBAePx/17ARdotHH1RN31zOtCF+YpuOFTpU//tJRF4dEvegfDKLUaZnuG38II1POmVFdzBbzt87cTDr0TsEKHyHScN9PURHrhRer7liusjRrLR5QF4n1ZYJJYF3C+3bgC9YJsJyQhY/HAgVZQ53gq7n4I63CgXhYA+tzNN3CnHqdStgY0wLK0mCu/tb1kReSrXYMbre27SB5t2eRA7Wl+OKasKCOk7sYCJ8VkT+NaD+s6+NVTX2Au3RmUGVxYdjapo0vc7nnjvfmpTJHviyGJZIGIdXWw='
}
}
]
},
{
role: 'tool',
content: "{ 'time': '10PM' }",
tool_call_id: 'toolu_014jEfKqGbfFvRaKfiauxgPv'
}
],
tools: [
{
type: 'function',
function: {
name: 'get_current_time',
description: 'Get the current time for a specific location',
parameters: {
type: 'object',
properties: {
location: {
type: 'string',
description: 'The city and state, e.g., San Francisco, CA'
}
},
required: [
'location'
]
}
}
},
{
type: 'function',
function: {
name: 'get_current_temperature',
description: 'Get the current temperature for a specific location',
parameters: {
type: 'object',
properties: {
location: {
type: 'string',
description: 'The city and state, e.g., San Francisco, CA'
},
unit: {
type: 'string',
enum: [
'Celsius',
'Fahrenheit'
],
description: 'The temperature unit to use. Infer this from the user\'s location.'
}
},
required: [
'location',
'unit'
]
}
}
}
]
});
console.log(response);
```
The `thought_signature` is automatically generated by the model and returned in the tool call response. You must preserve this signature when including the assistant's message in subsequent requests.
***
## Computer Use (Browser Automation) (Preview)
Set strict\_open\_ai\_compliance to false to use the Computer Use tool.
### Single turn conversation
```ts NodeJS theme={"system"}
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider: '@PROVIDER',
strictOpenAiCompliance: false
});
const response = await portkey.chat.completions.create({
model: 'gemini-2.5-computer-use-preview-10-2025',
stream: false,
messages: [
{ role: 'system', content: 'You are a helpful assistant' },
{ role: 'user', content: "Go to google.com and search for 'weather in New York'" }
],
tools: [
{
type: 'function',
function: {
name: 'computer_use',
parameters: { environment: 'ENVIRONMENT_BROWSER' }
}
}
]
});
console.log(response);
```
```py Python theme={"system"}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="@PROVIDER",
strict_open_ai_compliance=False
)
response = portkey.chat.completions.create(
model="gemini-2.5-computer-use-preview-10-2025",
stream=False,
messages=[
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Go to google.com and search for 'weather in New York'"}
],
tools=[{
"type": "function",
"function": {
"name": "computer_use",
"parameters": {"environment": "ENVIRONMENT_BROWSER"}
}
}]
)
print(response)
```
```js OpenAI NodeJS theme={"system"}
import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';
const openai = new OpenAI({
apiKey: 'GEMINI_API_KEY',
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
provider: 'google',
apiKey: 'PORTKEY_API_KEY',
strictOpenAiCompliance: false
})
});
const response = await openai.chat.completions.create({
model: 'gemini-2.5-computer-use-preview-10-2025',
stream: false,
messages: [
{ role: 'system', content: 'You are a helpful assistant' },
{ role: 'user', content: "Go to google.com and search for 'weather in New York'" }
],
tools: [{
type: 'function',
function: { name: 'computer_use', parameters: { environment: 'ENVIRONMENT_BROWSER' } }
}]
});
console.log(response);
```
```py OpenAI Python theme={"system"}
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
openai = OpenAI(
api_key='GEMINI_API_KEY',
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider='google',
api_key='PORTKEY_API_KEY',
strict_open_ai_compliance=False
)
)
response = openai.chat.completions.create(
model='gemini-2.5-computer-use-preview-10-2025',
stream=False,
messages=[
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Go to google.com and search for 'weather in New York'"}
],
tools=[{
"type": "function",
"function": {"name": "computer_use", "parameters": {"environment": "ENVIRONMENT_BROWSER"}}
}]
)
print(response)
```
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: @my-vertex-ai-provider' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: your-api-key' \
--header 'x-portkey-strict-open-ai-compliance: false' \
--data '{
"model": "gemini-2.5-computer-use-preview-10-2025",
"stream": false,
"messages": [
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Go to google.com and search for '\''weather in New York'\''"}
],
"tools": [
{"type": "function", "function": {"name": "computer_use", "parameters": {"environment": "ENVIRONMENT_BROWSER"}}}
]
}'
```
### Multi turn conversation
```ts NodeJS theme={"system"}
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'PORTKEY_API_KEY',
provider: '@PROVIDER',
strictOpenAiCompliance: false
});
const response = await portkey.chat.completions.create({
model: 'gemini-2.5-computer-use-preview-10-2025',
stream: false,
messages: [
{ role: 'system', content: 'You are a helpful assistant' },
{ role: 'user', content: "Go to google.com and search for 'weather in New York'" },
{
role: 'assistant',
tool_calls: [
{ id: 'portkey-50925c03-b8cc-4057-948b-13a9d9de19e0', type: 'function', function: { name: 'open_web_browser', arguments: '{}' } }
]
},
{ role: 'user', content: "I've opened the browser" }
],
tools: [{
type: 'function',
function: { name: 'computerUse', parameters: { environment: 'ENVIRONMENT_BROWSER' } }
}]
});
console.log(response);
```
```py Python theme={"system"}
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="@PROVIDER",
strict_open_ai_compliance=False
)
response = portkey.chat.completions.create(
model="gemini-2.5-computer-use-preview-10-2025",
stream=False,
messages=[
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Go to google.com and search for 'weather in New York'"},
{
"role": "assistant",
"tool_calls": [
{"id": "portkey-50925c03-b8cc-4057-948b-13a9d9de19e0", "type": "function", "function": {"name": "open_web_browser", "arguments": "{}"}}
]
},
{"role": "user", "content": "I've opened the browser"}
],
tools=[{
"type": "function",
"function": {"name": "computerUse", "parameters": {"environment": "ENVIRONMENT_BROWSER"}}
}]
)
print(response)
```
```js OpenAI NodeJS theme={"system"}
import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';
const openai = new OpenAI({
apiKey: 'GEMINI_API_KEY',
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({ provider: 'google', apiKey: 'PORTKEY_API_KEY', strictOpenAiCompliance: false })
});
const response = await openai.chat.completions.create({
model: 'gemini-2.5-computer-use-preview-10-2025',
stream: false,
messages: [
{ role: 'system', content: 'You are a helpful assistant' },
{ role: 'user', content: "Go to google.com and search for 'weather in New York'" },
{ role: 'assistant', tool_calls: [{ id: 'portkey-50925c03-b8cc-4057-948b-13a9d9de19e0', type: 'function', function: { name: 'open_web_browser', arguments: '{}' } }] },
{ role: 'user', content: "I've opened the browser" }
],
tools: [{ type: 'function', function: { name: 'computerUse', parameters: { environment: 'ENVIRONMENT_BROWSER' } } }]
});
console.log(response);
```
```py OpenAI Python theme={"system"}
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
openai = OpenAI(
api_key='GEMINI_API_KEY',
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(provider='google', api_key='PORTKEY_API_KEY', strict_open_ai_compliance=False)
)
response = openai.chat.completions.create(
model='gemini-2.5-computer-use-preview-10-2025',
stream=False,
messages=[
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Go to google.com and search for 'weather in New York'"},
{"role": "assistant", "tool_calls": [{"id": "portkey-50925c03-b8cc-4057-948b-13a9d9de19e0", "type": "function", "function": {"name": "open_web_browser", "arguments": "{}"}}]},
{"role": "user", "content": "I've opened the browser"}
],
tools=[{ "type": "function", "function": { "name": "computerUse", "parameters": { "environment": "ENVIRONMENT_BROWSER" } } }]
)
print(response)
```
```sh cURL theme={"system"}
curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: @my-vertex-ai-provider' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: your-api-key' \
--header 'x-portkey-strict-open-ai-compliance: false' \
--data '{
"model": "gemini-2.5-computer-use-preview-10-2025",
"stream": false,
"messages": [
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Go to google.com and search for 'weather in New York'"},
{"role": "assistant", "tool_calls": [{"id": "portkey-50925c03-b8cc-4057-948b-13a9d9de19e0", "type": "function", "function": {"name": "open_web_browser", "arguments": "{}"}}]},
{"role": "user", "content": "I've opened the browser"}
],
"tools": [{"type": "function", "function": {"name": "computerUse", "parameters": {"environment": "ENVIRONMENT_BROWSER"}}}]
}'
```
## Grounding with Google Search
Vertex AI supports grounding with Google Search. This is a feature that allows you to ground your LLM responses with real-time search results.
Grounding is invoked by passing the `google_search` tool (for newer models like gemini-2.0-flash-001), and `google_search_retrieval` (for older models like gemini-1.5-flash) in the `tools` array.
```json theme={"system"}
"tools": [
{
"type": "function",
"function": {
"name": "google_search" // or google_search_retrieval for older models
}
}]
```
If you mix regular tools with grounding tools, vertex might throw an error saying only one tool can be used at a time.
## Extended Thinking (Reasoning Models) (Beta)
The assistants thinking response is returned in the `response_chunk.choices[0].delta.content_blocks` array, not the `response.choices[0].message.content` string.
Models like `gemini-2.5-flash-preview-04-17` `gemini-2.5-flash-preview-04-17` support [extended thinking](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude#claude-3-7-sonnet).
This is similar to openai thinking, but you get the model's reasoning as it processes the request as well.
Note that you will have to set [`strict_open_ai_compliance=False`](/product/ai-gateway/strict-open-ai-compliance) in the headers to use this feature.
### Single turn conversation
```py Python theme={"system"}
from portkey_ai import Portkey
# Initialize the Portkey client
portkey = Portkey(
api_key="PORTKEY_API_KEY", # Replace with your Portkey API key
provider="@PROVIDER",
strict_open_ai_compliance=False
)
# Create the request
response = portkey.chat.completions.create(
model="gemini-2.5-flash-preview-04-17",
max_tokens=3000,
thinking={
"type": "enabled",
"budget_tokens": 2030
},
stream=True,
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?"
}
]
}
]
)
print(response)
# in case of streaming responses you'd have to parse the response_chunk.choices[0].delta.content_blocks array
# response = portkey.chat.completions.create(
# ...same config as above but with stream: true
# )
# for chunk in response:
# if chunk.choices[0].delta:
# content_blocks = chunk.choices[0].delta.get("content_blocks")
# if content_blocks is not None:
# for content_block in content_blocks:
# print(content_block)
```
```ts NodeJS theme={"system"}
import Portkey from 'portkey-ai';
// Initialize the Portkey client
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY", // Replace with your Portkey API key
provider:"@PROVIDER", // your Vertex AI provider slug
strictOpenAiCompliance: false
});
// Generate a chat completion
async function getChatCompletionFunctions() {
const response = await portkey.chat.completions.create({
model: "gemini-2.5-flash-preview-04-17",
max_tokens: 3000,
thinking: {
type: "enabled",
budget_tokens: 2030
},
stream: true,
messages: [
{
role: "user",
content: [
{
type: "text",
text: "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?"
}
]
}
]
});
console.log(response);
// in case of streaming responses you'd have to parse the response_chunk.choices[0].delta.content_blocks array
// const response = await portkey.chat.completions.create({
// ...same config as above but with stream: true
// });
// for await (const chunk of response) {
// if (chunk.choices[0].delta?.content_blocks) {
// for (const contentBlock of chunk.choices[0].delta.content_blocks) {
// console.log(contentBlock);
// }
// }
// }
}
// Call the function
getChatCompletionFunctions();
```
```js OpenAI NodeJS theme={"system"}
import OpenAI from 'openai'; // We're using the v4 SDK
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'
const openai = new OpenAI({
apiKey: 'VERTEX_API_KEY', // defaults to process.env["OPENAI_API_KEY"],
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
provider: "vertex-ai",
apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
strictOpenAiCompliance: false
})
});
// Generate a chat completion with streaming
async function getChatCompletionFunctions(){
const response = await openai.chat.completions.create({
model: "gemini-2.5-flash-preview-04-17",
max_tokens: 3000,
thinking: {
type: "enabled",
budget_tokens: 2030
},
stream: true,
messages: [
{
role: "user",
content: [
{
type: "text",
text: "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?"
}
]
}
],
});
console.log(response)
// in case of streaming responses you'd have to parse the response_chunk.choices[0].delta.content_blocks array
// const response = await openai.chat.completions.create({
// ...same config as above but with stream: true
// });
// for await (const chunk of response) {
// if (chunk.choices[0].delta?.content_blocks) {
// for (const contentBlock of chunk.choices[0].delta.content_blocks) {
// console.log(contentBlock);
// }
// }
// }
}
await getChatCompletionFunctions();
```
```py OpenAI Python theme={"system"}
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
openai = OpenAI(
api_key='VERTEX_API_KEY',
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="vertex-ai",
api_key="PORTKEY_API_KEY",
strict_open_ai_compliance=False
)
)
response = openai.chat.completions.create(
model="gemini-2.5-flash-preview-04-17",
max_tokens=3000,
thinking={
"type": "enabled",
"budget_tokens": 2030
},
stream=True,
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?"
}
]
}
]
)
print(response)
```
```sh cURL theme={"system"}
curl "https://api.portkey.ai/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-provider: vertex-ai" \
-H "x-api-key: $VERTEX_API_KEY" \
-H "x-portkey-strict-open-ai-compliance: false" \
-d '{
"model": "gemini-2.5-flash-preview-04-17",
"max_tokens": 3000,
"thinking": {
"type": "enabled",
"budget_tokens": 2030
},
"stream": true,
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?"
}
]
}
]
}'
```
To disable thinking for gemini models like `gemini-2.5-flash-preview-04-17`, you are required to explicitly set `budget_tokens` to `0`.
```json theme={"system"}
"thinking": {
"type": "enabled",
"budget_tokens": 0
}
```
Gemini grounding mode may not work via Portkey SDK. [Contact support](mailto:support@portkey.ai) for assistance.
***
## Image Generation (nano banana 🍌)
Gemini models like `gemini-3-pro-image-preview` support native image generation capabilities. You can generate images by setting `modalities` to include `"image"` in your request.
You must set [`strict_open_ai_compliance=False`](/product/ai-gateway/strict-open-ai-compliance) in the headers to use image generation, as the response format includes non-standard fields like `content_parts`.
The generated image data is returned in the `content_parts` field of the response and can be used in multi-turn conversations for iterative image editing.
```python Python theme={"system"}
from portkey_ai import Portkey
# Initialize the Portkey client
portkey = Portkey(
api_key="PORTKEY_API_KEY", # Replace with your Portkey API key
provider="@google", # Your Google Provider Slug
strict_open_ai_compliance=False
)
# Create the request
response = portkey.chat.completions.create(
model="gemini-3-pro-image-preview",
max_tokens=32768,
stream=False,
modalities=["image"],
messages=[
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Create a picture of a cat eating a nano-banana in a fancy restaurant under the Gemini constellation."
}
]
}
]
)
print(response)
# To access the generated image data:
# The image will be in response.choices[0].message.content_parts
# Each content_part with type "image" contains base64-encoded image data
```
```typescript NodeJS theme={"system"}
import Portkey from 'portkey-ai';
// Initialize the Portkey client
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY", // Replace with your Portkey API key
provider: "@google", // Your Google Provider Slug
strictOpenAiCompliance: false
});
// Generate an image
async function generateImage() {
const response = await portkey.chat.completions.create({
model: "gemini-3-pro-image-preview",
max_tokens: 32768,
stream: false,
modalities: ["image"],
messages: [
{
role: "system",
content: "You are a helpful assistant."
},
{
role: "user",
content: [
{
type: "text",
text: "Create a picture of a cat eating a nano-banana in a fancy restaurant under the Gemini constellation."
}
]
}
]
});
console.log(response);
// To access the generated image data:
// The image will be in response.choices[0].message.content_parts
// Each content_part with type "image" contains base64-encoded image data
}
generateImage();
```
```javascript OpenAI NodeJS theme={"system"}
import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';
const openai = new OpenAI({
apiKey: 'GEMINI_API_KEY',
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
provider: "@google", // Your Google Provider Slug
apiKey: "PORTKEY_API_KEY",
strictOpenAiCompliance: false
})
});
async function generateImage() {
const response = await openai.chat.completions.create({
model: "gemini-3-pro-image-preview",
max_tokens: 32768,
stream: false,
modalities: ["image"],
messages: [
{
role: "system",
content: "You are a helpful assistant."
},
{
role: "user",
content: [
{
type: "text",
text: "Create a picture of a cat eating a nano-banana in a fancy restaurant under the Gemini constellation."
}
]
}
]
});
console.log(response);
}
generateImage();
```
```python OpenAI Python theme={"system"}
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
openai = OpenAI(
api_key='GEMINI_API_KEY',
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="@google", // Your Google Provider Slug
api_key="PORTKEY_API_KEY",
strict_open_ai_compliance=False
)
)
response = openai.chat.completions.create(
model="gemini-3-pro-image-preview",
max_tokens=32768,
stream=False,
modalities=["image"],
messages=[
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Create a picture of a cat eating a nano-banana in a fancy restaurant under the Gemini constellation."
}
]
}
]
)
print(response)
```
```sh cURL theme={"system"}
curl "https://api.portkey.ai/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-provider: @YOUR_" \
-H "Authorization: $GEMINI_API_KEY" \
-H "x-portkey-strict-open-ai-compliance: false" \
-d '{
"model": "gemini-3-pro-image-preview",
"max_tokens": 32768,
"stream": false,
"modalities": ["image"],
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Create a picture of a cat eating a nano-banana in a fancy restaurant under the Gemini constellation."
}
]
}
]
}'
```
### Image Generation with Text Response
You can also generate images along with text explanations by including both `"text"` and `"image"` in the modalities array:
```python Python theme={"system"}
response = portkey.chat.completions.create(
model="gemini-3-pro-image-preview",
max_tokens=32768,
stream=False,
modalities=["text", "image"], # Include both text and image
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Create a picture of a sunset over mountains and describe what you created."
}
]
}
]
)
print(response)
```
```typescript NodeJS theme={"system"}
const response = await portkey.chat.completions.create({
model: "gemini-3-pro-image-preview",
max_tokens: 32768,
stream: false,
modalities: ["text", "image"], // Include both text and image
messages: [
{
role: "user",
content: [
{
type: "text",
text: "Create a picture of a sunset over mountains and describe what you created."
}
]
}
]
});
console.log(response);
```
```sh cURL theme={"system"}
curl "https://api.portkey.ai/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: $PORTKEY_API_KEY" \
-H "x-portkey-provider: google" \
-H "Authorization: $GEMINI_API_KEY" \
-H "x-portkey-strict-open-ai-compliance: false" \
-d '{
"model": "gemini-3-pro-image-preview",
"max_tokens": 32768,
"stream": false,
"modalities": ["text", "image"],
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Create a picture of a sunset over mountains and describe what you created."
}
]
}
]
}'
```
### Image Editing (Multi-turn)
You can edit generated images by continuing the conversation. Pass the image data from the previous response back in the messages:
```python Python theme={"system"}
# First, generate an initial image
initial_response = portkey.chat.completions.create(
model="gemini-3-pro-image-preview",
max_tokens=32768,
modalities=["text", "image"],
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Create a picture of a croissant on a plate."
}
]
}
]
)
# Extract the assistant's response with the image
assistant_message = initial_response.choices[0].message
# Continue the conversation to edit the image
edit_response = portkey.chat.completions.create(
model="gemini-3-pro-image-preview",
max_tokens=32768,
modalities=["text", "image"],
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Create a picture of a croissant on a plate."
}
]
},
assistant_message, # Include the previous response with image
{
"role": "user",
"content": [
{
"type": "text",
"text": "Add some chocolate drizzle on top of the croissant."
}
]
}
]
)
print(edit_response)
```
```typescript NodeJS theme={"system"}
// First, generate an initial image
const initialResponse = await portkey.chat.completions.create({
model: "gemini-3-pro-image-preview",
max_tokens: 32768,
modalities: ["text", "image"],
messages: [
{
role: "user",
content: [
{
type: "text",
text: "Create a picture of a croissant on a plate."
}
]
}
]
});
// Extract the assistant's response with the image
const assistantMessage = initialResponse.choices[0].message;
// Continue the conversation to edit the image
const editResponse = await portkey.chat.completions.create({
model: "gemini-3-pro-image-preview",
max_tokens: 32768,
modalities: ["text", "image"],
messages: [
{
role: "user",
content: [
{
type: "text",
text: "Create a picture of a croissant on a plate."
}
]
},
assistantMessage, // Include the previous response with image
{
role: "user",
content: [
{
type: "text",
text: "Add some chocolate drizzle on top of the croissant."
}
]
}
]
});
console.log(editResponse);
```
## Next Steps
Complete SDK documentation and API reference
Add metadata to your Gemini requests
Configure advanced gateway features
Trace and monitor your Gemini requests
Create fallback configurations between providers