Use Google Generative AI SDK for Gemini AI service chats with Kong AI Gateway
Configure the AI Proxy Advanced plugin with llm_format set to gemini, then use the Google Generative AI SDK to send requests through Kong AI Gateway.
Prerequisites
Kong Konnect
This is a Konnect tutorial and requires a Konnect personal access token.
-
Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
-
Export your token to an environment variable:
export KONNECT_TOKEN='YOUR_KONNECT_PAT'Copied! -
Run the quickstart script to automatically provision a Control Plane and Data Plane, and configure your environment:
curl -Ls https://get.konghq.com/quickstart | bash -s -- -k $KONNECT_TOKEN --deck-outputCopied!This sets up a Konnect Control Plane named
quickstart, provisions a local Data Plane, and prints out the following environment variable exports:export DECK_KONNECT_TOKEN=$KONNECT_TOKEN export DECK_KONNECT_CONTROL_PLANE_NAME=quickstart export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com export KONNECT_PROXY_URL='http://localhost:8000'Copied!Copy and paste these into your terminal to configure your session.
Kong Gateway running
This tutorial requires Kong Gateway Enterprise. If you don’t have Kong Gateway set up yet, you can use the quickstart script with an enterprise license to get an instance of Kong Gateway running almost instantly.
-
Export your license to an environment variable:
export KONG_LICENSE_DATA='LICENSE-CONTENTS-GO-HERE'Copied! -
Run the quickstart script:
curl -Ls https://get.konghq.com/quickstart | bash -s -- -e KONG_LICENSE_DATACopied!Once Kong Gateway is ready, you will see the following message:
Kong Gateway Ready
decK v1.43+
decK is a CLI tool for managing Kong Gateway declaratively with state files. To complete this tutorial, install decK version 1.43 or later.
This guide uses deck gateway apply, which directly applies entity configuration to your Gateway instance.
We recommend upgrading your decK installation to take advantage of this tool.
You can check your current decK version with deck version.
Required entities
For this tutorial, you’ll need Kong Gateway entities, like Gateway Services and Routes, pre-configured. These entities are essential for Kong Gateway to function but installing them isn’t the focus of this guide. Follow these steps to pre-configure them:
-
Run the following command:
echo ' _format_version: "3.0" services: - name: gemini-service url: http://httpbin.konghq.com/ routes: - name: gemini-route paths: - "/gemini" service: name: gemini-service ' | deck gateway apply -Copied!
To learn more about entities, you can read our entities documentation.
Gemini AI
Before you begin, you must get the Gemini API key from Google Cloud:
- Go to the Google Cloud Console.
- Select or create a project.
- Enable the Generative Language API:
- Navigate to APIs & Services > Library.
- Search for “Generative Language API”.
- Click Enable.
- Create an API key:
- Navigate to APIs & Services > Credentials.
- Click Create Credentials > API Key.
- Copy the generated API key.
Export the API key as an environment variable:
export DECK_GEMINI_API_KEY="<your_gemini_api_key>"
Python
To complete this tutorial, you’ll need Python (version 3.7 or later) and pip installed on your machine. You can verify it by running:
python3
python3 -m pip --version
-
Create a virtual env:
python3 -m venv myenvCopied! -
Activate it:
source myenv/bin/activateCopied!
Google Generative AI SDK
Install the Google Generative AI SDK:
pip install google-generativeai
Configure the AI Proxy plugin
The AI Proxy plugin supports Google’s Gemini models and works with the Google Generative AI SDK. This configuration allows you to use the standard Gemini SDK. Apply the plugin configuration with your GCP service account credentials:
echo '
_format_version: "3.0"
plugins:
- name: ai-proxy
service: gemini-service
config:
route_type: llm/v1/chat
llm_format: gemini
auth:
param_name: key
param_value: "${{ env "DECK_GEMINI_API_KEY" }}"
param_location: query
model:
provider: gemini
name: gemini-2.0-flash-exp
' | deck gateway apply -
Test with Google Generative AI SDK
Create a test script that uses the Google Generative AI SDK. The script initializes a client with a dummy API key because Kong AI Gateway handles authentication, then sends a generation request through the gateway:
cat << 'EOF' > gemini.py
#!/usr/bin/env python3
import os
from google import genai
BASE_URL = "http://localhost:8000/gemini"
def gemini_chat():
try:
print(f"Connecting to: {BASE_URL}")
client = genai.Client(
api_key=os.environ.get("DECK_GEMINI_API_KEY"),
vertexai=False,
http_options={
"base_url": BASE_URL
}
)
print("Sending message...")
response = client.models.generate_content(
model="gemini-2.0-flash-exp",
contents="Hello! How are you?"
)
print(f"Response: {response.text}")
except Exception as e:
print(f"Error: {e}")
import traceback
traceback.print_exc()
if __name__ == "__main__":
gemini_chat()
EOF
Run the script:
python3 gemini.py
Expected output:
Connecting to: http://localhost:8000/gemini
Sending message...
Response: Hello! I'm doing well, thank you for asking. As a large language model, I don't experience feelings or emotions in the way humans do, but I'm functioning properly and ready to assist you. How can I help you today?
Cleanup
Clean up Konnect environment
If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.
Destroy the Kong Gateway container
curl -Ls https://get.konghq.com/quickstart | bash -s -- -d