Create a Gateway Service and a Route, then enable the AI Proxy plugin and configure it with the Vertex AI provider and add the model and your API key.
This is a Konnect tutorial and requires a Konnect personal access token.
Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
Export your token to an environment variable:
export KONNECT_TOKEN='YOUR_KONNECT_PAT'
Run the quickstart script to automatically provision a Control Plane and Data Plane, and configure your environment:
curl -Ls https://get.konghq.com/quickstart | bash -s -- -k $KONNECT_TOKEN --deck-output
This sets up a Konnect Control Plane named quickstart, provisions a local Data Plane, and prints out the following environment variable exports:
export DECK_KONNECT_TOKEN=$KONNECT_TOKEN
export DECK_KONNECT_CONTROL_PLANE_NAME=quickstart
export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
export KONNECT_PROXY_URL='http://localhost:8000'
Copy and paste these into your terminal to configure your session.
This tutorial requires Kong Gateway Enterprise. If you don’t have Kong Gateway set up yet, you can use the quickstart script with an enterprise license to get an instance of Kong Gateway running almost instantly.
Export your license to an environment variable:
export KONG_LICENSE_DATA='LICENSE-CONTENTS-GO-HERE'
Run the quickstart script:
curl -Ls https://get.konghq.com/quickstart | bash -s -- -e KONG_LICENSE_DATA
Once Kong Gateway is ready, you will see the following message:
Kong Gateway Ready
decK is a CLI tool for managing Kong Gateway declaratively with state files. To complete this tutorial, install decK version 1.43 or later.
This guide uses deck gateway apply, which directly applies entity configuration to your Gateway instance.
We recommend upgrading your decK installation to take advantage of this tool.
You can check your current decK version with deck version.
For this tutorial, you’ll need Kong Gateway entities, like Gateway Services and Routes, pre-configured. These entities are essential for Kong Gateway to function but installing them isn’t the focus of this guide. Follow these steps to pre-configure them:
Run the following command:
echo '
_format_version: "3.0"
services:
- name: example-service
url: http://httpbin.konghq.com/anything
routes:
- name: example-route
paths:
- "/anything"
service:
name: example-service
' | deck gateway apply -
To learn more about entities, you can read our entities documentation.
Before you begin, you must get the following credentials from Google Cloud:
https://aiplatform.googleapis.com
After creating the key, convert the contents of modelarmor-admin-key.json into a single-line JSON string.
Escape all necessary characters. Quotes (") become \" and newlines become \n. The result must be a valid one-line JSON string.
Then export your credentials as environment variables:
export DECK_GCP_SERVICE_ACCOUNT_JSON="<single-line-escaped-json>"
export DECK_GCP_LOCATION_ID="<your_location_id>"
export DECK_GCP_API_ENDPOINT="<your_gcp_api_endpoint>"
export DECK_GCP_PROJECT_ID="<your-gcp-project-id>"
Set up GCP Application Default Credentials (ADC) with your quota project:
gcloud auth application-default set-quota-project <your_gcp_project_id>
Replace <your_gcp_project_id> with your actual project ID. This configures ADC to use your project for API quota and billing.
To set up AI Proxy with Vertex AI, specify the model and set the appropriate authentication header.
In this example, we’ll use the Gemini 2.0 Flash Exp model:
echo '
_format_version: "3.0"
plugins:
- name: ai-proxy
config:
route_type: llm/v1/chat
model:
provider: gemini
name: gemini-2.0-flash-exp
options:
gemini:
api_endpoint: "${{ env "DECK_GCP_API_ENDPOINT" }}"
project_id: "${{ env "DECK_GCP_PROJECT_ID" }}"
location_id: "${{ env "DECK_GCP_LOCATION_ID" }}"
auth:
gcp_use_service_account: true
gcp_service_account_json: |-
${{ env "DECK_GCP_SERVICE_ACCOUNT_JSON" }}
' | deck gateway apply -
Send a request to the Route to validate.
curl -X POST "$KONNECT_PROXY_URL/anything" \
--no-progress-meter --fail-with-body \
-H "Accept: application/json"\
-H "Content-Type: application/json" \
--json '{
"messages": [
{
"role": "system",
"content": "You are a mathematician"
},
{
"role": "user",
"content": "What is 1+1?"
}
]
}'
curl -X POST "http://localhost:8000/anything" \
--no-progress-meter --fail-with-body \
-H "Accept: application/json"\
-H "Content-Type: application/json" \
--json '{
"messages": [
{
"role": "system",
"content": "You are a mathematician"
},
{
"role": "user",
"content": "What is 1+1?"
}
]
}'
If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.
curl -Ls https://get.konghq.com/quickstart | bash -s -- -d