Request prompt function with Google Cloud Memorystore cluster auth
Protect your LLM services with rate limiting and Google Cloud Memorystore cluster auth. The AI Rate Limiting Advanced Policy analyzes query costs and token responses to provide an enterprise-grade rate limiting strategy.
The following example uses request prompt rate limiting, which lets you rate limit requests based on a custom token.
Prerequisites
-
An existing AI Model.
-
A running Redis instance on a Google Cloud Memorystore cluster..
-
Port
6379, or your custom Redis port, is open and reachable from AI Gateway. - Assign the principal to the corresponding role:
- Cloud Memorystore Redis DB Connection User(
roles/redis.dbConnectionUser) for Memorystore for Redis Cluster - Memorystore DB Connector User (
roles/memorystore.dbConnectionUser) for Memorystore for Valkey
Environment variables
-
CLUSTER_ADDRESS: The Memorystore cluster address. -
GCP_SERVICE_ACCOUNT: The GCP service account JSON.
ai_gateway_policies:
- ref: request-prompt-count-gcp-cluster
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
name: request-prompt-count-gcp-cluster
display_name: Request prompt function with Google Cloud Memorystore cluster auth
type: ai-rate-limiting-advanced
enabled: true
config:
strategy: redis
redis:
cluster_nodes:
- ip: !env CLUSTER_ADDRESS
port: 6379
port: 6379
cloud_authentication:
auth_provider: gcp
gcp_service_account_json: !env GCP_SERVICE_ACCOUNT
sync_rate: 0
policies:
- match:
- type: provider
partition_by: true
values:
- cohere
limits:
- limit: 100
window_size: 60
- limit: 1000
window_size: 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
global: falseMake sure to replace the following placeholders with your own values:
-
AI_GATEWAY_ID: Theidof your AI Gateway. -
!env CLUSTER_ADDRESS: The Memorystore cluster address. -
!env GCP_SERVICE_ACCOUNT: The GCP service account JSON.
curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "request-prompt-count-gcp-cluster",
"display_name": "Request prompt function with Google Cloud Memorystore cluster auth",
"type": "ai-rate-limiting-advanced",
"enabled": true,
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"port": 6379,
"cloud_authentication": {
"auth_provider": "gcp",
"gcp_service_account_json": "'$GCP_SERVICE_ACCOUNT'"
}
},
"sync_rate": 0,
"policies": [
{
"match": [
{
"type": "provider",
"partition_by": true,
"values": [
"cohere"
]
}
],
"limits": [
{
"limit": 100,
"window_size": 60
},
{
"limit": 1000,
"window_size": 3600
}
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"global": false
}
'Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
AIGatewayId: Theidof the AI Gateway.
See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}Add the following to your Terraform configuration:
resource "konnect_ai_gateway_policy" "my_policy" {
name = "request-prompt-count-gcp-cluster"
display_name = "Request prompt function with Google Cloud Memorystore cluster auth"
type = "ai-rate-limiting-advanced"
enabled = true
global = false
config = jsonencode({
strategy = "redis"
redis = {
cluster_nodes = [
{
ip = var.cluster_address
port = 6379
} ]
port = 6379
cloud_authentication = {
auth_provider = "gcp"
gcp_service_account_json = var.gcp_service_account
}
}
sync_rate = 0
policies = [
{
match = [
{
type = "provider"
partition_by = true
values = ["cohere"]
} ]
limits = [
{
limit = 100
window_size = 60
},
{
limit = 1000
window_size = 3600
} ]
} ]
request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
EOF
})
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "cluster_address" {
type = string
}
variable "gcp_service_account" {
type = string
}ai_gateway_policies:
- ref: request-prompt-count-gcp-cluster
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
name: request-prompt-count-gcp-cluster
display_name: Request prompt function with Google Cloud Memorystore cluster auth
type: ai-rate-limiting-advanced
enabled: true
config:
strategy: redis
redis:
cluster_nodes:
- ip: !env CLUSTER_ADDRESS
port: 6379
port: 6379
cloud_authentication:
auth_provider: gcp
gcp_service_account_json: !env GCP_SERVICE_ACCOUNT
sync_rate: 0
policies:
- match:
- type: provider
partition_by: true
values:
- cohere
limits:
- limit: 100
window_size: 60
- limit: 1000
window_size: 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
global: falseMake sure to replace the following placeholders with your own values:
-
AI_GATEWAY_ID: Theidof your AI Gateway. -
!env CLUSTER_ADDRESS: The Memorystore cluster address. -
!env GCP_SERVICE_ACCOUNT: The GCP service account JSON.
curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "request-prompt-count-gcp-cluster",
"display_name": "Request prompt function with Google Cloud Memorystore cluster auth",
"type": "ai-rate-limiting-advanced",
"enabled": true,
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"port": 6379,
"cloud_authentication": {
"auth_provider": "gcp",
"gcp_service_account_json": "'$GCP_SERVICE_ACCOUNT'"
}
},
"sync_rate": 0,
"policies": [
{
"match": [
{
"type": "provider",
"partition_by": true,
"values": [
"cohere"
]
}
],
"limits": [
{
"limit": 100,
"window_size": 60
},
{
"limit": 1000,
"window_size": 3600
}
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"global": false
}
'Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
AIGatewayId: Theidof the AI Gateway.
See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}Add the following to your Terraform configuration:
resource "konnect_ai_gateway_policy" "my_policy" {
name = "request-prompt-count-gcp-cluster"
display_name = "Request prompt function with Google Cloud Memorystore cluster auth"
type = "ai-rate-limiting-advanced"
enabled = true
global = false
config = jsonencode({
strategy = "redis"
redis = {
cluster_nodes = [
{
ip = var.cluster_address
port = 6379
} ]
port = 6379
cloud_authentication = {
auth_provider = "gcp"
gcp_service_account_json = var.gcp_service_account
}
}
sync_rate = 0
policies = [
{
match = [
{
type = "provider"
partition_by = true
values = ["cohere"]
} ]
limits = [
{
limit = 100
window_size = 60
},
{
limit = 1000
window_size = 3600
} ]
} ]
request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
EOF
})
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "cluster_address" {
type = string
}
variable "gcp_service_account" {
type = string
}ai_gateway_policies:
- ref: request-prompt-count-gcp-cluster
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
name: request-prompt-count-gcp-cluster
display_name: Request prompt function with Google Cloud Memorystore cluster auth
type: ai-rate-limiting-advanced
enabled: true
config:
strategy: redis
redis:
cluster_nodes:
- ip: !env CLUSTER_ADDRESS
port: 6379
port: 6379
cloud_authentication:
auth_provider: gcp
gcp_service_account_json: !env GCP_SERVICE_ACCOUNT
sync_rate: 0
policies:
- match:
- type: provider
partition_by: true
values:
- cohere
limits:
- limit: 100
window_size: 60
- limit: 1000
window_size: 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
global: falseMake sure to replace the following placeholders with your own values:
-
AI_GATEWAY_ID: Theidof your AI Gateway. -
!env CLUSTER_ADDRESS: The Memorystore cluster address. -
!env GCP_SERVICE_ACCOUNT: The GCP service account JSON.
curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "request-prompt-count-gcp-cluster",
"display_name": "Request prompt function with Google Cloud Memorystore cluster auth",
"type": "ai-rate-limiting-advanced",
"enabled": true,
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"port": 6379,
"cloud_authentication": {
"auth_provider": "gcp",
"gcp_service_account_json": "'$GCP_SERVICE_ACCOUNT'"
}
},
"sync_rate": 0,
"policies": [
{
"match": [
{
"type": "provider",
"partition_by": true,
"values": [
"cohere"
]
}
],
"limits": [
{
"limit": 100,
"window_size": 60
},
{
"limit": 1000,
"window_size": 3600
}
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"global": false
}
'Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
AIGatewayId: Theidof the AI Gateway.
See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}Add the following to your Terraform configuration:
resource "konnect_ai_gateway_policy" "my_policy" {
name = "request-prompt-count-gcp-cluster"
display_name = "Request prompt function with Google Cloud Memorystore cluster auth"
type = "ai-rate-limiting-advanced"
enabled = true
global = false
config = jsonencode({
strategy = "redis"
redis = {
cluster_nodes = [
{
ip = var.cluster_address
port = 6379
} ]
port = 6379
cloud_authentication = {
auth_provider = "gcp"
gcp_service_account_json = var.gcp_service_account
}
}
sync_rate = 0
policies = [
{
match = [
{
type = "provider"
partition_by = true
values = ["cohere"]
} ]
limits = [
{
limit = 100
window_size = 60
},
{
limit = 1000
window_size = 3600
} ]
} ]
request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
EOF
})
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "cluster_address" {
type = string
}
variable "gcp_service_account" {
type = string
}ai_gateway_policies:
- ref: request-prompt-count-gcp-cluster
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
name: request-prompt-count-gcp-cluster
display_name: Request prompt function with Google Cloud Memorystore cluster auth
type: ai-rate-limiting-advanced
enabled: true
config:
strategy: redis
redis:
cluster_nodes:
- ip: !env CLUSTER_ADDRESS
port: 6379
port: 6379
cloud_authentication:
auth_provider: gcp
gcp_service_account_json: !env GCP_SERVICE_ACCOUNT
sync_rate: 0
policies:
- match:
- type: provider
partition_by: true
values:
- cohere
limits:
- limit: 100
window_size: 60
- limit: 1000
window_size: 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
global: trueMake sure to replace the following placeholders with your own values:
-
AI_GATEWAY_ID: Theidof your AI Gateway. -
!env CLUSTER_ADDRESS: The Memorystore cluster address. -
!env GCP_SERVICE_ACCOUNT: The GCP service account JSON.
curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "request-prompt-count-gcp-cluster",
"display_name": "Request prompt function with Google Cloud Memorystore cluster auth",
"type": "ai-rate-limiting-advanced",
"enabled": true,
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"port": 6379,
"cloud_authentication": {
"auth_provider": "gcp",
"gcp_service_account_json": "'$GCP_SERVICE_ACCOUNT'"
}
},
"sync_rate": 0,
"policies": [
{
"match": [
{
"type": "provider",
"partition_by": true,
"values": [
"cohere"
]
}
],
"limits": [
{
"limit": 100,
"window_size": 60
},
{
"limit": 1000,
"window_size": 3600
}
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"global": true
}
'Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
AIGatewayId: Theidof the AI Gateway.
See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}Add the following to your Terraform configuration:
resource "konnect_ai_gateway_policy" "my_policy" {
name = "request-prompt-count-gcp-cluster"
display_name = "Request prompt function with Google Cloud Memorystore cluster auth"
type = "ai-rate-limiting-advanced"
enabled = true
global = true
config = jsonencode({
strategy = "redis"
redis = {
cluster_nodes = [
{
ip = var.cluster_address
port = 6379
} ]
port = 6379
cloud_authentication = {
auth_provider = "gcp"
gcp_service_account_json = var.gcp_service_account
}
}
sync_rate = 0
policies = [
{
match = [
{
type = "provider"
partition_by = true
values = ["cohere"]
} ]
limits = [
{
limit = 100
window_size = 60
},
{
limit = 1000
window_size = 3600
} ]
} ]
request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
EOF
})
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "cluster_address" {
type = string
}
variable "gcp_service_account" {
type = string
}After creating the policy, add request-prompt-count-gcp-cluster to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
ai_gateway_models:
- ref: my-gpt-4o
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
display_name: my-gpt-4o
name: my-gpt-4o
type: model
capabilities:
- generate
formats:
- type: openai
targets:
- name: gpt-4o
provider: my-openai-account
config:
type: openai
config:
route:
paths:
- "/v1"
model:
body_param: model
values:
- my-gpt-4o
logging:
payloads: false
policies:
- !ref request-prompt-count-gcp-cluster#nameAfter creating the policy, add request-prompt-count-gcp-cluster to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/models/my-gpt-4o \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"display_name": "my-gpt-4o",
"name": "my-gpt-4o",
"type": "model",
"capabilities": [
"generate"
],
"formats": [
{
"type": "openai"
}
],
"policies": [
"request-prompt-count-gcp-cluster"
],
"targets": [
{
"name": "gpt-4o",
"provider": "my-openai-account",
"config": {
"type": "openai"
}
}
],
"config": {
"route": {
"paths": [
"/v1"
],
"model": {
"body_param": "model",
"values": [
"my-gpt-4o"
]
}
},
"logging": {
"payloads": false
}
}
}
'After creating the policy, add request-prompt-count-gcp-cluster to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
resource "konnect_ai_gateway_model" "my_model" {
display_name = "my-gpt-4o"
name = "my-gpt-4o"
type = "model"
capabilities = ["generate"]
formats = [
{
type = "openai"
} ]
targets = [
{
name = "gpt-4o"
provider = "my-openai-account"
config = {
type = "openai"
}
} ]
config = jsonencode({
route = {
paths = ["/v1"]
model = {
body_param = "model"
values = ["my-gpt-4o"]
}
}
logging = {
payloads = false
}
})
policies = [konnect_ai_gateway_policy.my_policy.name]
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "cluster_address" {
type = string
}
variable "gcp_service_account" {
type = string
}After creating the policy, add request-prompt-count-gcp-cluster to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.
ai_gateway_consumers:
- ref: mobile-app-production
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
display_name: Mobile App - Production
name: mobile-app-production
type: api-key
policies:
- !ref request-prompt-count-gcp-cluster#nameAfter creating the policy, add request-prompt-count-gcp-cluster to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.
curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/consumers/mobile-app-production \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"display_name": "Mobile App - Production",
"name": "mobile-app-production",
"type": "api-key",
"policies": [
"request-prompt-count-gcp-cluster"
]
}
'After creating the policy, add request-prompt-count-gcp-cluster to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.
resource "konnect_ai_gateway_consumer" "my_consumer" {
display_name = "Mobile App - Production"
name = "mobile-app-production"
type = "api-key"
policies = [konnect_ai_gateway_policy.my_policy.name]
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "cluster_address" {
type = string
}
variable "gcp_service_account" {
type = string
}After creating the policy, add request-prompt-count-gcp-cluster to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.
ai_gateway_consumer_groups:
- ref: internal-teams
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
display_name: Internal Teams
name: internal-teams
policies:
- !ref request-prompt-count-gcp-cluster#nameAfter creating the policy, add request-prompt-count-gcp-cluster to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.
curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/consumer-groups/internal-teams \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"display_name": "Internal Teams",
"name": "internal-teams",
"policies": [
"request-prompt-count-gcp-cluster"
]
}
'After creating the policy, add request-prompt-count-gcp-cluster to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.
resource "konnect_ai_gateway_consumer_group" "my_consumer_group" {
display_name = "Internal Teams"
name = "internal-teams"
policies = [konnect_ai_gateway_policy.my_policy.name]
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "cluster_address" {
type = string
}
variable "gcp_service_account" {
type = string
}