Request prompt function with Azure Managed Redis cluster auth
Protect your LLM services with rate limiting and Azure Managed Redis cluster auth. The AI Rate Limiting Advanced plugin will analyze query costs and token response to provide an enterprise-grade rate limiting strategy.
The following example uses request prompt rate limiting, which lets you you rate limit requests based on a custom token. See the how-to guide for a step-by-step walkthrough.
Prerequisites
-
AI Proxy plugin or AI Proxy Advanced plugin configured with an LLM service
-
A running Redis instance on an Azure Managed Redis cluster with Entra authentication configured
-
Port
6379, or your custom Redis port is open and reachable from Kong Gateway. -
Add the user/service principal/identity to the “Microsoft Entra Authentication Redis user” list for the Azure Managed Redis instance
Environment variables
-
CLUSTER_ADDRESS: The Azure Managed Redis cluster address. -
CLUSTER_USERNAME: The object (principal) ID of the Principal/Identity with essential access. -
AZURE_CLIENT_ID: The client ID of the Principal/Identity. -
AZURE_CLIENT_SECRET: (Optional) The client secret of the Principal/Identity. -
AZURE_TENANT_ID: (Optional) The tenant ID of the Principal/Identity.
Add this section to your kong.yaml configuration file:
kong.yaml
Copied!
_format_version: "3.0"
plugins:
- name: ai-rate-limiting-advanced
config:
strategy: redis
redis:
cluster_nodes:
- ip: ${{ env "DECK_CLUSTER_ADDRESS" }}
port: 6379
username: ${{ env "DECK_CLUSTER_USERNAME" }}
port: 6379
cloud_authentication:
auth_provider: azure
azure_client_id: ${{ env "DECK_AZURE_CLIENT_ID" }}
azure_client_secret: ${{ env "DECK_AZURE_CLIENT_SECRET" }}
azure_tenant_id: ${{ env "DECK_AZURE_TENANT_ID" }}
sync_rate: 0
llm_providers:
- name: cohere
limit:
- 100
- 1000
window_size:
- 60
- 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
Make the following request:
curl -i -X POST http://localhost:8001/plugins/ \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '
{
"name": "ai-rate-limiting-advanced",
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"username": "'$CLUSTER_USERNAME'",
"port": 6379,
"cloud_authentication": {
"auth_provider": "azure",
"azure_client_id": "'$AZURE_CLIENT_ID'",
"azure_client_secret": "'$AZURE_CLIENT_SECRET'",
"azure_tenant_id": "'$AZURE_TENANT_ID'"
}
},
"sync_rate": 0,
"llm_providers": [
{
"name": "cohere",
"limit": [
100,
1000
],
"window_size": [
60,
3600
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"tags": []
}
'
Copied!
Make the following request:
curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/plugins/ \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "ai-rate-limiting-advanced",
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"username": "'$CLUSTER_USERNAME'",
"port": 6379,
"cloud_authentication": {
"auth_provider": "azure",
"azure_client_id": "'$AZURE_CLIENT_ID'",
"azure_client_secret": "'$AZURE_CLIENT_SECRET'",
"azure_tenant_id": "'$AZURE_TENANT_ID'"
}
},
"sync_rate": 0,
"llm_providers": [
{
"name": "cohere",
"limit": [
100,
1000
],
"window_size": [
60,
3600
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"tags": []
}
'
Copied!
Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
controlPlaneId: Theidof the control plane.
See the Konnect API reference to learn about region-specific URLs and personal access tokens.
echo "
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
name: ai-rate-limiting-advanced
namespace: kong
annotations:
kubernetes.io/ingress.class: kong
konghq.com/tags: ''
labels:
global: 'true'
config:
strategy: redis
redis:
cluster_nodes:
- ip: '$CLUSTER_ADDRESS'
port: 6379
username: '$CLUSTER_USERNAME'
port: 6379
cloud_authentication:
auth_provider: azure
azure_client_id: '$AZURE_CLIENT_ID'
azure_client_secret: '$AZURE_CLIENT_SECRET'
azure_tenant_id: '$AZURE_TENANT_ID'
sync_rate: 0
llm_providers:
- name: cohere
limit:
- 100
- 1000
window_size:
- 60
- 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header('x-prompt-count'))
if header_count then
return header_count
end
return 0
plugin: ai-rate-limiting-advanced
" | kubectl apply -f -
Copied!
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}
Copied!
Add the following to your Terraform configuration to create a Konnect Gateway Plugin:
resource "konnect_gateway_plugin_ai_rate_limiting_advanced" "my_ai_rate_limiting_advanced" {
enabled = true
config = {
strategy = "redis"
redis = {
cluster_nodes = [
{
ip = var.cluster_address
port = 6379
} ]
username = var.cluster_username
port = 6379
cloud_authentication = {
auth_provider = "azure"
azure_client_id = var.azure_client_id
azure_client_secret = var.azure_client_secret
azure_tenant_id = var.azure_tenant_id
}
}
sync_rate = 0
llm_providers = [
{
name = "cohere"
limit = [100, 1000]
window_size = [60, 3600]
} ]
request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
EOF
}
tags = []
control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
}
Copied!
This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "azure_tenant_id" {
type = string
}
Copied!
Add this section to your kong.yaml configuration file:
kong.yaml
Copied!
_format_version: "3.0"
plugins:
- name: ai-rate-limiting-advanced
service: serviceName|Id
config:
strategy: redis
redis:
cluster_nodes:
- ip: ${{ env "DECK_CLUSTER_ADDRESS" }}
port: 6379
username: ${{ env "DECK_CLUSTER_USERNAME" }}
port: 6379
cloud_authentication:
auth_provider: azure
azure_client_id: ${{ env "DECK_AZURE_CLIENT_ID" }}
azure_client_secret: ${{ env "DECK_AZURE_CLIENT_SECRET" }}
azure_tenant_id: ${{ env "DECK_AZURE_TENANT_ID" }}
sync_rate: 0
llm_providers:
- name: cohere
limit:
- 100
- 1000
window_size:
- 60
- 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
Make sure to replace the following placeholders with your own values:
-
serviceName|Id: Theidornameof the service the plugin configuration will target.
Make the following request:
curl -i -X POST http://localhost:8001/services/{serviceName|Id}/plugins/ \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '
{
"name": "ai-rate-limiting-advanced",
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"username": "'$CLUSTER_USERNAME'",
"port": 6379,
"cloud_authentication": {
"auth_provider": "azure",
"azure_client_id": "'$AZURE_CLIENT_ID'",
"azure_client_secret": "'$AZURE_CLIENT_SECRET'",
"azure_tenant_id": "'$AZURE_TENANT_ID'"
}
},
"sync_rate": 0,
"llm_providers": [
{
"name": "cohere",
"limit": [
100,
1000
],
"window_size": [
60,
3600
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"tags": []
}
'
Copied!
Make sure to replace the following placeholders with your own values:
-
serviceName|Id: Theidornameof the service the plugin configuration will target.
Make the following request:
curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/services/{serviceId}/plugins/ \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "ai-rate-limiting-advanced",
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"username": "'$CLUSTER_USERNAME'",
"port": 6379,
"cloud_authentication": {
"auth_provider": "azure",
"azure_client_id": "'$AZURE_CLIENT_ID'",
"azure_client_secret": "'$AZURE_CLIENT_SECRET'",
"azure_tenant_id": "'$AZURE_TENANT_ID'"
}
},
"sync_rate": 0,
"llm_providers": [
{
"name": "cohere",
"limit": [
100,
1000
],
"window_size": [
60,
3600
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"tags": []
}
'
Copied!
Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
controlPlaneId: Theidof the control plane. -
serviceId: Theidof the service the plugin configuration will target.
See the Konnect API reference to learn about region-specific URLs and personal access tokens.
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: ai-rate-limiting-advanced
namespace: kong
annotations:
kubernetes.io/ingress.class: kong
konghq.com/tags: ''
config:
strategy: redis
redis:
cluster_nodes:
- ip: '$CLUSTER_ADDRESS'
port: 6379
username: '$CLUSTER_USERNAME'
port: 6379
cloud_authentication:
auth_provider: azure
azure_client_id: '$AZURE_CLIENT_ID'
azure_client_secret: '$AZURE_CLIENT_SECRET'
azure_tenant_id: '$AZURE_TENANT_ID'
sync_rate: 0
llm_providers:
- name: cohere
limit:
- 100
- 1000
window_size:
- 60
- 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header('x-prompt-count'))
if header_count then
return header_count
end
return 0
plugin: ai-rate-limiting-advanced
" | kubectl apply -f -
Copied!
Next, apply the KongPlugin resource by annotating the service resource:
kubectl annotate -n kong service SERVICE_NAME konghq.com/plugins=ai-rate-limiting-advanced
Copied!
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}
Copied!
Add the following to your Terraform configuration to create a Konnect Gateway Plugin:
resource "konnect_gateway_plugin_ai_rate_limiting_advanced" "my_ai_rate_limiting_advanced" {
enabled = true
config = {
strategy = "redis"
redis = {
cluster_nodes = [
{
ip = var.cluster_address
port = 6379
} ]
username = var.cluster_username
port = 6379
cloud_authentication = {
auth_provider = "azure"
azure_client_id = var.azure_client_id
azure_client_secret = var.azure_client_secret
azure_tenant_id = var.azure_tenant_id
}
}
sync_rate = 0
llm_providers = [
{
name = "cohere"
limit = [100, 1000]
window_size = [60, 3600]
} ]
request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
EOF
}
tags = []
control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
service = {
id = konnect_gateway_service.my_service.id
}
}
Copied!
This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "azure_tenant_id" {
type = string
}
Copied!
Add this section to your kong.yaml configuration file:
kong.yaml
Copied!
_format_version: "3.0"
plugins:
- name: ai-rate-limiting-advanced
route: routeName|Id
config:
strategy: redis
redis:
cluster_nodes:
- ip: ${{ env "DECK_CLUSTER_ADDRESS" }}
port: 6379
username: ${{ env "DECK_CLUSTER_USERNAME" }}
port: 6379
cloud_authentication:
auth_provider: azure
azure_client_id: ${{ env "DECK_AZURE_CLIENT_ID" }}
azure_client_secret: ${{ env "DECK_AZURE_CLIENT_SECRET" }}
azure_tenant_id: ${{ env "DECK_AZURE_TENANT_ID" }}
sync_rate: 0
llm_providers:
- name: cohere
limit:
- 100
- 1000
window_size:
- 60
- 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
Make sure to replace the following placeholders with your own values:
-
routeName|Id: Theidornameof the route the plugin configuration will target.
Make the following request:
curl -i -X POST http://localhost:8001/routes/{routeName|Id}/plugins/ \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '
{
"name": "ai-rate-limiting-advanced",
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"username": "'$CLUSTER_USERNAME'",
"port": 6379,
"cloud_authentication": {
"auth_provider": "azure",
"azure_client_id": "'$AZURE_CLIENT_ID'",
"azure_client_secret": "'$AZURE_CLIENT_SECRET'",
"azure_tenant_id": "'$AZURE_TENANT_ID'"
}
},
"sync_rate": 0,
"llm_providers": [
{
"name": "cohere",
"limit": [
100,
1000
],
"window_size": [
60,
3600
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"tags": []
}
'
Copied!
Make sure to replace the following placeholders with your own values:
-
routeName|Id: Theidornameof the route the plugin configuration will target.
Make the following request:
curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/routes/{routeId}/plugins/ \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "ai-rate-limiting-advanced",
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"username": "'$CLUSTER_USERNAME'",
"port": 6379,
"cloud_authentication": {
"auth_provider": "azure",
"azure_client_id": "'$AZURE_CLIENT_ID'",
"azure_client_secret": "'$AZURE_CLIENT_SECRET'",
"azure_tenant_id": "'$AZURE_TENANT_ID'"
}
},
"sync_rate": 0,
"llm_providers": [
{
"name": "cohere",
"limit": [
100,
1000
],
"window_size": [
60,
3600
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"tags": []
}
'
Copied!
Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
controlPlaneId: Theidof the control plane. -
routeId: Theidof the route the plugin configuration will target.
See the Konnect API reference to learn about region-specific URLs and personal access tokens.
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: ai-rate-limiting-advanced
namespace: kong
annotations:
kubernetes.io/ingress.class: kong
konghq.com/tags: ''
config:
strategy: redis
redis:
cluster_nodes:
- ip: '$CLUSTER_ADDRESS'
port: 6379
username: '$CLUSTER_USERNAME'
port: 6379
cloud_authentication:
auth_provider: azure
azure_client_id: '$AZURE_CLIENT_ID'
azure_client_secret: '$AZURE_CLIENT_SECRET'
azure_tenant_id: '$AZURE_TENANT_ID'
sync_rate: 0
llm_providers:
- name: cohere
limit:
- 100
- 1000
window_size:
- 60
- 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header('x-prompt-count'))
if header_count then
return header_count
end
return 0
plugin: ai-rate-limiting-advanced
" | kubectl apply -f -
Copied!
Next, apply the KongPlugin resource by annotating the httproute or ingress resource:
kubectl annotate -n kong httproute konghq.com/plugins=ai-rate-limiting-advanced
Copied!
kubectl annotate -n kong ingress konghq.com/plugins=ai-rate-limiting-advanced
Copied!
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}
Copied!
Add the following to your Terraform configuration to create a Konnect Gateway Plugin:
resource "konnect_gateway_plugin_ai_rate_limiting_advanced" "my_ai_rate_limiting_advanced" {
enabled = true
config = {
strategy = "redis"
redis = {
cluster_nodes = [
{
ip = var.cluster_address
port = 6379
} ]
username = var.cluster_username
port = 6379
cloud_authentication = {
auth_provider = "azure"
azure_client_id = var.azure_client_id
azure_client_secret = var.azure_client_secret
azure_tenant_id = var.azure_tenant_id
}
}
sync_rate = 0
llm_providers = [
{
name = "cohere"
limit = [100, 1000]
window_size = [60, 3600]
} ]
request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
EOF
}
tags = []
control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
route = {
id = konnect_gateway_route.my_route.id
}
}
Copied!
This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "azure_tenant_id" {
type = string
}
Copied!
Add this section to your kong.yaml configuration file:
kong.yaml
Copied!
_format_version: "3.0"
plugins:
- name: ai-rate-limiting-advanced
consumer: consumerName|Id
config:
strategy: redis
redis:
cluster_nodes:
- ip: ${{ env "DECK_CLUSTER_ADDRESS" }}
port: 6379
username: ${{ env "DECK_CLUSTER_USERNAME" }}
port: 6379
cloud_authentication:
auth_provider: azure
azure_client_id: ${{ env "DECK_AZURE_CLIENT_ID" }}
azure_client_secret: ${{ env "DECK_AZURE_CLIENT_SECRET" }}
azure_tenant_id: ${{ env "DECK_AZURE_TENANT_ID" }}
sync_rate: 0
llm_providers:
- name: cohere
limit:
- 100
- 1000
window_size:
- 60
- 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
Make sure to replace the following placeholders with your own values:
-
consumerName|Id: Theidornameof the consumer the plugin configuration will target.
Make the following request:
curl -i -X POST http://localhost:8001/consumers/{consumerName|Id}/plugins/ \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '
{
"name": "ai-rate-limiting-advanced",
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"username": "'$CLUSTER_USERNAME'",
"port": 6379,
"cloud_authentication": {
"auth_provider": "azure",
"azure_client_id": "'$AZURE_CLIENT_ID'",
"azure_client_secret": "'$AZURE_CLIENT_SECRET'",
"azure_tenant_id": "'$AZURE_TENANT_ID'"
}
},
"sync_rate": 0,
"llm_providers": [
{
"name": "cohere",
"limit": [
100,
1000
],
"window_size": [
60,
3600
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"tags": []
}
'
Copied!
Make sure to replace the following placeholders with your own values:
-
consumerName|Id: Theidornameof the consumer the plugin configuration will target.
Make the following request:
curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/consumers/{consumerId}/plugins/ \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "ai-rate-limiting-advanced",
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"username": "'$CLUSTER_USERNAME'",
"port": 6379,
"cloud_authentication": {
"auth_provider": "azure",
"azure_client_id": "'$AZURE_CLIENT_ID'",
"azure_client_secret": "'$AZURE_CLIENT_SECRET'",
"azure_tenant_id": "'$AZURE_TENANT_ID'"
}
},
"sync_rate": 0,
"llm_providers": [
{
"name": "cohere",
"limit": [
100,
1000
],
"window_size": [
60,
3600
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"tags": []
}
'
Copied!
Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
controlPlaneId: Theidof the control plane. -
consumerId: Theidof the consumer the plugin configuration will target.
See the Konnect API reference to learn about region-specific URLs and personal access tokens.
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: ai-rate-limiting-advanced
namespace: kong
annotations:
kubernetes.io/ingress.class: kong
konghq.com/tags: ''
config:
strategy: redis
redis:
cluster_nodes:
- ip: '$CLUSTER_ADDRESS'
port: 6379
username: '$CLUSTER_USERNAME'
port: 6379
cloud_authentication:
auth_provider: azure
azure_client_id: '$AZURE_CLIENT_ID'
azure_client_secret: '$AZURE_CLIENT_SECRET'
azure_tenant_id: '$AZURE_TENANT_ID'
sync_rate: 0
llm_providers:
- name: cohere
limit:
- 100
- 1000
window_size:
- 60
- 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header('x-prompt-count'))
if header_count then
return header_count
end
return 0
plugin: ai-rate-limiting-advanced
" | kubectl apply -f -
Copied!
Next, apply the KongPlugin resource by annotating the KongConsumer resource:
kubectl annotate -n kong kongconsumer CONSUMER_NAME konghq.com/plugins=ai-rate-limiting-advanced
Copied!
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}
Copied!
Add the following to your Terraform configuration to create a Konnect Gateway Plugin:
resource "konnect_gateway_plugin_ai_rate_limiting_advanced" "my_ai_rate_limiting_advanced" {
enabled = true
config = {
strategy = "redis"
redis = {
cluster_nodes = [
{
ip = var.cluster_address
port = 6379
} ]
username = var.cluster_username
port = 6379
cloud_authentication = {
auth_provider = "azure"
azure_client_id = var.azure_client_id
azure_client_secret = var.azure_client_secret
azure_tenant_id = var.azure_tenant_id
}
}
sync_rate = 0
llm_providers = [
{
name = "cohere"
limit = [100, 1000]
window_size = [60, 3600]
} ]
request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
EOF
}
tags = []
control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
consumer = {
id = konnect_gateway_consumer.my_consumer.id
}
}
Copied!
This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "azure_tenant_id" {
type = string
}
Copied!
Add this section to your kong.yaml configuration file:
kong.yaml
Copied!
_format_version: "3.0"
plugins:
- name: ai-rate-limiting-advanced
consumer_group: consumerGroupName|Id
config:
strategy: redis
redis:
cluster_nodes:
- ip: ${{ env "DECK_CLUSTER_ADDRESS" }}
port: 6379
username: ${{ env "DECK_CLUSTER_USERNAME" }}
port: 6379
cloud_authentication:
auth_provider: azure
azure_client_id: ${{ env "DECK_AZURE_CLIENT_ID" }}
azure_client_secret: ${{ env "DECK_AZURE_CLIENT_SECRET" }}
azure_tenant_id: ${{ env "DECK_AZURE_TENANT_ID" }}
sync_rate: 0
llm_providers:
- name: cohere
limit:
- 100
- 1000
window_size:
- 60
- 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
Make sure to replace the following placeholders with your own values:
-
consumerGroupName|Id: Theidornameof the consumer group the plugin configuration will target.
Make the following request:
curl -i -X POST http://localhost:8001/consumer_groups/{consumerGroupName|Id}/plugins/ \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '
{
"name": "ai-rate-limiting-advanced",
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"username": "'$CLUSTER_USERNAME'",
"port": 6379,
"cloud_authentication": {
"auth_provider": "azure",
"azure_client_id": "'$AZURE_CLIENT_ID'",
"azure_client_secret": "'$AZURE_CLIENT_SECRET'",
"azure_tenant_id": "'$AZURE_TENANT_ID'"
}
},
"sync_rate": 0,
"llm_providers": [
{
"name": "cohere",
"limit": [
100,
1000
],
"window_size": [
60,
3600
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"tags": []
}
'
Copied!
Make sure to replace the following placeholders with your own values:
-
consumerGroupName|Id: Theidornameof the consumer group the plugin configuration will target.
Make the following request:
curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/consumer_groups/{consumerGroupId}/plugins/ \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "ai-rate-limiting-advanced",
"config": {
"strategy": "redis",
"redis": {
"cluster_nodes": [
{
"ip": "'$CLUSTER_ADDRESS'",
"port": 6379
}
],
"username": "'$CLUSTER_USERNAME'",
"port": 6379,
"cloud_authentication": {
"auth_provider": "azure",
"azure_client_id": "'$AZURE_CLIENT_ID'",
"azure_client_secret": "'$AZURE_CLIENT_SECRET'",
"azure_tenant_id": "'$AZURE_TENANT_ID'"
}
},
"sync_rate": 0,
"llm_providers": [
{
"name": "cohere",
"limit": [
100,
1000
],
"window_size": [
60,
3600
]
}
],
"request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n return header_count\nend\nreturn 0\n"
},
"tags": []
}
'
Copied!
Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
controlPlaneId: Theidof the control plane. -
consumerGroupId: Theidof the consumer group the plugin configuration will target.
See the Konnect API reference to learn about region-specific URLs and personal access tokens.
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: ai-rate-limiting-advanced
namespace: kong
annotations:
kubernetes.io/ingress.class: kong
konghq.com/tags: ''
config:
strategy: redis
redis:
cluster_nodes:
- ip: '$CLUSTER_ADDRESS'
port: 6379
username: '$CLUSTER_USERNAME'
port: 6379
cloud_authentication:
auth_provider: azure
azure_client_id: '$AZURE_CLIENT_ID'
azure_client_secret: '$AZURE_CLIENT_SECRET'
azure_tenant_id: '$AZURE_TENANT_ID'
sync_rate: 0
llm_providers:
- name: cohere
limit:
- 100
- 1000
window_size:
- 60
- 3600
request_prompt_count_function: |
local header_count = tonumber(kong.request.get_header('x-prompt-count'))
if header_count then
return header_count
end
return 0
plugin: ai-rate-limiting-advanced
" | kubectl apply -f -
Copied!
Next, apply the KongPlugin resource by annotating the KongConsumerGroup resource:
kubectl annotate -n kong kongconsumergroup CONSUMERGROUP_NAME konghq.com/plugins=ai-rate-limiting-advanced
Copied!
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}
Copied!
Add the following to your Terraform configuration to create a Konnect Gateway Plugin:
resource "konnect_gateway_plugin_ai_rate_limiting_advanced" "my_ai_rate_limiting_advanced" {
enabled = true
config = {
strategy = "redis"
redis = {
cluster_nodes = [
{
ip = var.cluster_address
port = 6379
} ]
username = var.cluster_username
port = 6379
cloud_authentication = {
auth_provider = "azure"
azure_client_id = var.azure_client_id
azure_client_secret = var.azure_client_secret
azure_tenant_id = var.azure_tenant_id
}
}
sync_rate = 0
llm_providers = [
{
name = "cohere"
limit = [100, 1000]
window_size = [60, 3600]
} ]
request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
return header_count
end
return 0
EOF
}
tags = []
control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
consumer_group = {
id = konnect_gateway_consumer_group.my_consumer_group.id
}
}
Copied!
This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "azure_tenant_id" {
type = string
}
Copied!