Enable AI Consumer Group rate limiting
Protect your LLM services with AI Consumer Group rate limiting. The AI Rate Limiting Advanced Policy analyzes query costs and token responses to provide an enterprise-grade rate limiting strategy.
In this example, we’re rate limiting a specific AI Consumer Group using its name. You can also target an AI Consumer Group using its ID by changing the config.policies.match.key value.
Prerequisites
-
An existing AI Model.
-
An existing AI Consumer Group.
Environment variables
CONSUMER_GROUP_NAME: The name of the AI Consumer Group to rate limit.
ai_gateway_policies:
- ref: consumer-group-rate-limiting
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
name: consumer-group-rate-limiting
display_name: AI Consumer Group rate limiting
type: ai-rate-limiting-advanced
enabled: true
config:
policies:
- match:
- type: consumer_group
key: name
values:
- !env CONSUMER_GROUP_NAME
limits:
- limit: 100
window_size: 60
- limit: 1000
window_size: 3600
global: falseMake sure to replace the following placeholders with your own values:
-
AI_GATEWAY_ID: Theidof your AI Gateway. -
!env CONSUMER_GROUP_NAME: The name of the AI Consumer Group to rate limit.
curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "consumer-group-rate-limiting",
"display_name": "AI Consumer Group rate limiting",
"type": "ai-rate-limiting-advanced",
"enabled": true,
"config": {
"policies": [
{
"match": [
{
"type": "consumer_group",
"key": "name",
"values": [
"'$CONSUMER_GROUP_NAME'"
]
}
],
"limits": [
{
"limit": 100,
"window_size": 60
},
{
"limit": 1000,
"window_size": 3600
}
]
}
]
},
"global": false
}
'Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
AIGatewayId: Theidof the AI Gateway.
See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}Add the following to your Terraform configuration:
resource "konnect_ai_gateway_policy" "my_policy" {
name = "consumer-group-rate-limiting"
display_name = "AI Consumer Group rate limiting"
type = "ai-rate-limiting-advanced"
enabled = true
global = false
config = jsonencode({
policies = [
{
match = [
{
type = "consumer_group"
key = "name"
values = [var.consumer_group_name]
} ]
limits = [
{
limit = 100
window_size = 60
},
{
limit = 1000
window_size = 3600
} ]
} ]
})
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "consumer_group_name" {
type = string
}ai_gateway_policies:
- ref: consumer-group-rate-limiting
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
name: consumer-group-rate-limiting
display_name: AI Consumer Group rate limiting
type: ai-rate-limiting-advanced
enabled: true
config:
policies:
- match:
- type: consumer_group
key: name
values:
- !env CONSUMER_GROUP_NAME
limits:
- limit: 100
window_size: 60
- limit: 1000
window_size: 3600
global: trueMake sure to replace the following placeholders with your own values:
-
AI_GATEWAY_ID: Theidof your AI Gateway. -
!env CONSUMER_GROUP_NAME: The name of the AI Consumer Group to rate limit.
curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "consumer-group-rate-limiting",
"display_name": "AI Consumer Group rate limiting",
"type": "ai-rate-limiting-advanced",
"enabled": true,
"config": {
"policies": [
{
"match": [
{
"type": "consumer_group",
"key": "name",
"values": [
"'$CONSUMER_GROUP_NAME'"
]
}
],
"limits": [
{
"limit": 100,
"window_size": 60
},
{
"limit": 1000,
"window_size": 3600
}
]
}
]
},
"global": true
}
'Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
AIGatewayId: Theidof the AI Gateway.
See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}Add the following to your Terraform configuration:
resource "konnect_ai_gateway_policy" "my_policy" {
name = "consumer-group-rate-limiting"
display_name = "AI Consumer Group rate limiting"
type = "ai-rate-limiting-advanced"
enabled = true
global = true
config = jsonencode({
policies = [
{
match = [
{
type = "consumer_group"
key = "name"
values = [var.consumer_group_name]
} ]
limits = [
{
limit = 100
window_size = 60
},
{
limit = 1000
window_size = 3600
} ]
} ]
})
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "consumer_group_name" {
type = string
}After creating the policy, add consumer-group-rate-limiting to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
ai_gateway_models:
- ref: my-gpt-4o
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
display_name: my-gpt-4o
name: my-gpt-4o
type: model
capabilities:
- generate
formats:
- type: openai
targets:
- name: gpt-4o
provider: my-openai-account
config:
type: openai
config:
route:
paths:
- "/v1"
model:
body_param: model
values:
- my-gpt-4o
logging:
payloads: false
policies:
- !ref consumer-group-rate-limiting#nameAfter creating the policy, add consumer-group-rate-limiting to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/models/my-gpt-4o \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"display_name": "my-gpt-4o",
"name": "my-gpt-4o",
"type": "model",
"capabilities": [
"generate"
],
"formats": [
{
"type": "openai"
}
],
"policies": [
"consumer-group-rate-limiting"
],
"targets": [
{
"name": "gpt-4o",
"provider": "my-openai-account",
"config": {
"type": "openai"
}
}
],
"config": {
"route": {
"paths": [
"/v1"
],
"model": {
"body_param": "model",
"values": [
"my-gpt-4o"
]
}
},
"logging": {
"payloads": false
}
}
}
'After creating the policy, add consumer-group-rate-limiting to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
resource "konnect_ai_gateway_model" "my_model" {
display_name = "my-gpt-4o"
name = "my-gpt-4o"
type = "model"
capabilities = ["generate"]
formats = [
{
type = "openai"
} ]
targets = [
{
name = "gpt-4o"
provider = "my-openai-account"
config = {
type = "openai"
}
} ]
config = jsonencode({
route = {
paths = ["/v1"]
model = {
body_param = "model"
values = ["my-gpt-4o"]
}
}
logging = {
payloads = false
}
})
policies = [konnect_ai_gateway_policy.my_policy.name]
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "consumer_group_name" {
type = string
}