Enable rate limiting with multiple conditions
Protect your LLM services with rate limiting based on multiple conditions. The AI Rate Limiting Advanced Policy analyzes query costs and token responses to provide an enterprise-grade rate limiting strategy.
In this example, we’re enabling rate limiting for a specific AI Consumer using the OpenAI provider. The limits specified only apply when both conditions are met. When the limit is reached, the same AI Consumer may be able to send requests to a different provider.
Prerequisites
-
An existing AI Model.
-
An existing AI Consumer.
Environment variables
CONSUMER_USERNAME: The username of the AI Consumer to rate limit.
ai_gateway_policies:
- ref: rate-limiting-multiple-conditions
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
name: rate-limiting-multiple-conditions
display_name: Rate limiting with multiple conditions
type: ai-rate-limiting-advanced
enabled: true
config:
policies:
- match:
- type: provider
partition_by: true
values:
- openai
- type: consumer
key: username
values:
- !env CONSUMER_USERNAME
limits:
- limit: 100
window_size: 60
- limit: 1000
window_size: 3600
global: falseMake sure to replace the following placeholders with your own values:
-
AI_GATEWAY_ID: Theidof your AI Gateway. -
!env CONSUMER_USERNAME: The username of the AI Consumer to rate limit.
curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "rate-limiting-multiple-conditions",
"display_name": "Rate limiting with multiple conditions",
"type": "ai-rate-limiting-advanced",
"enabled": true,
"config": {
"policies": [
{
"match": [
{
"type": "provider",
"partition_by": true,
"values": [
"openai"
]
},
{
"type": "consumer",
"key": "username",
"values": [
"'$CONSUMER_USERNAME'"
]
}
],
"limits": [
{
"limit": 100,
"window_size": 60
},
{
"limit": 1000,
"window_size": 3600
}
]
}
]
},
"global": false
}
'Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
AIGatewayId: Theidof the AI Gateway.
See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}Add the following to your Terraform configuration:
resource "konnect_ai_gateway_policy" "my_policy" {
name = "rate-limiting-multiple-conditions"
display_name = "Rate limiting with multiple conditions"
type = "ai-rate-limiting-advanced"
enabled = true
global = false
config = jsonencode({
policies = [
{
match = [
{
type = "provider"
partition_by = true
values = ["openai"]
},
{
type = "consumer"
key = "username"
values = [var.consumer_username]
} ]
limits = [
{
limit = 100
window_size = 60
},
{
limit = 1000
window_size = 3600
} ]
} ]
})
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "consumer_username" {
type = string
}ai_gateway_policies:
- ref: rate-limiting-multiple-conditions
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
name: rate-limiting-multiple-conditions
display_name: Rate limiting with multiple conditions
type: ai-rate-limiting-advanced
enabled: true
config:
policies:
- match:
- type: provider
partition_by: true
values:
- openai
- type: consumer
key: username
values:
- !env CONSUMER_USERNAME
limits:
- limit: 100
window_size: 60
- limit: 1000
window_size: 3600
global: trueMake sure to replace the following placeholders with your own values:
-
AI_GATEWAY_ID: Theidof your AI Gateway. -
!env CONSUMER_USERNAME: The username of the AI Consumer to rate limit.
curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "rate-limiting-multiple-conditions",
"display_name": "Rate limiting with multiple conditions",
"type": "ai-rate-limiting-advanced",
"enabled": true,
"config": {
"policies": [
{
"match": [
{
"type": "provider",
"partition_by": true,
"values": [
"openai"
]
},
{
"type": "consumer",
"key": "username",
"values": [
"'$CONSUMER_USERNAME'"
]
}
],
"limits": [
{
"limit": 100,
"window_size": 60
},
{
"limit": 1000,
"window_size": 3600
}
]
}
]
},
"global": true
}
'Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
AIGatewayId: Theidof the AI Gateway.
See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}Add the following to your Terraform configuration:
resource "konnect_ai_gateway_policy" "my_policy" {
name = "rate-limiting-multiple-conditions"
display_name = "Rate limiting with multiple conditions"
type = "ai-rate-limiting-advanced"
enabled = true
global = true
config = jsonencode({
policies = [
{
match = [
{
type = "provider"
partition_by = true
values = ["openai"]
},
{
type = "consumer"
key = "username"
values = [var.consumer_username]
} ]
limits = [
{
limit = 100
window_size = 60
},
{
limit = 1000
window_size = 3600
} ]
} ]
})
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "consumer_username" {
type = string
}After creating the policy, add rate-limiting-multiple-conditions to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
ai_gateway_models:
- ref: my-gpt-4o
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
display_name: my-gpt-4o
name: my-gpt-4o
type: model
capabilities:
- generate
formats:
- type: openai
targets:
- name: gpt-4o
provider: my-openai-account
config:
type: openai
config:
route:
paths:
- "/v1"
model:
body_param: model
values:
- my-gpt-4o
logging:
payloads: false
policies:
- !ref rate-limiting-multiple-conditions#nameAfter creating the policy, add rate-limiting-multiple-conditions to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/models/my-gpt-4o \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"display_name": "my-gpt-4o",
"name": "my-gpt-4o",
"type": "model",
"capabilities": [
"generate"
],
"formats": [
{
"type": "openai"
}
],
"policies": [
"rate-limiting-multiple-conditions"
],
"targets": [
{
"name": "gpt-4o",
"provider": "my-openai-account",
"config": {
"type": "openai"
}
}
],
"config": {
"route": {
"paths": [
"/v1"
],
"model": {
"body_param": "model",
"values": [
"my-gpt-4o"
]
}
},
"logging": {
"payloads": false
}
}
}
'After creating the policy, add rate-limiting-multiple-conditions to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
resource "konnect_ai_gateway_model" "my_model" {
display_name = "my-gpt-4o"
name = "my-gpt-4o"
type = "model"
capabilities = ["generate"]
formats = [
{
type = "openai"
} ]
targets = [
{
name = "gpt-4o"
provider = "my-openai-account"
config = {
type = "openai"
}
} ]
config = jsonencode({
route = {
paths = ["/v1"]
model = {
body_param = "model"
values = ["my-gpt-4o"]
}
}
logging = {
payloads = false
}
})
policies = [konnect_ai_gateway_policy.my_policy.name]
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "consumer_username" {
type = string
}