Kong authenticates the AI Consumer, checks the token window, and forwards the request to OpenAI. When Kong sends the response, the Post-Function Policy reads the rate limit headers and, at 80% used, starts a timer that sends the alert, so the client gets the response without waiting for the webhook.
sequenceDiagram
participant C as Client
participant K as Kong AI Gateway
participant O as OpenAI
participant W as Webhook
C->>K: POST /ai-rate-limit-alerts/chat/completions (apikey, model chat)
activate K
K->>K: Key Auth AI Auth Strategy identifies the AI Consumer (else 401)
K->>K: AI Rate Limiting Advanced Policy checks the token window (else 429)
K->>O: Request with the OpenAI key
activate O
O-->>K: Completion and token usage
deactivate O
K-->>C: 200 with the rate limit headers
K->>K: Post-Function Policy computes the used share from the headers
K->>W: Timer sends the JSON alert at 80% used, once in 60 seconds
deactivate KAuthentication is an AI Auth Strategy, not an AI Policy. The Key Auth AI Auth Strategy reads the apikey header and matches the key against the AI Consumer credentials. On a match, Kong attaches the AI Consumer to the request. The AI Consumer is the identity that rate limits, ACLs, and analytics use, and the Post-Function Policy puts its name in the alert.
The following settings configure the Key Auth AI Auth Strategy and the AI Consumer:
|
Setting
|
Purpose
|
key_names: [apikey]
|
Kong reads the key from the apikey header. Key Auth compares the whole header value, so it can’t read a key from an Authorization: Bearer header. The note in Run the demo shows the workaround.
|
hide_credentials: true
|
Kong removes the key before it forwards the request. OpenAI never sees the AI Consumer key.
|
credentials[].api_key
|
The AI Consumer’s key. The field is write-only, so it takes a !secret reference to CONSUMER_API_KEY, even for a demo value.
|
Kong adds X-Consumer-Username and related headers to the upstream request, not to the response. Per-consumer usage appears in Konnect analytics.
The AI Rate Limiting Advanced Policy counts prompt and completion tokens, the unit that drives LLM cost. It allows 50 tokens in a 60-second window, which the demo uses up in a few requests. Kong counts a request’s tokens after OpenAI answers, so concurrent requests can go over the limit. A request whose prompt is larger than the tokens left gets 429 before Kong calls OpenAI.
The following settings configure the AI Rate Limiting Advanced Policy:
|
Setting
|
Purpose
|
config.policies[].limits
|
The token limit and the window size in seconds: 50 and 60.
|
window_type: fixed
|
Assigns each request to one 60-second bucket, so the quota restores at the end of the window. A sliding window weighs the current window against the previous one.
|
identifier: consumer
|
Counts usage for each AI Consumer. In a test on runtime 2.3.1, two AI Consumers that called from the same machine shared one counter. This cookbook hasn’t confirmed separate budgets for AI Consumers on different machines.
|
tokens_count_strategy: total_tokens
|
Counts prompt and completion tokens. Other values count only prompt tokens, only completion tokens, or cost.
|
hide_client_headers: false
|
Kong adds the limit and remaining headers to each response. The Post-Function Policy reads these headers, so with true, no alert is sent.
|
strategy: local
|
Keeps the counters in memory on each data plane node. For several nodes, use redis so that the nodes share the counters.
|
The Post-Function Policy runs Lua code from its config.body_filter list while Kong sends the response body. It’s listed on the AI Model, so it runs after authentication, and kong.client.get_consumer() returns the AI Consumer. The alert carries only counters and the AI Consumer name. Don’t add prompts, responses, or keys to it, because the webhook receiver is outside Kong.
The following settings configure the Post-Function Policy:
|
Setting
|
Purpose
|
body_filter
|
Runs the code for each chunk of the response body. if not ngx.arg[2] returns early until the last chunk, so the code runs once for each response, after Kong has set the rate limit headers.
|
X-AI-RateLimit-Remaining-* and X-AI-RateLimit-Limit-*
|
The code reads both headers with any suffix and skips the reset and retry headers. The suffix depends on the window and the policy index, for example -minute-policy-1.
|
local threshold = 80
|
The share of the limit, in percent, that triggers the alert. The code computes (limit - remaining) / limit * 100.
|
ngx.shared.kong and dict:add(..., 1, 60)
|
Stores a key for each AI Consumer for 60 seconds. add fails when the key exists, so the code sends at most one alert for each AI Consumer in 60 seconds on each data plane node.
|
os.getenv("ALERT_WEBHOOK_URL")
|
Reads the webhook URL from the data plane environment. If the variable is empty, the code sends no alert.
|
ngx.timer.at(0, ...)
|
Runs the POST in a zero-delay timer. OpenResty disables network sockets in the body filter phase, and recommends a zero-delay timer when the request doesn’t need the result (OpenResty documentation).
|
resty.http request_uri
|
Sends the alert as a JSON POST with a 3-second timeout and TLS certificate verification (ssl_verify = true).
|
|
Alert fields
|
text (a summary), username (the AI Consumer name), remaining, limit, rate (the tokens used), threshold, and window.
|
KONG_UNTRUSTED_LUA=on turns off the Lua sandbox on the data plane. Lua code in every AI Policy on that data plane, not only this Post-Function Policy, then has complete access to the environment and to all modules, with no protections (sandboxing). Use this setting only for this local demo, and don’t use it on a data plane that serves other traffic.
The AI Model Provider holds the OpenAI API key and adds it to each request to OpenAI. The AI Model ai-rate-limit-alerts-model accepts OpenAI-format requests on /ai-rate-limit-alerts, answers to the chat alias, and lists both AI Policies.
The following settings configure the AI Model Provider and the AI Model:
|
Setting
|
Purpose
|
config.auth (AI Model Provider)
|
The OpenAI API key, from OPENAI_TOKEN. Kong adds it to each request to OpenAI. The field is write-only, so Konnect never returns it.
|
config.route.model
|
Kong reads the model field of the request body (body_param: model) and matches it against chat. Another value, including gpt-4o-mini, matches no AI Model and returns 500 with failed the initial dns/balancer resolve for 'ai-gateway.upstream.local'.
|
policies
|
Attaches the AI Rate Limiting Advanced Policy and the Post-Function Policy to the AI Model. A Post-Function Policy runs after the other AI Policies in each phase.
|
access.auth_strategies
|
Each request must have a valid AI Consumer key.
|
formats: [{type: openai}] and capabilities: [generate]
|
The AI Model accepts OpenAI-format chat completion requests.
|
config.route.paths
|
/ai-rate-limit-alerts. The generate capability adds /chat/completions, so the endpoint is /ai-rate-limit-alerts/chat/completions.
|
targets[].name
|
The OpenAI model, from CHAT_MODEL.
|
config.route.protocols: [http, https]
|
Accepts HTTP, so you can call the local data plane at http://localhost:8000.
|
config.route.strip_path: true
|
Kong removes /ai-rate-limit-alerts from the path before it forwards the request.
|
config.max_request_body_size: 10485760
|
Accepts request bodies up to 10 MB, instead of the 8 MB default.
|
config.response_streaming: allow
|
Clients can request streamed responses.
|
config.logging.payloads: true
|
Kong logs request and response payloads. The default is false.
|
Production credentials: This cookbook stores the AI Consumer’s API key and the OpenAI API key in environment variables for simplicity. In production, use AI Vaults to reference them from a secret manager, for example AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, or Azure Key Vault.
The AI Rate Limiting Advanced Policy adds the following headers to each response:
|
Header
|
Description
|
X-AI-RateLimit-Limit-minute-policy-1
|
The token limit for the 60-second window of the first entry in config.policies.
|
X-AI-RateLimit-Remaining-minute-policy-1
|
The tokens left in the current window. The demo and the Post-Function Policy compute the used share from this header and the limit header.
|
X-AI-RateLimit-Reset
|
The seconds until the quota is restored.
|
X-AI-RateLimit-Retry-After
|
On a 429 response, the seconds until the AI Consumer can send requests again. There is no plain Retry-After header.
|