AI Rate Limit Alerts

Minimum Version
AI Gateway - 2.0
Incompatible with
on-prem

Overview

Send an alert to a webhook when an AI Consumer uses 80% of its token quota, so that you can act before Kong AI Gateway rejects requests with HTTP 429. You’ll set up:

  • A /ai-rate-limit-alerts chat endpoint with a token quota that the AI Rate Limiting Advanced Policy enforces
  • A Post-Function Policy that sends a JSON alert to a webhook at 80% of the quota, while requests still succeed
  • AI Consumer authentication, with the OpenAI API key kept in AI Gateway

Prerequisites

This tutorial uses Kong Konnect. The AI Gateway quickstart script provisions a recipe-scoped AI Gateway and local data plane.

  1. Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
  2. Export your token. You reuse the same token later for kongctl commands:

    export KONNECT_TOKEN='YOUR_KONNECT_PAT'
  3. Open webhook.site and copy your unique URL. The data plane sends the alert to this URL and reads it when the data plane starts, so export it before you run the quickstart script:

    export ALERT_WEBHOOK_URL='https://webhook.site/YOUR-UUID'
  4. Set the recipe-scoped AI Gateway name and run the quickstart script. The three -e settings let the Post-Function Policy send the alert:

    • KONG_UNTRUSTED_LUA=on turns off the Lua sandbox (untrusted_lua). The policy uses ngx.timer.at, ngx.shared, and os.getenv, which the sandbox doesn’t allow in the default strict mode or in lax mode. It also loads resty.http, which only lax mode allows. See the warning in Post-Function Policy.
    • ALERT_WEBHOOK_URL passes the webhook URL from your shell to the data plane container.
    • KONG_NGINX_MAIN_ENV=ALERT_WEBHOOK_URL adds the env ALERT_WEBHOOK_URL; directive to the NGINX main context (NGINX injected directives). NGINX removes inherited environment variables unless an env directive keeps them (NGINX documentation), so the policy can read the URL with os.getenv only with this setting.
    export KONNECT_CONTROL_PLANE_NAME='ai-rate-limit-alerts-recipe'
    curl -Ls https://get.konghq.com/ai | \
      bash -s -- -k $KONNECT_TOKEN \
        -e KONG_UNTRUSTED_LUA=on \
        -e ALERT_WEBHOOK_URL \
        -e KONG_NGINX_MAIN_ENV=ALERT_WEBHOOK_URL

    This provisions an AI Gateway named ai-rate-limit-alerts-recipe, a local data plane connected to it, and prints export lines for the rest of the session vars, including AI_GATEWAY_ID. Paste those into your shell when prompted.

This tutorial uses kongctl to manage AI Gateway configuration.

  1. Install kongctl from the kongctl documentation.
  2. Verify it’s installed:

    kongctl version

Export the credentials of the provider that the AI Model calls. The provider’s tab in Apply the Kong configuration reads these variables.

The demo script requires Python 3.11 or later. Set up an isolated environment:

python3 -m venv .venv
source .venv/bin/activate
pip install 'openai>=1.0.0'

What this cookbook solves

Rate limits usually tell clients about a quota only when a request fails with 429, which causes the following problems:

Problem

Solution

A limit on the request count doesn’t follow LLM cost. A short greeting and a long RAG prompt each count as one request, but the provider bills tokens. The AI Rate Limiting Advanced Policy counts the tokens that OpenAI reports for each request and returns 429 when the window has no tokens left.
A 429 is a hard stop. Operations and finance teams learn about the quota only after callers fail, too late to raise the quota or shed load. A Post-Function Policy sends a JSON alert to a webhook when 80% of the window is used. The request that crosses the threshold still gets 200.
Each client must read the rate limit headers and repeat the same threshold logic in each language. The Post-Function Policy reads the rate limit headers on the data plane, so clients don’t change.
An alert on every request after the threshold floods the channel. The Post-Function Policy sends at most one alert for each AI Consumer in 60 seconds.

Try it out

Apply the configuration, then run a demo script that sends short chat requests until the token quota runs out, checks that the webhook received one alert, and sends one request with an invalid API key.

Apply the Kong configuration

The configuration creates an AI Model Provider with your OpenAI API key, a Key Auth AI Auth Strategy, an AI Consumer with a demo API key, an AI Rate Limiting Advanced Policy with a 50-token window, a Post-Function Policy that sends the alert, and an AI Model on /ai-rate-limit-alerts that answers to the chat alias.

Adopt the quickstart AI Gateway into a kongctl namespace. Adoption adds the KONGCTL-namespace label to the AI Gateway:

kongctl adopt ai-gateway "${KONNECT_CONTROL_PLANE_NAME}" \
  --namespace "${KONNECT_CONTROL_PLANE_NAME}" \
  --pat "${KONNECT_TOKEN}"

Export the demo AI Consumer’s API key. The credential’s api_key field is write-only, so its value must come from an environment variable, even for a fixed demo key:

export CONSUMER_API_KEY='demo-api-key'

Select your provider. The tab exports CHAT_MODEL, the OpenAI model that the AI Model calls, then applies the configuration. The provider credentials and the webhook URL come from the prerequisites.

Run the demo

The script sends hi chat requests with max_tokens=1 until the 50-token window is empty, and prints the remaining tokens from the rate limit headers of each response. After the 429, it checks the webhook.site inbox for the alert, then sends one request with an invalid API key. In the output, look for the [THRESHOLD] line, the [BLOCKED] 429 line, and the [WEBHOOK] line.

The demo passes the API key in default_headers because the OpenAI SDK always sends api_key as an Authorization: Bearer header. To let clients use api_key directly, add a global Pre-Function Policy (global: true) that copies the Bearer token to the apikey header. A global Pre-Function Policy runs before authentication, while one attached to the AI Model runs after it.

Create the demo script:

cat <<'EOF' > demo.py
"""AI rate limit early-alert demo for Kong AI Gateway.

Sends short chat completions through Kong until the AI Consumer uses its
50-token, 60-second window. At 80% used, the Post-Function Policy sends a JSON
alert to ALERT_WEBHOOK_URL. When the window is empty, Kong returns 429. Then the
script sends one request with an invalid API key.

Expected output:
  - Early requests: [ALLOWED] 200, with the remaining tokens going down
  - At 80% used: [ALLOWED] 200 and [THRESHOLD], and the webhook inbox grows by one
  - After the quota: [BLOCKED] 429
  - Invalid API key: [BLOCKED] 401

Run: python demo.py (set PROXY_URL if the data plane is not at http://localhost:8000,
and CONSUMER_API_KEY if you changed the demo key). ALERT_WEBHOOK_URL must be your
webhook.site URL.
"""

from __future__ import annotations

import json
import os
import sys
import time
import urllib.error
import urllib.parse
import urllib.request

from openai import APIStatusError, OpenAI

PROXY_URL = os.getenv("PROXY_URL", "http://localhost:8000")
API_KEY = os.getenv("CONSUMER_API_KEY", "demo-api-key")
WEBHOOK_URL = os.getenv("ALERT_WEBHOOK_URL", "")
MODEL_ALIAS = "chat"
LIMIT = 50
THRESHOLD_PCT = 80

_USE_COLOR = sys.stdout.isatty() and "NO_COLOR" not in os.environ


def _c(code: str, s: str) -> str:
    return f"\033[{code}m{s}\033[0m" if _USE_COLOR else s


def BOLD(s: str) -> str:
    return _c("1", s)


def DIM(s: str) -> str:
    return _c("2", s)


def GREEN(s: str) -> str:
    return _c("32;1", s)


def RED(s: str) -> str:
    return _c("31;1", s)


def YELLOW(s: str) -> str:
    return _c("33;1", s)


def BLUE(s: str) -> str:
    return _c("34;1", s)


def remaining_from(headers: dict[str, str]) -> tuple[int | None, int | None]:
    rem = lim = None
    for raw_name, value in headers.items():
        name = raw_name.lower()
        if name.startswith("x-ai-ratelimit-remaining-") and "reset" not in name and "retry" not in name:
            rem = int(float(value))
        elif name.startswith("x-ai-ratelimit-limit-") and "reset" not in name and "retry" not in name:
            lim = int(float(value))
    return rem, lim


def webhook_token(url: str) -> str:
    path = urllib.parse.urlparse(url).path.strip("/")
    return path.split("/")[0] if path else ""


def webhook_count(token: str) -> int:
    if not token:
        return 0
    req = urllib.request.Request(
        f"https://webhook.site/token/{token}/requests?sorting=newest&per_page=20",
        headers={"Accept": "application/json"},
    )
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            body = json.load(resp)
    except (urllib.error.URLError, json.JSONDecodeError, TimeoutError):
        return 0
    return len(body.get("data") or [])


def make_client(api_key: str) -> OpenAI:
    return OpenAI(
        base_url=f"{PROXY_URL}/ai-rate-limit-alerts",
        api_key="unused",
        default_headers={"apikey": api_key},
    )


def chat(client: OpenAI):
    return client.chat.completions.with_raw_response.create(
        model=MODEL_ALIAS,
        messages=[{"role": "user", "content": "hi"}],
        max_tokens=1,
    )


def main() -> int:
    if not WEBHOOK_URL:
        print(RED("Set ALERT_WEBHOOK_URL to your webhook.site unique URL."))
        return 1

    client = make_client(API_KEY)
    token = webhook_token(WEBHOOK_URL)
    before = webhook_count(token)
    print(BOLD("[SETUP]"))
    print(f"  {DIM(f'proxy={PROXY_URL}/ai-rate-limit-alerts limit={LIMIT} threshold={THRESHOLD_PCT}%')}")
    print(f"  {DIM(f'webhook inbox before={before}')}")

    got_429 = False
    for i in range(1, 20):
        print(f"\n{BOLD('[REQUEST]')} chat {i}")
        try:
            raw = chat(client)
            headers = {k.lower(): v for k, v in raw.headers.items()}
            rem, lim = remaining_from(headers)
            used = (lim - rem) if rem is not None and lim else None
            pct = int(used / lim * 100) if used is not None and lim else None
            rem_s = f"{rem}/{lim}" if rem is not None and lim is not None else "n/a"
            used_s = f"{pct}%" if pct is not None else "n/a"
            remaining_s = f"remaining={rem_s}"
            print(f"  {GREEN('[ALLOWED] 200')} {BLUE(remaining_s)} {DIM('used=' + used_s)}")
            if pct is not None and pct >= THRESHOLD_PCT:
                print(f"  {YELLOW(f'[THRESHOLD] {THRESHOLD_PCT}% reached')}")
        except APIStatusError as err:
            print(f"  {RED(f'[BLOCKED] {err.status_code}')} {DIM(str(err.message)[:120])}")
            if err.status_code == 429:
                got_429 = True
                break

    hit = False
    after = before
    for _ in range(15):
        after = webhook_count(token)
        if after > before:
            hit = True
            break
        time.sleep(1)

    if hit:
        print(f"\n{GREEN('[WEBHOOK]')} inbox grew {before} -> {after}")
    else:
        print(f"\n{RED('[WEBHOOK]')} no new POST at {WEBHOOK_URL}")
        return 1

    if not got_429:
        print(RED("[FAIL] expected HTTP 429 after the token quota"))
        return 1

    bad_key = "not-a-real-key"
    print(f"\n{BOLD('[REQUEST]')} invalid API key")
    print(f"  Trying apikey={bad_key!r}")
    try:
        chat(make_client(bad_key))
        print(f"  {RED('[ALLOWED]')} Kong accepted the invalid key")
        return 1
    except APIStatusError as err:
        print(f"  {RED(f'[BLOCKED] {err.status_code}')}")

    print(f"\n{GREEN('[OK]')} early alert fired before the 429")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
EOF

Run it:

python demo.py

Example output:

[SETUP]
  proxy=http://localhost:8000/ai-rate-limit-alerts limit=50 threshold=80%
  webhook inbox before=0

[REQUEST] chat 1
  [ALLOWED] 200 remaining=48/50 used=4%

[REQUEST] chat 2
  [ALLOWED] 200 remaining=39/50 used=22%

[REQUEST] chat 3
  [ALLOWED] 200 remaining=30/50 used=40%

[REQUEST] chat 4
  [ALLOWED] 200 remaining=21/50 used=57%

[REQUEST] chat 5
  [ALLOWED] 200 remaining=12/50 used=76%

[REQUEST] chat 6
  [ALLOWED] 200 remaining=3/50 used=94%
  [THRESHOLD] 80% reached

[REQUEST] chat 7
  [BLOCKED] 429 Error code: 429 - AI token rate limit exceeded for provider(s):

[WEBHOOK] inbox grew 0 -> 1

[REQUEST] invalid API key
  Trying apikey='not-a-real-key'
  [BLOCKED] 401

[OK] early alert fired before the 429

The remaining values depend on how many tokens OpenAI counts for each request. The webhook.site inbox shows one POST with a JSON body like the following:

{
  "text": "ai-rate-limit-alerts-consumer reached 80% of the token quota. 3 tokens remaining.",
  "username": "ai-rate-limit-alerts-consumer",
  "remaining": 3,
  "limit": 50,
  "rate": 47,
  "threshold": 80,
  "window": "minute"
}

Check the results

The output confirms the following:

Behavior

Evidence

Token counting remaining goes down by the tokens of each request, 9 for each hi in this run, not by one for each request.
Alert before the quota runs out Chat 6 reaches 94% used and still gets 200. The [WEBHOOK] line shows that webhook.site received a POST.
Hard limit Chat 7 gets 429 from the AI Rate Limiting Advanced Policy, because the window has fewer tokens left than the request needs.
Authentication before the quota The invalid key gets 401, and the request never reaches OpenAI.

To review the created entities, and per-consumer usage, open the ai-rate-limit-alerts-recipe AI Gateway in Konnect. For traffic across all your AI Gateways, use the Observability menu.

How it works

Kong authenticates the AI Consumer, checks the token window, and forwards the request to OpenAI. When Kong sends the response, the Post-Function Policy reads the rate limit headers and, at 80% used, starts a timer that sends the alert, so the client gets the response without waiting for the webhook.

sequenceDiagram
    participant C as Client
    participant K as Kong AI Gateway
    participant O as OpenAI
    participant W as Webhook

    C->>K: POST /ai-rate-limit-alerts/chat/completions (apikey, model chat)
    activate K
    K->>K: Key Auth AI Auth Strategy identifies the AI Consumer (else 401)
    K->>K: AI Rate Limiting Advanced Policy checks the token window (else 429)
    K->>O: Request with the OpenAI key
    activate O
    O-->>K: Completion and token usage
    deactivate O
    K-->>C: 200 with the rate limit headers
    K->>K: Post-Function Policy computes the used share from the headers
    K->>W: Timer sends the JSON alert at 80% used, once in 60 seconds
    deactivate K

Key Auth AI Auth Strategy and AI Consumer

Authentication is an AI Auth Strategy, not an AI Policy. The Key Auth AI Auth Strategy reads the apikey header and matches the key against the AI Consumer credentials. On a match, Kong attaches the AI Consumer to the request. The AI Consumer is the identity that rate limits, ACLs, and analytics use, and the Post-Function Policy puts its name in the alert.

The following settings configure the Key Auth AI Auth Strategy and the AI Consumer:

Setting

Purpose

key_names: [apikey] Kong reads the key from the apikey header. Key Auth compares the whole header value, so it can’t read a key from an Authorization: Bearer header. The note in Run the demo shows the workaround.
hide_credentials: true Kong removes the key before it forwards the request. OpenAI never sees the AI Consumer key.
credentials[].api_key The AI Consumer’s key. The field is write-only, so it takes a !secret reference to CONSUMER_API_KEY, even for a demo value.

Kong adds X-Consumer-Username and related headers to the upstream request, not to the response. Per-consumer usage appears in Konnect analytics.

AI Rate Limiting Advanced Policy

The AI Rate Limiting Advanced Policy counts prompt and completion tokens, the unit that drives LLM cost. It allows 50 tokens in a 60-second window, which the demo uses up in a few requests. Kong counts a request’s tokens after OpenAI answers, so concurrent requests can go over the limit. A request whose prompt is larger than the tokens left gets 429 before Kong calls OpenAI.

The following settings configure the AI Rate Limiting Advanced Policy:

Setting

Purpose

config.policies[].limits The token limit and the window size in seconds: 50 and 60.
window_type: fixed Assigns each request to one 60-second bucket, so the quota restores at the end of the window. A sliding window weighs the current window against the previous one.
identifier: consumer Counts usage for each AI Consumer. In a test on runtime 2.3.1, two AI Consumers that called from the same machine shared one counter. This cookbook hasn’t confirmed separate budgets for AI Consumers on different machines.
tokens_count_strategy: total_tokens Counts prompt and completion tokens. Other values count only prompt tokens, only completion tokens, or cost.
hide_client_headers: false Kong adds the limit and remaining headers to each response. The Post-Function Policy reads these headers, so with true, no alert is sent.
strategy: local Keeps the counters in memory on each data plane node. For several nodes, use redis so that the nodes share the counters.

Post-Function Policy

The Post-Function Policy runs Lua code from its config.body_filter list while Kong sends the response body. It’s listed on the AI Model, so it runs after authentication, and kong.client.get_consumer() returns the AI Consumer. The alert carries only counters and the AI Consumer name. Don’t add prompts, responses, or keys to it, because the webhook receiver is outside Kong.

The following settings configure the Post-Function Policy:

Setting

Purpose

body_filter Runs the code for each chunk of the response body. if not ngx.arg[2] returns early until the last chunk, so the code runs once for each response, after Kong has set the rate limit headers.
X-AI-RateLimit-Remaining-* and X-AI-RateLimit-Limit-* The code reads both headers with any suffix and skips the reset and retry headers. The suffix depends on the window and the policy index, for example -minute-policy-1.
local threshold = 80 The share of the limit, in percent, that triggers the alert. The code computes (limit - remaining) / limit * 100.
ngx.shared.kong and dict:add(..., 1, 60) Stores a key for each AI Consumer for 60 seconds. add fails when the key exists, so the code sends at most one alert for each AI Consumer in 60 seconds on each data plane node.
os.getenv("ALERT_WEBHOOK_URL") Reads the webhook URL from the data plane environment. If the variable is empty, the code sends no alert.
ngx.timer.at(0, ...) Runs the POST in a zero-delay timer. OpenResty disables network sockets in the body filter phase, and recommends a zero-delay timer when the request doesn’t need the result (OpenResty documentation).
resty.http request_uri Sends the alert as a JSON POST with a 3-second timeout and TLS certificate verification (ssl_verify = true).
Alert fields text (a summary), username (the AI Consumer name), remaining, limit, rate (the tokens used), threshold, and window.

KONG_UNTRUSTED_LUA=on turns off the Lua sandbox on the data plane. Lua code in every AI Policy on that data plane, not only this Post-Function Policy, then has complete access to the environment and to all modules, with no protections (sandboxing). Use this setting only for this local demo, and don’t use it on a data plane that serves other traffic.

AI Model Provider and AI Model

The AI Model Provider holds the OpenAI API key and adds it to each request to OpenAI. The AI Model ai-rate-limit-alerts-model accepts OpenAI-format requests on /ai-rate-limit-alerts, answers to the chat alias, and lists both AI Policies.

The following settings configure the AI Model Provider and the AI Model:

Setting

Purpose

config.auth (AI Model Provider) The OpenAI API key, from OPENAI_TOKEN. Kong adds it to each request to OpenAI. The field is write-only, so Konnect never returns it.
config.route.model Kong reads the model field of the request body (body_param: model) and matches it against chat. Another value, including gpt-4o-mini, matches no AI Model and returns 500 with failed the initial dns/balancer resolve for 'ai-gateway.upstream.local'.
policies Attaches the AI Rate Limiting Advanced Policy and the Post-Function Policy to the AI Model. A Post-Function Policy runs after the other AI Policies in each phase.
access.auth_strategies Each request must have a valid AI Consumer key.
formats: [{type: openai}] and capabilities: [generate] The AI Model accepts OpenAI-format chat completion requests.
config.route.paths /ai-rate-limit-alerts. The generate capability adds /chat/completions, so the endpoint is /ai-rate-limit-alerts/chat/completions.
targets[].name The OpenAI model, from CHAT_MODEL.
config.route.protocols: [http, https] Accepts HTTP, so you can call the local data plane at http://localhost:8000.
config.route.strip_path: true Kong removes /ai-rate-limit-alerts from the path before it forwards the request.
config.max_request_body_size: 10485760 Accepts request bodies up to 10 MB, instead of the 8 MB default.
config.response_streaming: allow Clients can request streamed responses.
config.logging.payloads: true Kong logs request and response payloads. The default is false.

Production credentials: This cookbook stores the AI Consumer’s API key and the OpenAI API key in environment variables for simplicity. In production, use AI Vaults to reference them from a secret manager, for example AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, or Azure Key Vault.

Response headers

The AI Rate Limiting Advanced Policy adds the following headers to each response:

Header

Description

X-AI-RateLimit-Limit-minute-policy-1 The token limit for the 60-second window of the first entry in config.policies.
X-AI-RateLimit-Remaining-minute-policy-1 The tokens left in the current window. The demo and the Post-Function Policy compute the used share from this header and the limit header.
X-AI-RateLimit-Reset The seconds until the quota is restored.
X-AI-RateLimit-Retry-After On a 429 response, the seconds until the AI Consumer can send requests again. There is no plain Retry-After header.

Variations and next steps

You can change or extend this configuration in the following ways:

Variation

How

Change the quota or the threshold Change limit and window_size in the AI Rate Limiting Advanced Policy, or local threshold = 80 in the Post-Function Policy, and re-apply the configuration from Apply the Kong configuration. For a daily quota, set window_size: 86400, and change window = "minute" in the alert to match.
Send the alert to Slack Before you run the quickstart script, export ALERT_WEBHOOK_URL with a Slack incoming webhook URL. Slack posts the text field of the alert as the message.
Send the alert to PagerDuty Change the alert body in the Post-Function Policy to the PagerDuty Events API v2 format (routing_key, event_action: trigger, and a payload with summary, source, and severity), and set ALERT_WEBHOOK_URL to https://events.pagerduty.com/v2/enqueue. See Send an alert event.
Share counters across data plane nodes Set strategy: redis and point the AI Rate Limiting Advanced Policy at a shared Redis instance. The 60-second alert suppression in ngx.shared stays on each node, so each node can send one alert.
Give each AI Consumer Group its own quota Set identifier: consumer-group on the AI Rate Limiting Advanced Policy and list the policy in the AI Consumer Group’s policies. See the LLM cost optimization cookbook for an example.
Authenticate users with an identity provider Replace the Key Auth AI Auth Strategy with an OpenID Connect AI Auth Strategy and map JWT claims to AI Consumers. The AI Model, AI Model Provider, and AI Policies don’t change. See the Claude Code SSO cookbook for an example with Okta.

Cleanup

The quickstart created an AI Gateway for this recipe only, so this teardown removes only this recipe’s configuration. Tear down the local data plane and delete the AI Gateway from Konnect:

export KONNECT_CONTROL_PLANE_NAME='ai-rate-limit-alerts-recipe' && curl -Ls https://get.konghq.com/ai | bash -s -- -d -k $KONNECT_TOKEN

Leave the virtual environment and remove the files and environment variables that the recipe created:

deactivate
rm -rf .venv demo.py
unset CONSUMER_API_KEY CHAT_MODEL ALERT_WEBHOOK_URL

If you created the OpenAI API key only for this recipe, delete it in the OpenAI dashboard.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!