AI Rate Limiting Advanced Policy

The AI Rate Limiting Advanced Policy provides rate limiting for all AI Policies. The AI Rate Limiting Advanced Policy extends the Rate Limiting Advanced Policy.

This Policy uses the token data returned by the LLM provider to calculate the costs of queries. The same HTTP request can vary greatly in cost depending on the calculation of the LLM providers. See Model cost management for how AI Gateway prices a request.

A common pattern to protect your AI API is to analyze and assign costs to incoming queries, then rate limit the consumer’s cost for a given time window and provider or policy. You can also create a generic prompt rate limit using the request prompt provider.

The AI Rate Limiting Advanced Policy supports three rate limiting strategies: local, cluster, and redis. This is controlled by the config.strategy parameter.

[1]: Only when config.sync_rate option is set to 0 (synchronous behavior).

Two common use cases for rate limiting are:

1. Every transaction counts: The highest level of accuracy is needed. An example is a transaction with financial consequences.
2. Backend protection: Accuracy is not as relevant. The requirement is only to protect backend services from overloading that’s caused either by specific users or by attacks.

Every transaction counts

In this scenario, because accuracy is important, the local strategy is not an option. Consider the support effort you might need for Redis, and then choose either cluster or redis.

You could start with the cluster strategy, and move to redis if performance reduces drastically.

If using a very high sync frequency, use redis. Very high sync frequencies with cluster mode are not scalable and not recommended. The sync frequency becomes higher when the sync_rate setting is a lower number. For example, a sync_rate of 0.1 is a much higher sync frequency (10 counter syncs per second) than a sync_rate of 1 (1 counter sync per second).

You can calculate what is considered a very high sync rate in your environment based on your topology, number of AI Policies, their sync rates, and tolerance for loose rate limits.

Together, the interaction between sync rate and window size affects how accurately the AI Rate Limiting Advanced Policy can determine traffic across all nodes. For example, the following table represents the worst-case scenario where a full sync interval’s worth of data hasn’t yet propagated across nodes:

If you choose to switch strategies, note that you can’t port the existing usage metrics from the Kong Gateway data store to Redis. This might not be a problem with short-lived metrics (for example, seconds or minutes) but if you use metrics with a longer time frame (for example, months), plan your switch carefully.

Backend protection

If accuracy is less important, choose the local strategy. You might need to experiment a little before you get a setting that works for your scenario. As AI Gateway scales to more nodes, more user requests are handled. When the number of nodes scales down, the probability of false negatives increases. Make sure to adjust your rate limits when scaling.

For example, if a user can make 100 requests every second, and you have an equally balanced 5-node AI Gateway deployment, you can set the local limit to 30 requests every second. If you see too many false negatives, increase the limit.

To minimize inaccuracies, consider using a consistent-hashing load balancer in front of AI Gateway. The load balancer ensures that a user is always directed to the same AI Gateway node, which reduces inaccuracies and prevents scaling problems.

Using cloud authentication with Redis

If your AI Policy uses a Redis datastore, you can authenticate to it with a cloud Redis provider. This allows you to rotate credentials without relying on static passwords.

The following providers are supported:

• AWS ElastiCache
• Azure Managed Redis
• Google Cloud Memorystore (with or without Valkey)

Each provider also supports an instance and cluster configuration.

config:
  strategy: redis
  redis:
    host: $INSTANCE_ADDRESS
    username: $INSTANCE_USERNAME
    port: 6379
    cloud_authentication:
      auth_provider: aws
      aws_cache_name: $AWS_CACHE_NAME
      aws_is_serverless: false
      aws_region: $AWS_REGION
      aws_access_key_id: $AWS_ACCESS_KEY_ID
      aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
  strategy: redis
  redis:
    cluster_nodes:
    - ip: $CLUSTER_ADDRESS
      port: 6379
    username: $CLUSTER_USERNAME
    port: 6379
    cloud_authentication:
      auth_provider: aws
      aws_cache_name: $AWS_CACHE_NAME
      aws_is_serverless: false
      aws_region: $AWS_REGION
      aws_access_key_id: $AWS_ACCESS_KEY_ID
      aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
  strategy: redis
  redis:
    host: $INSTANCE_ADDRESS
    username: $INSTANCE_USERNAME
    port: 10000
    cloud_authentication:
      auth_provider: azure
      azure_client_id: $AZURE_CLIENT_ID
      azure_client_secret: $AZURE_CLIENT_SECRET
      azure_tenant_id: $AZURE_TENANT_ID

config:
  strategy: redis
  redis:
    cluster_nodes:
    - ip: $CLUSTER_ADDRESS
      port: 10000
    username: $CLUSTER_USERNAME
    port: 10000
    cloud_authentication:
      auth_provider: azure
      azure_client_id: $AZURE_CLIENT_ID
      azure_client_secret: $AZURE_CLIENT_SECRET
      azure_tenant_id: $AZURE_TENANT_ID

config:
  strategy: redis
  redis:
    host: $INSTANCE_ADDRESS
    port: 6379
    cloud_authentication:
      auth_provider: gcp
      gcp_service_account_json: $GCP_SERVICE_ACCOUNT

config:
  strategy: redis
  redis:
    cluster_nodes:
    - ip: $CLUSTER_ADDRESS
      port: 6379
    port: 6379
    cloud_authentication:
      auth_provider: gcp
      gcp_service_account_json: $GCP_SERVICE_ACCOUNT

Fallback from Redis

When the redis strategy is used and an AI Gateway node is disconnected from Redis, the plugin will fall back to local rate limiting. This can happen when the Redis server is down or the connection to Redis is broken. AI Gateway keeps the local counters for rate limiting and syncs with Redis once the connection is re-established. AI Gateway will still rate limit, but the AI Gateway nodes can’t sync the counters. As a result, users will be able to perform more requests than the limit, but there will still be a limit per node.

The config.policies field allows you to define rate limiting at the AI Consumer, AI Consumer Group, IP address, header, path, model, and provider level. The match conditions under config.policies.match use an AND logic, so you can combine these to set up multi-dimensional rate limiting. For example, you can set different rate limiting policies for a specific AI Consumer and model:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "AI Rate Limiting Advanced - Consumer and Model",
      "type": "ai-rate-limiting-advanced",
      "name": "ai-rate-limiting-advanced",
      "config": {
        "policies": [
          {
            "match": [
              {
                "type": "consumer",
                "key": "id",
                "values": [
                  "'$CONSUMER_ID'"
                ]
              },
              {
                "type": "model",
                "partition_by": true,
                "values": [
                  "gpt-4o"
                ]
              }
            ],
            "limits": [
              {
                "limit": 100,
                "window_size": 60
              },
              {
                "limit": 1000,
                "window_size": 3600
              }
            ]
          }
        ]
      }
    }
    '

ai_gateway_policies:
  - ref: ai-rate-limiting-advanced
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: AI Rate Limiting Advanced - Consumer and Model
    type: ai-rate-limiting-advanced
    name: ai-rate-limiting-advanced
    config:
      policies:
      - match:
        - type: consumer
          key: id
          values:
          - !env CONSUMER_ID
        - type: model
          partition_by: true
          values:
          - gpt-4o
        limits:
        - limit: 100
          window_size: 60
        - limit: 1000
          window_size: 3600

In this example, the limits will apply only to requests made by the specified AI Consumer to the gpt-4o model.

Policies without match conditions act as fallback and match all requests.

When defining rate limits for a specific model, these limits apply to the requested model. If a request is redirected to a different model after a failover, the request may succeed even if the final model has reached its limit.

Window types

Each policy sets a window_type: fixed, sliding, or calendar.

• sliding (default): Weighs the current window against the previous one to produce a dynamically calculated rate.
• fixed: Assigns each request to a single time bucket based on its timestamp.
• calendar: Aligns the window to a real calendar boundary in a specific time zone, instead of a rolling window_size.

Calendar windows

Calendar windows budget tokens against the same period your billing already uses, such as a monthly allowance that resets on the 1st, instead of a rolling window that drifts from the billing cycle.

Set window_type: calendar on the policy and timezone to an IANA time zone, such as America/New_York. Each limit then sets a period instead of window_size:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "AI Rate Limiting Advanced - Calendar Window",
      "name": "ai-rate-limiting-advanced",
      "type": "ai-rate-limiting-advanced",
      "config": {
        "policies": [
          {
            "window_type": "calendar",
            "timezone": "America/New_York",
            "match": [
              {
                "type": "consumer_group",
                "values": [
                  "premium"
                ]
              }
            ],
            "limits": [
              {
                "limit": 2000000,
                "period": "month",
                "month_day": 1,
                "tokens_count_strategy": "total_tokens"
              }
            ]
          }
        ]
      }
    }
    '

ai_gateway_policies:
  - ref: ai-rate-limiting-advanced
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: AI Rate Limiting Advanced - Calendar Window
    name: ai-rate-limiting-advanced
    type: ai-rate-limiting-advanced
    config:
      policies:
      - window_type: calendar
        timezone: America/New_York
        match:
        - type: consumer_group
          values:
          - premium
        limits:
        - limit: 2000000
          period: month
          month_day: 1
          tokens_count_strategy: total_tokens

This policy grants the premium AI Consumer Group 2,000,000 tokens per calendar month, resetting at local midnight on the 1st in America/New_York.

For calendar windows, X-AI-RateLimit-Reset and X-AI-RateLimit-Retry-After point to the next calendar boundary (the start of the next week or month in the configured time zone), not now + window_size.

Known issues

• When defining a policy matching a model and/or a provider, you must set the config.policies.match.partition_by field to true, otherwise the policy is not enforced.

When the AI Rate Limiting Advanced Policy is enabled, AI Gateway sends some additional headers back to the client, indicating the allowed limits, how many requests are available, and how long it will take until the quota is restored. It also sends the limits in the time frame and the number of remaining minutes for each provider or policy.

For example:

X-AI-RateLimit-Reset: 51
X-AI-RateLimit-Retry-After: 51
X-AI-RateLimit-Limit-90-policy-1: 20
X-AI-RateLimit-Remaining-90-policy-1: 0

You can optionally hide the limit and remaining headers with the config.hide_client_headers option.

If more than one limit is set, the AI Rate Limiting Advanced Policy returns multiple time limit headers. For example:

X-AI-RateLimit-Limit-30-azure: 1000
X-AI-RateLimit-Remaining-30-azure: 950
X-AI-RateLimit-Limit-40-cohere: 2000
X-AI-RateLimit-Remaining-40-cohere: 1150

If any of the limits are reached, the AI Rate Limiting Advanced Policy returns an HTTP/1.1 429 status code to the client with the following JSON body:

{ "message": "API rate limit exceeded for provider azure, cohere" }

For each provider or policy, the AI Rate Limiting Advanced Policy also indicates how long it will take until the quota is restored:

X-AI-RateLimit-Retry-After-30-azure: 1500
X-AI-RateLimit-Reset-30-azure: 1500

If using the request prompt provider, the AI Rate Limiting Advanced Policy will send the query cost:

X-AI-RateLimit-Query-Cost: 100

The Retry-After headers will be present on 429 errors to indicate how long the service is expected to be unavailable to the client. When using window_type=sliding and RateLimit-Reset, Retry-After may increase due to the rate calculation for the sliding window.

The headers RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset are based on the Internet-Draft RateLimit Header Fields for HTTP and may change in the future to respect specification updates.

The AI Rate Limiting Advanced Policy supports three strategies to calculate the number of tokens. Configure the strategy with tokens_count_strategy.

Request prompt function

You can decide to use a custom function to count the tokens for a requests. To configure it, specify the function in config.request_prompt_count_function.

When using the request prompt provider, it will call the function to get the token count at the request level and implement a limit.

The cost is only reflected during the next request.

For example, if a request is made and returns a token cost of 100 for the OpenAI provider:

• The request is made to the OpenAI provider and the response is returned to the user
• If the rate limit is reached, the next request will be blocked

Additionally, config.disable_penalty only works for the requestPrompt function.