Rate Limiting Advanced Policy

Rate limit how many HTTP requests can be made in a given time frame using multiple rate limits and window sizes, and applying sliding windows. This Policy is a more advanced version of the Rate Limiting Policy, which only allows one fixed rate limiting window.

If the AI Model, AI Agent, or AI MCP Server handling the request has no authentication layer, the client IP address is used to identify clients. Otherwise, the AI Consumer is used once an AI Auth Strategy has authenticated the request.

Advanced features of this Policy include:

• Sliding window support, which provides better performance than fixed rate limiting
• Multiple limits and window sizes
• Support for Redis Sentinel, Redis cluster, and Redis SSL
• Control over which requests contribute to incrementing the rate limiting counters through the config.disable_penalty parameter

To rate limit on LLM token usage or cost rather than request count, use the AI Rate Limiting Advanced Policy instead.

The Rate Limiting Advanced Policy supports the following window types, set with config.window_type:

• Fixed window: Fixed windows consist of buckets that are statically assigned to a definitive time range. Each request is mapped to only one fixed window based on its timestamp and will affect only that window’s counters.
• Sliding window (default): A sliding window tracks the number of hits assigned to a specific key (such as an IP address, AI Consumer, or credential) within a given time window, taking into account previous hit rates to create a dynamically calculated rate. The default (and recommended) sliding window type ensures a resource is not consumed at a higher rate than what is configured.

An arbitrary number of limits or window sizes can be applied per Policy instance. This allows you to create multiple rate limiting windows (for example, rate limit per minute and per hour, and per any arbitrary window size). Each nth limit applies to each nth window size. For example:

ai_gateway_policies:
  - ref: rate-limiting-advanced-global
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: Rate Limiting Advanced - Global
    name: rate-limiting-advanced-global
    type: rate-limiting-advanced
    enabled: true
    global: true
    config:
      limit:
      - 10
      - 100
      window_size:
      - 60
      - 3600

Make sure to replace the following placeholders with your own values:

• AI_GATEWAY_ID: The id of your AI Gateway.

This example applies two rate limits, one of which will trip when 10 hits have been counted in 60 seconds, and the other when 100 hits have been counted in 3600 seconds.

The number of configured window sizes and limits parameters must be equal, otherwise you will get the following error:

You must provide the same number of windows and limits

The config.namespace field is a logical grouping for the counter data used by the rate limiting algorithm. It’s auto-generated for the Policy instance and is optional when you create the Policy.

Set the namespace explicitly in your declarative configuration when you manage AI Gateway with kongctl. Otherwise the field is regenerated automatically with every update.

The Rate Limiting Advanced Policy supports two rate limiting strategies: local and redis. This is controlled by the config.strategy parameter.

[1]: Only when config.sync_rate option is set to 0 (synchronous behavior).

Two common use cases for rate limiting are:

1. Every transaction counts: The highest level of accuracy is needed. An example is a transaction with financial consequences.
2. Backend protection: Accuracy is not as relevant. The requirement is only to protect backend services from overloading that’s caused either by specific users or by attacks.

Every transaction counts

In this scenario, because accuracy is important, the local strategy is not an option. Use redis, and consider the support effort it requires.

If you use a very high sync frequency, redis is the only workable choice. The sync frequency becomes higher when the sync_rate setting is a lower number. For example, a sync_rate of 0.1 is a much higher sync frequency (10 counter syncs per second) than a sync_rate of 1 (1 counter sync per second).

You can calculate what is considered a very high sync rate in your environment based on your topology, number of AI Policies, their sync rates, and tolerance for loose rate limits.

Together, the interaction between sync rate and window size affects how accurately the Rate Limiting Advanced Policy can determine traffic across all nodes. For example, the following table represents the worst-case scenario where a full sync interval’s worth of data hasn’t yet propagated across nodes:

If you choose to switch strategies, note that you can’t port existing usage metrics between strategies. This might not be a problem with short-lived metrics (for example, seconds or minutes) but if you use metrics with a longer time frame (for example, months), plan your switch carefully.

Backend protection

If accuracy is less important, choose the local strategy. You might need to experiment a little before you get a setting that works for your scenario. As AI Gateway scales to more nodes, more user requests are handled. When the number of nodes scales down, the probability of false negatives increases. Make sure to adjust your rate limits when scaling.

For example, if a user can make 100 requests every second, and you have an equally balanced 5-node AI Gateway deployment, you can set the local limit to 30 requests every second. If you see too many false negatives, increase the limit.

To minimize inaccuracies, consider using a consistent-hashing load balancer in front of AI Gateway. The load balancer ensures that a user is always directed to the same AI Gateway node, which reduces inaccuracies and prevents scaling problems.

Using cloud authentication with Redis

If your AI Policy uses a Redis datastore, you can authenticate to it with a cloud Redis provider. This allows you to rotate credentials without relying on static passwords.

The following providers are supported:

• AWS ElastiCache
• Azure Managed Redis
• Google Cloud Memorystore (with or without Valkey)

Each provider also supports an instance and cluster configuration.

config:
  strategy: redis
  redis:
    host: $INSTANCE_ADDRESS
    username: $INSTANCE_USERNAME
    port: 6379
    cloud_authentication:
      auth_provider: aws
      aws_cache_name: $AWS_CACHE_NAME
      aws_is_serverless: false
      aws_region: $AWS_REGION
      aws_access_key_id: $AWS_ACCESS_KEY_ID
      aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
  strategy: redis
  redis:
    cluster_nodes:
    - ip: $CLUSTER_ADDRESS
      port: 6379
    username: $CLUSTER_USERNAME
    port: 6379
    cloud_authentication:
      auth_provider: aws
      aws_cache_name: $AWS_CACHE_NAME
      aws_is_serverless: false
      aws_region: $AWS_REGION
      aws_access_key_id: $AWS_ACCESS_KEY_ID
      aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
  strategy: redis
  redis:
    host: $INSTANCE_ADDRESS
    username: $INSTANCE_USERNAME
    port: 10000
    cloud_authentication:
      auth_provider: azure
      azure_client_id: $AZURE_CLIENT_ID
      azure_client_secret: $AZURE_CLIENT_SECRET
      azure_tenant_id: $AZURE_TENANT_ID

config:
  strategy: redis
  redis:
    cluster_nodes:
    - ip: $CLUSTER_ADDRESS
      port: 10000
    username: $CLUSTER_USERNAME
    port: 10000
    cloud_authentication:
      auth_provider: azure
      azure_client_id: $AZURE_CLIENT_ID
      azure_client_secret: $AZURE_CLIENT_SECRET
      azure_tenant_id: $AZURE_TENANT_ID

config:
  strategy: redis
  redis:
    host: $INSTANCE_ADDRESS
    port: 6379
    cloud_authentication:
      auth_provider: gcp
      gcp_service_account_json: $GCP_SERVICE_ACCOUNT

config:
  strategy: redis
  redis:
    cluster_nodes:
    - ip: $CLUSTER_ADDRESS
      port: 6379
    port: 6379
    cloud_authentication:
      auth_provider: gcp
      gcp_service_account_json: $GCP_SERVICE_ACCOUNT

Fallback from Redis

When the redis strategy is used and an AI Gateway node is disconnected from Redis, the plugin will fall back to local rate limiting. This can happen when the Redis server is down or the connection to Redis is broken. AI Gateway keeps the local counters for rate limiting and syncs with Redis once the connection is re-established. AI Gateway will still rate limit, but the AI Gateway nodes can’t sync the counters. As a result, users will be able to perform more requests than the limit, but there will still be a limit per node.

Use config.identifier to choose what the Policy aggregates counters against:

To key counters on a combination of values, set config.compound_identifier to an array instead, for example ["ip", "consumer"]. When compound_identifier is set, it takes priority over identifier.

Limit by IP address

If limiting by IP address, it’s important to understand how AI Gateway determines the IP address of an incoming request.

The IP address is extracted from the request headers sent to AI Gateway by downstream clients. Typically, these headers are named X-Real-IP or X-Forwarded-For.

By default, AI Gateway uses the header name X-Real-IP to identify the client’s IP address. If your environment requires a different header, you can specify this by setting the real_ip_header Nginx property. Depending on your network setup, you may also need to configure the trusted_ips Nginx property to include the load balancer IP address. This ensures that AI Gateway correctly interprets the client’s IP address, even when the request passes through multiple network layers.

When this Policy is enabled, AI Gateway sends some additional headers back to the client, indicating the state of the rate limits in place:

You can optionally hide the limit and remaining headers with the config.hide_client_headers option.

If more than one limit is set, the Policy returns multiple time limit headers. For example:

X-RateLimit-Limit-Second: 5
X-RateLimit-Remaining-Second: 4
X-RateLimit-Limit-Minute: 10
X-RateLimit-Remaining-Minute: 9

If any of the limits are reached, the Policy returns an HTTP/1.1 429 status code to the client with the following JSON body:

{ "message": "API rate limit exceeded" }

Override the status code with config.error_code and the message with config.error_message.

The headers RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset are based on the Internet-Draft RateLimit Header Fields for HTTP and may change in the future to respect specification updates.

Use config.retry_after_jitter_max to add a random delay of up to the configured number of seconds to the Retry-After header on denied requests. This prevents all clients from retrying at the same moment.

You can use the AI Consumer Group entity to manage custom rate limiting configurations for subsets of AI Consumers.

Set config.enforce_consumer_groups to true and list the groups allowed to override the settings in config.consumer_groups.

You can enable request throttling to improve clients’ experience and protect upstream AI Model Providers from being overwhelmed by traffic spikes. With throttling, requests that exceed the rate limit threshold can be delayed and retried, rather than immediately rejected with a 429 status code.

We recommend setting config.disable_penalty to true when using throttled rate limits with a sliding window. For the sliding window type, if you set disable_penalty to false, all requests, including denied ones, are still counted toward the rate limit. This can lead to a situation where every subsequent window immediately reaches the limit, causing all requests to be denied. In this case, the throttling mechanism will not take effect, because there are no accepted requests left to throttle.

Throttled rate limits work like the following:

1. When a request hits the rate limit, it’s placed into a “waiting room” or queue. The client’s connection is held during this delay.
   • This queue uses the configured rate limiting strategy to manage the queue of throttled requests using a counter-based approach.
2. Requests in the queue are automatically retried after a configurable interval (config.throttling.interval).
   • There’s a limit to retries for individual requests (config.throttling.retry_times), and a cap to the total number of requests waiting (config.throttling.queue_limit).
   • All concurrent requests will retry at approximately the same time once the specified interval has elapsed.
3. If a request exceeds its maximum retries or if the waiting room is full, it will ultimately be rejected with a 429 response.

Enable throttling by setting config.throttling.enabled to true:

ai_gateway_policies:
  - ref: rate-limiting-advanced-throttled
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: Rate Limiting Advanced - Throttled
    name: rate-limiting-advanced-throttled
    type: rate-limiting-advanced
    enabled: true
    global: true
    config:
      limit:
      - 10
      window_size:
      - 60
      window_type: sliding
      disable_penalty: true
      throttling:
        enabled: true
        interval: 5
        retry_times: 3
        queue_limit: 100

Make sure to replace the following placeholders with your own values:

• AI_GATEWAY_ID: The id of your AI Gateway.