AI Rate Limiting Advanced: Request prompt function with AWS ElastiCache instance auth - Policy

Request prompt function with AWS ElastiCache instance auth

Protect your LLM services with rate limiting and AWS ElastiCache instance auth. The AI Rate Limiting Advanced Policy analyzes query costs and token responses to provide an enterprise-grade rate limiting strategy.

The following example uses request prompt rate limiting, which lets you rate limit requests based on a custom token.

Prerequisites

An existing AI Model.
A running Redis instance on an AWS ElastiCache instance for Valkey 7.2 or later or ElastiCache for Redis OSS version 7.0 or later..
Port 6379, or your custom Redis port, is open and reachable from AI Gateway.
The ElastiCache user needs to set “Authentication mode” to “IAM”..

The following policy assigned to the IAM user/IAM role that is used to connect to the ElastiCache:

{
  "Version": "2012-10-17",
  "Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "elasticache:Connect"
        ],
        "Resource": [
            "arn:aws:elasticache:ARN_OF_THE_ELASTICACHE",
            "arn:aws:elasticache:ARN_OF_THE_ELASTICACHE_USER"
        ]
    }
]
}

Copied!

Environment variables

INSTANCE_ADDRESS: The ElastiCache instance address.
INSTANCE_USERNAME: The ElastiCache username with IAM Auth mode configured.
AWS_CACHE_NAME: Name of your AWS ElastiCache instance.
AWS_REGION: Your AWS ElastiCache instance region.
AWS_ACCESS_KEY_ID: (Optional) Your AWS access key ID.
AWS_ACCESS_SECRET_KEY: (Optional) Your AWS secret access key.

Set up the policy

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: request-prompt-count-aws-instance
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: request-prompt-count-aws-instance
    display_name: Request prompt function with AWS ElastiCache instance auth
    type: ai-rate-limiting-advanced
    enabled: true
    config:
      strategy: redis
      redis:
        host: !env INSTANCE_ADDRESS
        username: !env INSTANCE_USERNAME
        port: 6379
        cloud_authentication:
          auth_provider: aws
          aws_cache_name: !env AWS_CACHE_NAME
          aws_is_serverless: false
          aws_region: !env AWS_REGION
          aws_access_key_id: !env AWS_ACCESS_KEY_ID
          aws_secret_access_key: !env AWS_ACCESS_SECRET_KEY
      sync_rate: 0
      policies:
      - match:
        - type: provider
          partition_by: true
          values:
          - cohere
        limits:
        - limit: 100
          window_size: 60
        - limit: 1000
          window_size: 3600
      request_prompt_count_function: |
        local header_count = tonumber(kong.request.get_header("x-prompt-count"))
        if header_count then
          return header_count
        end
        return 0
    global: false

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.
!env INSTANCE_ADDRESS: The ElastiCache instance address.
!env INSTANCE_USERNAME: The ElastiCache username with IAM Auth mode configured.
!env AWS_CACHE_NAME: Name of your AWS ElastiCache instance.
!env AWS_REGION: Your AWS ElastiCache instance region.
!env AWS_ACCESS_KEY_ID: (Optional) Your AWS access key ID.
!env AWS_ACCESS_SECRET_KEY: (Optional) Your AWS secret access key.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "request-prompt-count-aws-instance",
      "display_name": "Request prompt function with AWS ElastiCache instance auth",
      "type": "ai-rate-limiting-advanced",
      "enabled": true,
      "config": {
        "strategy": "redis",
        "redis": {
          "host": "'$INSTANCE_ADDRESS'",
          "username": "'$INSTANCE_USERNAME'",
          "port": 6379,
          "cloud_authentication": {
            "auth_provider": "aws",
            "aws_cache_name": "'$AWS_CACHE_NAME'",
            "aws_is_serverless": false,
            "aws_region": "'$AWS_REGION'",
            "aws_access_key_id": "'$AWS_ACCESS_KEY_ID'",
            "aws_secret_access_key": "'$AWS_ACCESS_SECRET_KEY'"
          }
        },
        "sync_rate": 0,
        "policies": [
          {
            "match": [
              {
                "type": "provider",
                "partition_by": true,
                "values": [
                  "cohere"
                ]
              }
            ],
            "limits": [
              {
                "limit": 100,
                "window_size": 60
              },
              {
                "limit": 1000,
                "window_size": 3600
              }
            ]
          }
        ],
        "request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n  return header_count\nend\nreturn 0\n"
      },
      "global": false
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "request-prompt-count-aws-instance"
  display_name = "Request prompt function with AWS ElastiCache instance auth"
  type = "ai-rate-limiting-advanced"
  enabled = true
  global = false
  config = jsonencode({
    strategy = "redis"

    redis = {
      host = var.instance_address
      username = var.instance_username
      port = 6379

      cloud_authentication = {
        auth_provider = "aws"
        aws_cache_name = var.aws_cache_name
        aws_is_serverless = false
        aws_region = var.aws_region
        aws_access_key_id = var.aws_access_key_id
        aws_secret_access_key = var.aws_access_secret_key
      }
    }
    sync_rate = 0
    policies = [
      {
        match = [
          {
            type = "provider"
            partition_by = true
            values = ["cohere"]
          }        ]
        limits = [
          {
            limit = 100
            window_size = 60
          }, 

          {
            limit = 1000
            window_size = 3600
          }        ]
      }    ]
    request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
  return header_count
end
return 0
EOF
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "instance_address" {
  type = string
}
variable "instance_username" {
  type = string
}
variable "aws_cache_name" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_access_secret_key" {
  type = string
}

Copied!

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: request-prompt-count-aws-instance
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: request-prompt-count-aws-instance
    display_name: Request prompt function with AWS ElastiCache instance auth
    type: ai-rate-limiting-advanced
    enabled: true
    config:
      strategy: redis
      redis:
        host: !env INSTANCE_ADDRESS
        username: !env INSTANCE_USERNAME
        port: 6379
        cloud_authentication:
          auth_provider: aws
          aws_cache_name: !env AWS_CACHE_NAME
          aws_is_serverless: false
          aws_region: !env AWS_REGION
          aws_access_key_id: !env AWS_ACCESS_KEY_ID
          aws_secret_access_key: !env AWS_ACCESS_SECRET_KEY
      sync_rate: 0
      policies:
      - match:
        - type: provider
          partition_by: true
          values:
          - cohere
        limits:
        - limit: 100
          window_size: 60
        - limit: 1000
          window_size: 3600
      request_prompt_count_function: |
        local header_count = tonumber(kong.request.get_header("x-prompt-count"))
        if header_count then
          return header_count
        end
        return 0
    global: false

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.
!env INSTANCE_ADDRESS: The ElastiCache instance address.
!env INSTANCE_USERNAME: The ElastiCache username with IAM Auth mode configured.
!env AWS_CACHE_NAME: Name of your AWS ElastiCache instance.
!env AWS_REGION: Your AWS ElastiCache instance region.
!env AWS_ACCESS_KEY_ID: (Optional) Your AWS access key ID.
!env AWS_ACCESS_SECRET_KEY: (Optional) Your AWS secret access key.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "request-prompt-count-aws-instance",
      "display_name": "Request prompt function with AWS ElastiCache instance auth",
      "type": "ai-rate-limiting-advanced",
      "enabled": true,
      "config": {
        "strategy": "redis",
        "redis": {
          "host": "'$INSTANCE_ADDRESS'",
          "username": "'$INSTANCE_USERNAME'",
          "port": 6379,
          "cloud_authentication": {
            "auth_provider": "aws",
            "aws_cache_name": "'$AWS_CACHE_NAME'",
            "aws_is_serverless": false,
            "aws_region": "'$AWS_REGION'",
            "aws_access_key_id": "'$AWS_ACCESS_KEY_ID'",
            "aws_secret_access_key": "'$AWS_ACCESS_SECRET_KEY'"
          }
        },
        "sync_rate": 0,
        "policies": [
          {
            "match": [
              {
                "type": "provider",
                "partition_by": true,
                "values": [
                  "cohere"
                ]
              }
            ],
            "limits": [
              {
                "limit": 100,
                "window_size": 60
              },
              {
                "limit": 1000,
                "window_size": 3600
              }
            ]
          }
        ],
        "request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n  return header_count\nend\nreturn 0\n"
      },
      "global": false
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "request-prompt-count-aws-instance"
  display_name = "Request prompt function with AWS ElastiCache instance auth"
  type = "ai-rate-limiting-advanced"
  enabled = true
  global = false
  config = jsonencode({
    strategy = "redis"

    redis = {
      host = var.instance_address
      username = var.instance_username
      port = 6379

      cloud_authentication = {
        auth_provider = "aws"
        aws_cache_name = var.aws_cache_name
        aws_is_serverless = false
        aws_region = var.aws_region
        aws_access_key_id = var.aws_access_key_id
        aws_secret_access_key = var.aws_access_secret_key
      }
    }
    sync_rate = 0
    policies = [
      {
        match = [
          {
            type = "provider"
            partition_by = true
            values = ["cohere"]
          }        ]
        limits = [
          {
            limit = 100
            window_size = 60
          }, 

          {
            limit = 1000
            window_size = 3600
          }        ]
      }    ]
    request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
  return header_count
end
return 0
EOF
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "instance_address" {
  type = string
}
variable "instance_username" {
  type = string
}
variable "aws_cache_name" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_access_secret_key" {
  type = string
}

Copied!

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: request-prompt-count-aws-instance
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: request-prompt-count-aws-instance
    display_name: Request prompt function with AWS ElastiCache instance auth
    type: ai-rate-limiting-advanced
    enabled: true
    config:
      strategy: redis
      redis:
        host: !env INSTANCE_ADDRESS
        username: !env INSTANCE_USERNAME
        port: 6379
        cloud_authentication:
          auth_provider: aws
          aws_cache_name: !env AWS_CACHE_NAME
          aws_is_serverless: false
          aws_region: !env AWS_REGION
          aws_access_key_id: !env AWS_ACCESS_KEY_ID
          aws_secret_access_key: !env AWS_ACCESS_SECRET_KEY
      sync_rate: 0
      policies:
      - match:
        - type: provider
          partition_by: true
          values:
          - cohere
        limits:
        - limit: 100
          window_size: 60
        - limit: 1000
          window_size: 3600
      request_prompt_count_function: |
        local header_count = tonumber(kong.request.get_header("x-prompt-count"))
        if header_count then
          return header_count
        end
        return 0
    global: false

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.
!env INSTANCE_ADDRESS: The ElastiCache instance address.
!env INSTANCE_USERNAME: The ElastiCache username with IAM Auth mode configured.
!env AWS_CACHE_NAME: Name of your AWS ElastiCache instance.
!env AWS_REGION: Your AWS ElastiCache instance region.
!env AWS_ACCESS_KEY_ID: (Optional) Your AWS access key ID.
!env AWS_ACCESS_SECRET_KEY: (Optional) Your AWS secret access key.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "request-prompt-count-aws-instance",
      "display_name": "Request prompt function with AWS ElastiCache instance auth",
      "type": "ai-rate-limiting-advanced",
      "enabled": true,
      "config": {
        "strategy": "redis",
        "redis": {
          "host": "'$INSTANCE_ADDRESS'",
          "username": "'$INSTANCE_USERNAME'",
          "port": 6379,
          "cloud_authentication": {
            "auth_provider": "aws",
            "aws_cache_name": "'$AWS_CACHE_NAME'",
            "aws_is_serverless": false,
            "aws_region": "'$AWS_REGION'",
            "aws_access_key_id": "'$AWS_ACCESS_KEY_ID'",
            "aws_secret_access_key": "'$AWS_ACCESS_SECRET_KEY'"
          }
        },
        "sync_rate": 0,
        "policies": [
          {
            "match": [
              {
                "type": "provider",
                "partition_by": true,
                "values": [
                  "cohere"
                ]
              }
            ],
            "limits": [
              {
                "limit": 100,
                "window_size": 60
              },
              {
                "limit": 1000,
                "window_size": 3600
              }
            ]
          }
        ],
        "request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n  return header_count\nend\nreturn 0\n"
      },
      "global": false
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "request-prompt-count-aws-instance"
  display_name = "Request prompt function with AWS ElastiCache instance auth"
  type = "ai-rate-limiting-advanced"
  enabled = true
  global = false
  config = jsonencode({
    strategy = "redis"

    redis = {
      host = var.instance_address
      username = var.instance_username
      port = 6379

      cloud_authentication = {
        auth_provider = "aws"
        aws_cache_name = var.aws_cache_name
        aws_is_serverless = false
        aws_region = var.aws_region
        aws_access_key_id = var.aws_access_key_id
        aws_secret_access_key = var.aws_access_secret_key
      }
    }
    sync_rate = 0
    policies = [
      {
        match = [
          {
            type = "provider"
            partition_by = true
            values = ["cohere"]
          }        ]
        limits = [
          {
            limit = 100
            window_size = 60
          }, 

          {
            limit = 1000
            window_size = 3600
          }        ]
      }    ]
    request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
  return header_count
end
return 0
EOF
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "instance_address" {
  type = string
}
variable "instance_username" {
  type = string
}
variable "aws_cache_name" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_access_secret_key" {
  type = string
}

Copied!

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: request-prompt-count-aws-instance
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: request-prompt-count-aws-instance
    display_name: Request prompt function with AWS ElastiCache instance auth
    type: ai-rate-limiting-advanced
    enabled: true
    config:
      strategy: redis
      redis:
        host: !env INSTANCE_ADDRESS
        username: !env INSTANCE_USERNAME
        port: 6379
        cloud_authentication:
          auth_provider: aws
          aws_cache_name: !env AWS_CACHE_NAME
          aws_is_serverless: false
          aws_region: !env AWS_REGION
          aws_access_key_id: !env AWS_ACCESS_KEY_ID
          aws_secret_access_key: !env AWS_ACCESS_SECRET_KEY
      sync_rate: 0
      policies:
      - match:
        - type: provider
          partition_by: true
          values:
          - cohere
        limits:
        - limit: 100
          window_size: 60
        - limit: 1000
          window_size: 3600
      request_prompt_count_function: |
        local header_count = tonumber(kong.request.get_header("x-prompt-count"))
        if header_count then
          return header_count
        end
        return 0
    global: true

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.
!env INSTANCE_ADDRESS: The ElastiCache instance address.
!env INSTANCE_USERNAME: The ElastiCache username with IAM Auth mode configured.
!env AWS_CACHE_NAME: Name of your AWS ElastiCache instance.
!env AWS_REGION: Your AWS ElastiCache instance region.
!env AWS_ACCESS_KEY_ID: (Optional) Your AWS access key ID.
!env AWS_ACCESS_SECRET_KEY: (Optional) Your AWS secret access key.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "request-prompt-count-aws-instance",
      "display_name": "Request prompt function with AWS ElastiCache instance auth",
      "type": "ai-rate-limiting-advanced",
      "enabled": true,
      "config": {
        "strategy": "redis",
        "redis": {
          "host": "'$INSTANCE_ADDRESS'",
          "username": "'$INSTANCE_USERNAME'",
          "port": 6379,
          "cloud_authentication": {
            "auth_provider": "aws",
            "aws_cache_name": "'$AWS_CACHE_NAME'",
            "aws_is_serverless": false,
            "aws_region": "'$AWS_REGION'",
            "aws_access_key_id": "'$AWS_ACCESS_KEY_ID'",
            "aws_secret_access_key": "'$AWS_ACCESS_SECRET_KEY'"
          }
        },
        "sync_rate": 0,
        "policies": [
          {
            "match": [
              {
                "type": "provider",
                "partition_by": true,
                "values": [
                  "cohere"
                ]
              }
            ],
            "limits": [
              {
                "limit": 100,
                "window_size": 60
              },
              {
                "limit": 1000,
                "window_size": 3600
              }
            ]
          }
        ],
        "request_prompt_count_function": "local header_count = tonumber(kong.request.get_header(\"x-prompt-count\"))\nif header_count then\n  return header_count\nend\nreturn 0\n"
      },
      "global": true
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "request-prompt-count-aws-instance"
  display_name = "Request prompt function with AWS ElastiCache instance auth"
  type = "ai-rate-limiting-advanced"
  enabled = true
  global = true
  config = jsonencode({
    strategy = "redis"

    redis = {
      host = var.instance_address
      username = var.instance_username
      port = 6379

      cloud_authentication = {
        auth_provider = "aws"
        aws_cache_name = var.aws_cache_name
        aws_is_serverless = false
        aws_region = var.aws_region
        aws_access_key_id = var.aws_access_key_id
        aws_secret_access_key = var.aws_access_secret_key
      }
    }
    sync_rate = 0
    policies = [
      {
        match = [
          {
            type = "provider"
            partition_by = true
            values = ["cohere"]
          }        ]
        limits = [
          {
            limit = 100
            window_size = 60
          }, 

          {
            limit = 1000
            window_size = 3600
          }        ]
      }    ]
    request_prompt_count_function = <<EOF
local header_count = tonumber(kong.request.get_header("x-prompt-count"))
if header_count then
  return header_count
end
return 0
EOF
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "instance_address" {
  type = string
}
variable "instance_username" {
  type = string
}
variable "aws_cache_name" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_access_secret_key" {
  type = string
}

Copied!

Attach the policy

After creating the policy, add request-prompt-count-aws-instance to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.

kongctl

model.yaml

Copied!

ai_gateway_models:
  - ref: my-gpt-4o
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: my-gpt-4o
    name: my-gpt-4o
    type: model
    capabilities:
    - generate
    formats:
    - type: openai
    targets:
    - name: gpt-4o
      provider: my-openai-account
      config:
        type: openai
    config:
      route:
        paths:
        - "/v1"
        model:
          body_param: model
          values:
          - my-gpt-4o
      logging:
        payloads: false
    policies:
    - !ref request-prompt-count-aws-instance#name

After creating the policy, add request-prompt-count-aws-instance to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.

curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/models/my-gpt-4o \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "my-gpt-4o",
      "name": "my-gpt-4o",
      "type": "model",
      "capabilities": [
        "generate"
      ],
      "formats": [
        {
          "type": "openai"
        }
      ],
      "policies": [
        "request-prompt-count-aws-instance"
      ],
      "targets": [
        {
          "name": "gpt-4o",
          "provider": "my-openai-account",
          "config": {
            "type": "openai"
          }
        }
      ],
      "config": {
        "route": {
          "paths": [
            "/v1"
          ],
          "model": {
            "body_param": "model",
            "values": [
              "my-gpt-4o"
            ]
          }
        },
        "logging": {
          "payloads": false
        }
      }
    }
    '

Copied!

After creating the policy, add request-prompt-count-aws-instance to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.

resource "konnect_ai_gateway_model" "my_model" {
  display_name = "my-gpt-4o"
  name = "my-gpt-4o"
  type = "model"
  capabilities = ["generate"]
  formats = [
    {
      type = "openai"
    }  ]
  targets = [
    {
      name = "gpt-4o"
      provider = "my-openai-account"

      config = {
        type = "openai"
      }
    }  ]
  config = jsonencode({

    route = {
      paths = ["/v1"]

      model = {
        body_param = "model"
        values = ["my-gpt-4o"]
      }
    }

    logging = {
      payloads = false
    }
  })
  policies = [konnect_ai_gateway_policy.my_policy.name]


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "instance_address" {
  type = string
}
variable "instance_username" {
  type = string
}
variable "aws_cache_name" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_access_secret_key" {
  type = string
}

Copied!

After creating the policy, add request-prompt-count-aws-instance to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.

kongctl

consumer.yaml

Copied!

ai_gateway_consumers:
  - ref: mobile-app-production
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: Mobile App - Production
    name: mobile-app-production
    type: api-key
    policies:
    - !ref request-prompt-count-aws-instance#name

After creating the policy, add request-prompt-count-aws-instance to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.

curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/consumers/mobile-app-production \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "Mobile App - Production",
      "name": "mobile-app-production",
      "type": "api-key",
      "policies": [
        "request-prompt-count-aws-instance"
      ]
    }
    '

Copied!

After creating the policy, add request-prompt-count-aws-instance to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.

resource "konnect_ai_gateway_consumer" "my_consumer" {
  display_name = "Mobile App - Production"
  name = "mobile-app-production"
  type = "api-key"
  policies = [konnect_ai_gateway_policy.my_policy.name]


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "instance_address" {
  type = string
}
variable "instance_username" {
  type = string
}
variable "aws_cache_name" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_access_secret_key" {
  type = string
}

Copied!

After creating the policy, add request-prompt-count-aws-instance to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.

kongctl

consumer_group.yaml

Copied!

ai_gateway_consumer_groups:
  - ref: internal-teams
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: Internal Teams
    name: internal-teams
    policies:
    - !ref request-prompt-count-aws-instance#name

After creating the policy, add request-prompt-count-aws-instance to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.

curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/consumer-groups/internal-teams \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "Internal Teams",
      "name": "internal-teams",
      "policies": [
        "request-prompt-count-aws-instance"
      ]
    }
    '

Copied!

After creating the policy, add request-prompt-count-aws-instance to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.

resource "konnect_ai_gateway_consumer_group" "my_consumer_group" {
  display_name = "Internal Teams"
  name = "internal-teams"
  policies = [konnect_ai_gateway_policy.my_policy.name]


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "instance_address" {
  type = string
}
variable "instance_username" {
  type = string
}
variable "aws_cache_name" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_access_secret_key" {
  type = string
}

Copied!

AI Rate Limiting Advanced Policy

Request prompt function with AWS ElastiCache instance auth

Prerequisites

Environment variables

Set up the policy

Attach the policy

Help us make these docs great!

Still need help?