AI Semantic Cache: Enable exact-match caching with a custom TTL - Policy

Enable exact-match caching with a custom TTL

Configure the AI Semantic Cache policy to check for an exact match before it searches by similarity, and to keep cached entries for 10 minutes. Exact-match caching increases the size of the vector database.

Prerequisites

An existing AI Model.
An OpenAI account.
A Redis instance.
Port 6379, or your custom Redis port, is open and reachable from AI Gateway.

Environment variables

OPENAI_API_KEY: Your OpenAI API key
REDIS_HOST: The host where your Redis instance runs

Set up the policy

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: semantic-cache-exact-match-ttl
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: semantic-cache-exact-match-ttl
    display_name: Enable exact-match caching with a custom TTL
    type: ai-semantic-cache
    enabled: true
    config:
      embeddings:
        auth:
          header_name: Authorization
          header_value: Bearer $OPENAI_API_KEY
        model:
          provider: openai
          name: text-embedding-3-large
          options:
            upstream_url: https://api.openai.com/v1/embeddings
      vectordb:
        dimensions: 3072
        distance_metric: cosine
        strategy: redis
        threshold: 0.1
        redis:
          host: !env REDIS_HOST
          port: 6379
      exact_caching: true
      cache_ttl: 600
    global: false

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "semantic-cache-exact-match-ttl",
      "display_name": "Enable exact-match caching with a custom TTL",
      "type": "ai-semantic-cache",
      "enabled": true,
      "config": {
        "embeddings": {
          "auth": {
            "header_name": "Authorization",
            "header_value": "Bearer '$OPENAI_API_KEY'"
          },
          "model": {
            "provider": "openai",
            "name": "text-embedding-3-large",
            "options": {
              "upstream_url": "https://api.openai.com/v1/embeddings"
            }
          }
        },
        "vectordb": {
          "dimensions": 3072,
          "distance_metric": "cosine",
          "strategy": "redis",
          "threshold": 0.1,
          "redis": {
            "host": "'$REDIS_HOST'",
            "port": 6379
          }
        },
        "exact_caching": true,
        "cache_ttl": 600
      },
      "global": false
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "semantic-cache-exact-match-ttl"
  display_name = "Enable exact-match caching with a custom TTL"
  type = "ai-semantic-cache"
  enabled = true
  global = false
  config = jsonencode({

    embeddings = {

      auth = {
        header_name = "Authorization"
        header_value = "Bearer var.openai_api_key"
      }

      model = {
        provider = "openai"
        name = "text-embedding-3-large"

        options = {
          upstream_url = "https://api.openai.com/v1/embeddings"
        }
      }
    }

    vectordb = {
      dimensions = 3072
      distance_metric = "cosine"
      strategy = "redis"
      threshold = 0.1

      redis = {
        host = var.redis_host
        port = 6379
      }
    }
    exact_caching = true
    cache_ttl = 600
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "openai_api_key" {
  type = string
}
variable "redis_host" {
  type = string
}

Copied!

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: semantic-cache-exact-match-ttl
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: semantic-cache-exact-match-ttl
    display_name: Enable exact-match caching with a custom TTL
    type: ai-semantic-cache
    enabled: true
    config:
      embeddings:
        auth:
          header_name: Authorization
          header_value: Bearer $OPENAI_API_KEY
        model:
          provider: openai
          name: text-embedding-3-large
          options:
            upstream_url: https://api.openai.com/v1/embeddings
      vectordb:
        dimensions: 3072
        distance_metric: cosine
        strategy: redis
        threshold: 0.1
        redis:
          host: !env REDIS_HOST
          port: 6379
      exact_caching: true
      cache_ttl: 600
    global: false

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "semantic-cache-exact-match-ttl",
      "display_name": "Enable exact-match caching with a custom TTL",
      "type": "ai-semantic-cache",
      "enabled": true,
      "config": {
        "embeddings": {
          "auth": {
            "header_name": "Authorization",
            "header_value": "Bearer '$OPENAI_API_KEY'"
          },
          "model": {
            "provider": "openai",
            "name": "text-embedding-3-large",
            "options": {
              "upstream_url": "https://api.openai.com/v1/embeddings"
            }
          }
        },
        "vectordb": {
          "dimensions": 3072,
          "distance_metric": "cosine",
          "strategy": "redis",
          "threshold": 0.1,
          "redis": {
            "host": "'$REDIS_HOST'",
            "port": 6379
          }
        },
        "exact_caching": true,
        "cache_ttl": 600
      },
      "global": false
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "semantic-cache-exact-match-ttl"
  display_name = "Enable exact-match caching with a custom TTL"
  type = "ai-semantic-cache"
  enabled = true
  global = false
  config = jsonencode({

    embeddings = {

      auth = {
        header_name = "Authorization"
        header_value = "Bearer var.openai_api_key"
      }

      model = {
        provider = "openai"
        name = "text-embedding-3-large"

        options = {
          upstream_url = "https://api.openai.com/v1/embeddings"
        }
      }
    }

    vectordb = {
      dimensions = 3072
      distance_metric = "cosine"
      strategy = "redis"
      threshold = 0.1

      redis = {
        host = var.redis_host
        port = 6379
      }
    }
    exact_caching = true
    cache_ttl = 600
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "openai_api_key" {
  type = string
}
variable "redis_host" {
  type = string
}

Copied!

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: semantic-cache-exact-match-ttl
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: semantic-cache-exact-match-ttl
    display_name: Enable exact-match caching with a custom TTL
    type: ai-semantic-cache
    enabled: true
    config:
      embeddings:
        auth:
          header_name: Authorization
          header_value: Bearer $OPENAI_API_KEY
        model:
          provider: openai
          name: text-embedding-3-large
          options:
            upstream_url: https://api.openai.com/v1/embeddings
      vectordb:
        dimensions: 3072
        distance_metric: cosine
        strategy: redis
        threshold: 0.1
        redis:
          host: !env REDIS_HOST
          port: 6379
      exact_caching: true
      cache_ttl: 600
    global: false

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "semantic-cache-exact-match-ttl",
      "display_name": "Enable exact-match caching with a custom TTL",
      "type": "ai-semantic-cache",
      "enabled": true,
      "config": {
        "embeddings": {
          "auth": {
            "header_name": "Authorization",
            "header_value": "Bearer '$OPENAI_API_KEY'"
          },
          "model": {
            "provider": "openai",
            "name": "text-embedding-3-large",
            "options": {
              "upstream_url": "https://api.openai.com/v1/embeddings"
            }
          }
        },
        "vectordb": {
          "dimensions": 3072,
          "distance_metric": "cosine",
          "strategy": "redis",
          "threshold": 0.1,
          "redis": {
            "host": "'$REDIS_HOST'",
            "port": 6379
          }
        },
        "exact_caching": true,
        "cache_ttl": 600
      },
      "global": false
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "semantic-cache-exact-match-ttl"
  display_name = "Enable exact-match caching with a custom TTL"
  type = "ai-semantic-cache"
  enabled = true
  global = false
  config = jsonencode({

    embeddings = {

      auth = {
        header_name = "Authorization"
        header_value = "Bearer var.openai_api_key"
      }

      model = {
        provider = "openai"
        name = "text-embedding-3-large"

        options = {
          upstream_url = "https://api.openai.com/v1/embeddings"
        }
      }
    }

    vectordb = {
      dimensions = 3072
      distance_metric = "cosine"
      strategy = "redis"
      threshold = 0.1

      redis = {
        host = var.redis_host
        port = 6379
      }
    }
    exact_caching = true
    cache_ttl = 600
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "openai_api_key" {
  type = string
}
variable "redis_host" {
  type = string
}

Copied!

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: semantic-cache-exact-match-ttl
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: semantic-cache-exact-match-ttl
    display_name: Enable exact-match caching with a custom TTL
    type: ai-semantic-cache
    enabled: true
    config:
      embeddings:
        auth:
          header_name: Authorization
          header_value: Bearer $OPENAI_API_KEY
        model:
          provider: openai
          name: text-embedding-3-large
          options:
            upstream_url: https://api.openai.com/v1/embeddings
      vectordb:
        dimensions: 3072
        distance_metric: cosine
        strategy: redis
        threshold: 0.1
        redis:
          host: !env REDIS_HOST
          port: 6379
      exact_caching: true
      cache_ttl: 600
    global: true

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "semantic-cache-exact-match-ttl",
      "display_name": "Enable exact-match caching with a custom TTL",
      "type": "ai-semantic-cache",
      "enabled": true,
      "config": {
        "embeddings": {
          "auth": {
            "header_name": "Authorization",
            "header_value": "Bearer '$OPENAI_API_KEY'"
          },
          "model": {
            "provider": "openai",
            "name": "text-embedding-3-large",
            "options": {
              "upstream_url": "https://api.openai.com/v1/embeddings"
            }
          }
        },
        "vectordb": {
          "dimensions": 3072,
          "distance_metric": "cosine",
          "strategy": "redis",
          "threshold": 0.1,
          "redis": {
            "host": "'$REDIS_HOST'",
            "port": 6379
          }
        },
        "exact_caching": true,
        "cache_ttl": 600
      },
      "global": true
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "semantic-cache-exact-match-ttl"
  display_name = "Enable exact-match caching with a custom TTL"
  type = "ai-semantic-cache"
  enabled = true
  global = true
  config = jsonencode({

    embeddings = {

      auth = {
        header_name = "Authorization"
        header_value = "Bearer var.openai_api_key"
      }

      model = {
        provider = "openai"
        name = "text-embedding-3-large"

        options = {
          upstream_url = "https://api.openai.com/v1/embeddings"
        }
      }
    }

    vectordb = {
      dimensions = 3072
      distance_metric = "cosine"
      strategy = "redis"
      threshold = 0.1

      redis = {
        host = var.redis_host
        port = 6379
      }
    }
    exact_caching = true
    cache_ttl = 600
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "openai_api_key" {
  type = string
}
variable "redis_host" {
  type = string
}

Copied!

Attach the policy

After creating the policy, add semantic-cache-exact-match-ttl to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.

kongctl

model.yaml

Copied!

ai_gateway_models:
  - ref: my-gpt-4o
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: my-gpt-4o
    name: my-gpt-4o
    type: model
    capabilities:
    - generate
    formats:
    - type: openai
    targets:
    - name: gpt-4o
      provider: my-openai-account
      config:
        type: openai
    config:
      route:
        paths:
        - "/v1"
        model:
          body_param: model
          values:
          - my-gpt-4o
      logging:
        payloads: false
    policies:
    - !ref semantic-cache-exact-match-ttl#name

After creating the policy, add semantic-cache-exact-match-ttl to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.

curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/models/my-gpt-4o \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "my-gpt-4o",
      "name": "my-gpt-4o",
      "type": "model",
      "capabilities": [
        "generate"
      ],
      "formats": [
        {
          "type": "openai"
        }
      ],
      "policies": [
        "semantic-cache-exact-match-ttl"
      ],
      "targets": [
        {
          "name": "gpt-4o",
          "provider": "my-openai-account",
          "config": {
            "type": "openai"
          }
        }
      ],
      "config": {
        "route": {
          "paths": [
            "/v1"
          ],
          "model": {
            "body_param": "model",
            "values": [
              "my-gpt-4o"
            ]
          }
        },
        "logging": {
          "payloads": false
        }
      }
    }
    '

Copied!

After creating the policy, add semantic-cache-exact-match-ttl to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.

resource "konnect_ai_gateway_model" "my_model" {
  display_name = "my-gpt-4o"
  name = "my-gpt-4o"
  type = "model"
  capabilities = ["generate"]
  formats = [
    {
      type = "openai"
    }  ]
  targets = [
    {
      name = "gpt-4o"
      provider = "my-openai-account"

      config = {
        type = "openai"
      }
    }  ]
  config = jsonencode({

    route = {
      paths = ["/v1"]

      model = {
        body_param = "model"
        values = ["my-gpt-4o"]
      }
    }

    logging = {
      payloads = false
    }
  })
  policies = [konnect_ai_gateway_policy.my_policy.name]


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "openai_api_key" {
  type = string
}
variable "redis_host" {
  type = string
}

Copied!

After creating the policy, add semantic-cache-exact-match-ttl to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.

kongctl

consumer.yaml

Copied!

ai_gateway_consumers:
  - ref: mobile-app-production
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: Mobile App - Production
    name: mobile-app-production
    type: api-key
    policies:
    - !ref semantic-cache-exact-match-ttl#name

After creating the policy, add semantic-cache-exact-match-ttl to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.

curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/consumers/mobile-app-production \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "Mobile App - Production",
      "name": "mobile-app-production",
      "type": "api-key",
      "policies": [
        "semantic-cache-exact-match-ttl"
      ]
    }
    '

Copied!

After creating the policy, add semantic-cache-exact-match-ttl to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.

resource "konnect_ai_gateway_consumer" "my_consumer" {
  display_name = "Mobile App - Production"
  name = "mobile-app-production"
  type = "api-key"
  policies = [konnect_ai_gateway_policy.my_policy.name]


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "openai_api_key" {
  type = string
}
variable "redis_host" {
  type = string
}

Copied!

After creating the policy, add semantic-cache-exact-match-ttl to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.

kongctl

consumer_group.yaml

Copied!

ai_gateway_consumer_groups:
  - ref: internal-teams
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: Internal Teams
    name: internal-teams
    policies:
    - !ref semantic-cache-exact-match-ttl#name

After creating the policy, add semantic-cache-exact-match-ttl to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.

curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/consumer-groups/internal-teams \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "Internal Teams",
      "name": "internal-teams",
      "policies": [
        "semantic-cache-exact-match-ttl"
      ]
    }
    '

Copied!

After creating the policy, add semantic-cache-exact-match-ttl to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.

resource "konnect_ai_gateway_consumer_group" "my_consumer_group" {
  display_name = "Internal Teams"
  name = "internal-teams"
  policies = [konnect_ai_gateway_policy.my_policy.name]


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "openai_api_key" {
  type = string
}
variable "redis_host" {
  type = string
}

Copied!

AI Semantic Cache Policy

Enable exact-match caching with a custom TTL

Prerequisites

Environment variables

Set up the policy

Attach the policy

Help us make these docs great!

Still need help?