AI Semantic Cache Policy

The AI Semantic Cache Policy stores user requests to an LLM in a vector database based on semantic meaning. When a similar query is made, it uses these embeddings to retrieve relevant cached requests efficiently.

Semantic caching enhances data retrieval efficiency by focusing on the meaning or context of queries rather than just exact matches. It stores requests based on the underlying intent and semantic similarities between different queries and can then retrieve those cached queries when a similar request is made.

When a new request is made, the system can retrieve and reuse previously cached requests if they are contextually relevant, even if the phrasing is different. This method reduces redundant processing, speeds up response times, and ensures that answers are more relevant to the user’s intent, ultimately improving overall system performance and user experience.

For example, if a user asks, “how to integrate our API with a mobile app” and later asks, “what are the steps for connecting our API to a smartphone application?”, the system understands that both questions are asking for the same information. It can then retrieve and reuse previously cached responses, even if the wording is different. This approach reduces processing time and speeds up responses.

The AI Semantic Cache Policy may not be ideal if the following are true:

• You have limited hardware or budget. Storing semantic vectors and running similarity searches require a lot of storage and computing power, which could be an issue.
• Your data doesn’t rely on semantics, or exact matches work fine. In this case, semantic caching may offer little benefit. Traditional or keyword-based caching might be more efficient.

Semantic caching with the AI Semantic Cache Policy involves three parts: request handling, embedding generation, and response caching.

First, a user starts a chat request with the LLM. The AI Semantic Cache Policy queries the vector database to see if there are any semantically similar requests that have already been cached. If there is a match, the vector database returns the cached response to the user.

 
sequenceDiagram
    actor User
    participant AI Gateway/AI Semantic Cache Policy
    participant Vector database

    User->>AI Gateway/AI Semantic Cache Policy: LLM chat request
    AI Gateway/AI Semantic Cache Policy->>Vector database: Query for semantically similar previous requests
    Vector database-->>User: If response, return it or stream it back
  

If there isn’t a match, the AI Semantic Cache Policy prompts the embeddings LLM to generate an embedding for the response.

 
sequenceDiagram
    participant AI Gateway/AI Semantic Cache Policy
    participant Embeddings LLM

    AI Gateway/AI Semantic Cache Policy->>Embeddings LLM: Generate embeddings for `config.message_countback` messages
    Embeddings LLM-->>AI Gateway/AI Semantic Cache Policy: Return embeddings
  

The AI Semantic Cache Policy uses a vector database and cache to store responses to requests. The Policy can then retrieve a cached response if a new request matches the semantics of a previous request, or it can tell the vector database to store a new response if there are no matches.

 
sequenceDiagram
    participant AI Gateway/AI Semantic Cache Policy
    participant Prompt/Chat LLM
    participant Vector database
    actor User

    AI Gateway/AI Semantic Cache Policy->>Prompt/Chat LLM: Make LLM request
    Prompt/Chat LLM-->>AI Gateway/AI Semantic Cache Policy: Receive response
    AI Gateway/AI Semantic Cache Policy->>Vector database: Store vectors
    AI Gateway/AI Semantic Cache Policy->>Vector database: Store response message options
    AI Gateway/AI Semantic Cache Policy-->>User: Return realtime response
  

Cache management

With the AI Semantic Cache Policy, you can configure a cache of your choice to store the responses from the LLM.

The AI Semantic Cache Policy supports Redis and PostgreSQL with pgvector as a vector database. See Vector databases for configuration details.

Caching mechanisms

The AI Semantic Cache Policy improves how AI systems provide responses by using two kinds of caching mechanisms:

• Exact caching: This stores precise, unaltered responses for specific queries. If a user asks the same question multiple times, the system can quickly retrieve the pre-stored response rather than generating it again each time. This speeds up response times and reduces computational load.
• Semantic caching: This approach is more flexible and involves storing responses based on the meaning or intent behind the queries. Instead of relying on exact matches, the system can understand and reuse information that is conceptually similar. For instance, if a user asks about “Italian restaurants in New York City” and later about “New York City Italian cuisine,” semantic caching can help provide relevant information based on their related meanings.

Together, these caching methods enhance the efficiency and relevance of AI responses, making interactions faster and more contextually accurate.

When exact caching is enabled, the AI Semantic Cache Policy may still return results for queries that are similar but not identical. This is expected behavior: the Policy performs similarity-based caching regardless of the exact caching setting.

Headers sent to the client

When the AI Semantic Cache Policy is active, AI Gateway sends additional headers indicating the cache status and other relevant information:

X-Cache-Status: Hit
X-Cache-Status: Miss
X-Cache-Status: Bypass
X-Cache-Status: Refresh
X-Cache-Key: <cache_key>
X-Cache-Ttl: <ttl>
Age: <age>

These headers help clients understand whether a response was served from the cache, if the cache key was used, the remaining time-to-live, and the age of the cached response.

Cache control headers

The Policy respects cache control headers to determine if requests and responses should be cached or not. It supports the following directives:

• no-store: Prevents caching of the request or response
• no-cache: Forces validation with the origin server before serving the cached response
• private: Ensures the response is not cached by shared caches
• max-age and s-maxage: Sets the maximum age of the cached response. This causes the vector database to drop and delete the cached response message after expiration, so it’s never seen again.

As most AI services always send no-cache in the response headers, setting cache_control to true will always result in a cache bypass. Only consider setting no-cache if you are using self-hosted services and have control over the response Cache Control headers.

A vector database stores and compares vector embeddings—numerical representations of text, prompts, documents, or other content. When you configure semantic features in AI Models or AI Policies, embeddings are generated and stored in the vector database so that incoming requests can be compared against the stored vectors to find semantically similar matches. For example, an incoming prompt is embedded and compared against cached prompt keys, model descriptions, document chunks, or allow/deny lists to determine semantic similarity.

AI Gateway semantic features support the following vector databases:

• Using vectordb.strategy: redis and parameters in vectordb.redis:
  • Redis with Redis Vector Search
  • Redis Cloud
  • Valkey: When you configure vectordb.strategy: redis, Kong Gateway queries the server and checks the server name field. If it detects Valkey request, it automatically uses the Valkey-specific driver.
  • Managed Redis with cloud authentication:
    • AWS ElastiCache (auth_provider: aws)
    • Azure Managed Redis (auth_provider: azure)
    • Google Cloud Memorystore (auth_provider: gcp)

    For configuration details, see Using cloud authentication with Redis.

• Using vectordb.strategy: pgvector and parameters in vectordb.pgvector:
  • PostgreSQL with pgvector

Configure vector database settings in AI Models and AI Policies to enable semantic similarity features.

Using cloud authentication with Redis

If your Policy uses a Redis datastore, you can authenticate to it with a cloud Redis provider. This allows you to seamlessly rotate credentials without relying on static passwords.

config:
  vectordb:
    strategy: redis
    redis:
      host: $INSTANCE_ADDRESS
      username: $INSTANCE_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: aws
        aws_cache_name: $AWS_CACHE_NAME
        aws_is_serverless: false
        aws_region: $AWS_REGION
        aws_access_key_id: $AWS_ACCESS_KEY_ID
        aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
  vectordb:
    strategy: redis
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 6379
      username: $CLUSTER_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: aws
        aws_cache_name: $AWS_CACHE_NAME
        aws_is_serverless: false
        aws_region: $AWS_REGION
        aws_access_key_id: $AWS_ACCESS_KEY_ID
        aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
  vectordb:
    strategy: redis
    redis:
      host: $INSTANCE_ADDRESS
      username: $INSTANCE_USERNAME
      port: 10000
      cloud_authentication:
        auth_provider: azure
        azure_client_id: $AZURE_CLIENT_ID
        azure_client_secret: $AZURE_CLIENT_SECRET
        azure_tenant_id: $AZURE_TENANT_ID

config:
  vectordb:
    strategy: redis
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 10000
      username: $CLUSTER_USERNAME
      port: 10000
      cloud_authentication:
        auth_provider: azure
        azure_client_id: $AZURE_CLIENT_ID
        azure_client_secret: $AZURE_CLIENT_SECRET
        azure_tenant_id: $AZURE_TENANT_ID

config:
  vectordb:
    strategy: redis
    redis:
      host: $INSTANCE_ADDRESS
      port: 6379
      cloud_authentication:
        auth_provider: gcp
        gcp_service_account_json: $GCP_SERVICE_ACCOUNT

config:
  vectordb:
    strategy: redis
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 6379
      port: 6379
      cloud_authentication:
        auth_provider: gcp
        gcp_service_account_json: $GCP_SERVICE_ACCOUNT

Set config.proxy on this entity to route its outbound requests through an HTTP forward proxy. Use this in network-isolated deployments where AI Gateway cannot open direct connections to LLM providers or auxiliary services.

The proxy record is identical for AI Model, AI MCP Server, and supported AI Policy entities. Existing capabilities such as load balancing, health checking, streaming, WebSocket, and HTTP/2 continue to work when the proxy is active.

For the full field reference, traffic flow, and limitations, see Forward proxy support.