AI RAG Injector Policy

Retrieval-Augmented Generation (RAG) is a technique that improves the accuracy and relevance of language model responses by enriching prompts with external data at runtime. Instead of relying solely on what the model was trained on, RAG retrieves contextually relevant information such as documents, support articles, or internal knowledge from connected data sources. These sources are usually indexed in a vector database.

The RAG pipeline then automatically injects the retrieved context into the prompt before the model generates a response. RAG is a critical safeguard in specialized or high-stakes applications, where factual accuracy matters. LLMs are prone to hallucinations, plausible-sounding but factually incorrect or fabricated responses. RAG helps mitigate this by grounding the model’s output in real, verifiable data.

The following table describes the different use cases for RAG based on industry:

The AI RAG Injector Policy automates the retrieval and injection of contextual data for RAG pipelines without doing manual prompt engineering or retrieval logic. Integrated at the gateway level, it handles embedding generation, vector search, and context injection transparently for each request.

• Simplifies RAG workflows: Automatically embeds prompts, queries the vector DB, and injects relevant context without custom retrieval logic.
• Platform-level control: Shifts RAG logic from app code to infrastructure, allowing platform teams to enforce global policies, update configurations centrally, and reduce developer overhead.
• Improved security: Vector DB access is limited to the AI Gateway, eliminating the need to expose it to individual dev teams or AI agents.
• Enables RAG in restricted environments: Supports RAG even where direct access to the vector database is not possible, such as external-facing or isolated services.
• Developer productivity: Developers can focus on building AI features without needing to manage embeddings, similarity search, or context handling.
• Save LLM costs: When using the AI RAG Injector Policy with the AI Prompt Compressor Policy, you can wrap specific prompt parts in <LLMLINGUA> tags within your template to target only those sections for compression, preserving the rest of the prompt unchanged.

When a user sends a prompt, the AI RAG Injector Policy queries a configured vector database for relevant context and injects that information into the request before passing it to the language model.

1. You attach the AI RAG Injector Policy to an AI Model via the Konnect API, configuring vector database connection and embedding settings.
2. When a request reaches the AI Gateway, the AI Policy generates embeddings for request prompts, then queries the vector database for the top-k most similar embeddings.
3. The AI Policy injects the retrieved content from the vector search result into the request body, and forwards the request to the AI Model Provider.

The following diagram is a simplified overview of how the AI Policy works. See the following section for a more detailed description.

 
sequenceDiagram
    participant User
    participant AIGateway as AI Gateway (AI RAG Injector Policy)
    participant VectorDB as Vector DB (Knowledge store)
    participant LLM as Upstream LLM

    User->>AIGateway: Send request with prompt
    AIGateway->>VectorDB: Query for similar embeddings
    VectorDB-->>AIGateway: Return relevant context
    AIGateway->>LLM: Inject context and forward enriched request
    LLM-->>User: Return response
  

RAG Generation process

The RAG workflow consists of two critical phases:

1. Data preparation: Processes and embeds unstructured data into a vector index for efficient semantic search
2. Retrieval and generation: The system uses similarity search to dynamically assemble contextual prompts that guide the language model’s output.

Phase 1: Data Preparation

This phase sets up the foundation for semantic retrieval by converting raw data into a format that can be indexed and searched efficiently.

Step breakdown:

1. A document loader pulls content from various sources, such as PDFs, websites, emails, or internal systems.
2. The system breaks the unstructured data into smaller, semantically meaningful chunks to support precise retrieval.
3. Each chunk is transformed into a vector embedding (a numeric representation that captures its semantic content).
4. These embeddings are saved to a vector database, enabling a fast, similarity-based search during query time.

Phase 2: Retrieval and Generation

This phase runs in real time, taking user input and producing a context-aware response using the indexed data.

Step breakdown:

1. The user’s query is converted into an embedding using the same model used during data preparation.
2. A semantic similarity search locates the most relevant content chunks in the vector database.
3. The system builds a custom prompt by combining the retrieved chunks with the original query.
4. The LLM generates a contextually accurate response using both the retrieved context and its own internal knowledge.

The following diagram shows how data flows through both phases of the RAG pipeline, from ingestion and embedding to real-time query handling and response generation:

 
sequenceDiagram
    autonumber
    actor User
    participant RawData as Raw Data
    participant EmbeddingModel as Embedding Model
    participant VectorDB as Vector Database
    participant LLM

    par Data preparation
        activate RawData
        RawData->>EmbeddingModel: Load and chunk documents, generate embeddings
        deactivate RawData

        activate EmbeddingModel
        EmbeddingModel->>VectorDB: Store embeddings
        deactivate EmbeddingModel

        activate VectorDB
        deactivate VectorDB
    end

    par Retrieval & generation
        activate User
        User->>EmbeddingModel: (1) Submit query and generate query embedding

        activate EmbeddingModel
        EmbeddingModel->>VectorDB: (2) Search vector DB
        deactivate EmbeddingModel

        activate VectorDB
        VectorDB-->>EmbeddingModel: Return relevant chunks
        deactivate VectorDB

        activate EmbeddingModel
        EmbeddingModel->>LLM: (3) Assemble prompt and send
        deactivate EmbeddingModel

        activate LLM
        LLM-->>User: (4) Generate and return response
        deactivate LLM
        deactivate User
    end
  

Rather than guessing from memory, the LLM paired with the RAG pipeline now has the ability to look up the information it needs in real time, which reduces hallucinations and increases the accuracy of the AI output.

A vector database stores and compares vector embeddings—numerical representations of text, prompts, documents, or other content. When you configure semantic features in AI Models or AI Policies, embeddings are generated and stored in the vector database so that incoming requests can be compared against the stored vectors to find semantically similar matches. For example, an incoming prompt is embedded and compared against cached prompt keys, model descriptions, document chunks, or allow/deny lists to determine semantic similarity.

AI Gateway semantic features support the following vector databases:

• Using vectordb.strategy: redis and parameters in vectordb.redis:
  • Redis with Redis Vector Search
  • Redis Cloud
  • Valkey: When you configure vectordb.strategy: redis, Kong Gateway queries the server and checks the server name field. If it detects Valkey request, it automatically uses the Valkey-specific driver.
  • Managed Redis with cloud authentication:
    • AWS ElastiCache (auth_provider: aws)
    • Azure Managed Redis (auth_provider: azure)
    • Google Cloud Memorystore (auth_provider: gcp)

    For configuration details, see Using cloud authentication with Redis.

• Using vectordb.strategy: pgvector and parameters in vectordb.pgvector:
  • PostgreSQL with pgvector

Configure vector database settings in AI Models and AI Policies to enable semantic similarity features.

Using cloud authentication with Redis

If your AI Policy uses a Redis datastore, you can authenticate to it with a cloud Redis provider. This allows you to rotate credentials without relying on static passwords.

The following providers are supported:

• AWS ElastiCache
• Azure Managed Redis
• Google Cloud Memorystore (with or without Valkey)

Each provider also supports an instance and cluster configuration.

config:
  vectordb:
    strategy: redis
    redis:
      host: $INSTANCE_ADDRESS
      username: $INSTANCE_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: aws
        aws_cache_name: $AWS_CACHE_NAME
        aws_is_serverless: false
        aws_region: $AWS_REGION
        aws_access_key_id: $AWS_ACCESS_KEY_ID
        aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
  vectordb:
    strategy: redis
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 6379
      username: $CLUSTER_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: aws
        aws_cache_name: $AWS_CACHE_NAME
        aws_is_serverless: false
        aws_region: $AWS_REGION
        aws_access_key_id: $AWS_ACCESS_KEY_ID
        aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
  vectordb:
    strategy: redis
    redis:
      host: $INSTANCE_ADDRESS
      username: $INSTANCE_USERNAME
      port: 10000
      cloud_authentication:
        auth_provider: azure
        azure_client_id: $AZURE_CLIENT_ID
        azure_client_secret: $AZURE_CLIENT_SECRET
        azure_tenant_id: $AZURE_TENANT_ID

config:
  vectordb:
    strategy: redis
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 10000
      username: $CLUSTER_USERNAME
      port: 10000
      cloud_authentication:
        auth_provider: azure
        azure_client_id: $AZURE_CLIENT_ID
        azure_client_secret: $AZURE_CLIENT_SECRET
        azure_tenant_id: $AZURE_TENANT_ID

config:
  vectordb:
    strategy: redis
    redis:
      host: $INSTANCE_ADDRESS
      port: 6379
      cloud_authentication:
        auth_provider: gcp
        gcp_service_account_json: $GCP_SERVICE_ACCOUNT

config:
  vectordb:
    strategy: redis
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 6379
      port: 6379
      cloud_authentication:
        auth_provider: gcp
        gcp_service_account_json: $GCP_SERVICE_ACCOUNT

Once you’ve configured your vector database and ingested content, you can control which AI Consumers access specific knowledge base articles and refine query results using metadata filters.

Collections

A collection is a logical grouping of knowledge base articles with independent access control rules. When you ingest content via the Konnect API, assign it to a collection using the collection field in the metadata.

Example metadata structure:

{
  "content": "Quarterly revenue increased 15%...",
  "metadata": {
    "collection": "finance-reports",
    "date": "2023-10-14",
    "tags": ["finance", "quarterly"],
    "source": "internal"
  }
}

Configuration

Two independent mechanisms control which results consumers receive:

• ACL filtering: Server restricts collections based on AI Consumer Groups
• Metadata filtering: Clients specify criteria (tags, dates, sources) to narrow results within authorized collections

This example assumes you have already created AI Consumer Groups (finance, admin, contractor) and configured the Key Authentication Strategy for your AI Consumers.

Environment variables

Set the following environment variables before deploying:

• OPENAI_API_KEY: Your OpenAI API key
• DB_PASSWORD: Your PostgreSQL database password

Never hardcode credentials in your policy configuration. Always use environment variables or secrets.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "type": "ai-rag-injector",
      "name": "finance-db",
      "display_name": "Finance DB",
      "config": {
        "consumer_identifier": "consumer_group",
        "global_acl_config": {
          "allow": [],
          "deny": []
        },
        "collection_acl_config": {
          "finance-reports": {
            "allow": [
              "finance",
              "admin"
            ],
            "deny": [
              "contractor"
            ]
          },
          "public-docs": {
            "allow": [],
            "deny": []
          }
        },
        "embeddings": {
          "model": {
            "name": "text-embedding-3-small",
            "provider": "openai"
          },
          "auth": {
            "header_name": "Authorization",
            "header_value": "Bearer ${OPENAI_API_KEY}"
          }
        },
        "vectordb": {
          "strategy": "pgvector",
          "dimensions": 1536,
          "distance_metric": "cosine",
          "pgvector": {
            "host": "localhost",
            "port": 5432,
            "user": "postgres",
            "password": "${DB_PASSWORD}",
            "database": "kong-pgvector"
          }
        }
      }
    }
    '

This configuration creates the following access rules:

• finance-reports: Accessible only to AI Consumers in the finance or admin groups. Contractors are explicitly denied.
• public-docs: Accessible to all AI Consumers (empty allow and deny lists).
• Other collections: No access (empty global ACL means deny by default).

ACL evaluation

The AI Policy checks access in this order:

1. Deny list: If subject matches, deny access
2. Allow list: If list exists and subject doesn’t match, deny access
3. Empty ACL: If both lists are empty, allow access

Collections with their own ACL in collection_acl_config ignore global_acl_config entirely. They must explicitly list all allowed subjects.

Metadata filtering

LLM clients can refine search results by specifying filter criteria in the query request. Filters apply within the collections. The AI RAG Injector Policy uses a Bedrock-compatible filter grammar with the following operators:

• equals: Exact match
• greaterThan: Greater than (>)
• greaterThanOrEquals: Greater than or equal to (>=)
• lessThan: Less than (<)
• lessThanOrEquals: Less than or equal to (<=)
• in: Match any value in array
• andAll: Combine multiple filter clauses

You can combine multiple conditions with andAll:

{
  "andAll": [
    {"equals": {"key": "source", "value": "internal"}},
    {"in": {"key": "tags", "value": ["finance", "quarterly"]}},
    {"greaterThanOrEquals": {"key": "date", "value": "2023-01-01"}}
  ]
}

Filter parameters:

You can include filters in the ai-rag-injector parameter of your request:

curl "http://localhost:8000/" \
     -H "Content-Type: application/json" \
     --json '{
       "messages": [
         {
           "role": "user",
           "content": "What were Q4 results?"
         }
       ],
       "ai-rag-injector": {
         "filters": {
           "andAll": [
             {
               "equals": {
                 "key": "source",
                 "value": "internal"
               }
             },
             {
               "in": {
                 "key": "tags",
                 "value": [
                   "q4",
                   "quarterly"
                 ]
               }
             }
           ]
         },
         "filter_mode": "strict",
         "stop_on_filter_error": false
       }
     }'

Query flow

The following diagram shows how ACL and metadata filtering work together during query processing:

 
flowchart TB
    Start([Query Request]) --> Auth[Authenticate AI Consumer]
    Auth --> CheckACL{Authorized
Collections?}
    CheckACL -->|No| Deny[❌ Access Denied]
    CheckACL -->|Yes| HasFilter{Metadata
Filters
Specified?}
    HasFilter -->|No| SearchAll[Search all chunks
in authorized collections]
    HasFilter -->|Yes| FilterMode{filter_mode
setting?}
    FilterMode -->|compatible| SearchCompat[Return chunks matching filter
OR chunks with no metadata]
    FilterMode -->|strict| SearchStrict[Return only chunks
matching filter]
    SearchAll --> Return[✓ Return Results]
    SearchCompat --> Return
    SearchStrict --> Return