Reduce prompt size with the AI Prompt Compressor Policy

Uses: AI Gateway kongctl

Incompatible with

on-prem

Prerequisites

AI Gateway running

This is a Konnect tutorial and requires a Konnect personal access token.

Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
Export your token to an environment variable:
```
export KONNECT_TOKEN='YOUR_KONNECT_PAT'
```
Copied!
Run the AI Gateway quickstart script to automatically provision a control plane and data plane in Kong Konnect, and configure your environment:
curl -Ls https://get.konghq.com/ai | bash -s -- -k $KONNECT_TOKEN
Copied!

This sets up a AI Gateway control plane named ai-quickstart, provisions a local data plane, and prints out the following environment variables export:

export AI_GATEWAY_ID=your-gateway-id
export KONNECT_TOKEN=$KONNECT_TOKEN
export KONNECT_CONTROL_PLANE_NAME=ai-quickstart
export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
export KONNECT_PROXY_URL='http://localhost:8000'

Copied!

Copy and paste these into your terminal to configure your session.

Want a fully-hosted AI Gateway? See AI Gateway 2.0 on Dedicated Cloud Gateways.

kongctl v1.26.0+

This tutorial uses kongctl to manage Konnect resources programmatically. We recommend keeping kongctl up to date with the latest version (1.26.0).

Install kongctl from developer.konghq.com/kongctl.
Verify the installation:
```
kongctl version
```
Copied!

OpenAI API key

Create an OpenAI account.
Get an API key.

Export the API key as a variable:

 export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
 export OPENAI_AUTH_HEADER="Bearer $OPENAI_API_KEY"

Copied!

AI Prompt Compressor service

Kong distributes the AI Prompt Compressor service publicly on Docker Hub, under the kong/ai-compress-service repository. No authentication is required to pull it.

Run the service and publish port 8080:

docker run -d --name compress-service -p 8080:8080 kong/ai-compress-service:v0.0.6

Copied!

The service downloads and loads its model on the first request, so the first compression can take a few seconds.

Verify the service is running:

curl --fail-with-body http://localhost:8080/status

Copied!

For more detail, see the AI Prompt Compressor Policy documentation.

Create the AI Model Provider, AI Model, and AI Policies

Create an AI Model Provider, an AI Model, and an AI Prompt Compressor Policy that compresses prompts by ratio before the AI Model forwards them to OpenAI. Also create an AI Prompt Template Policy that wraps client input in <LLMLINGUA> tags, and a File Log Policy that records compression metrics for each request:

kongctl apply -f - --auto-approve --pat "$KONNECT_TOKEN" << 'EOF'
ai_gateway_model_providers:
  - ref: generic-openai
    name: generic-openai
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    type: openai
    config:
      auth:
        type: basic
        headers:
          - name: Authorization
            value: !secret {source: !env OPENAI_AUTH_HEADER}
ai_gateway_policies:
  - ref: my-ai-prompt-compressor-policy
    name: my-ai-prompt-compressor-policy
    display_name: my-ai-prompt-compressor-policy
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    type: ai-prompt-compressor
    enabled: true
    global: false
    config:
      compressor_type: rate
      compressor_url: http://host.docker.internal:8080
      compression_ranges:
        - min_tokens: 20
          max_tokens: 100
          value: 0.8
        - min_tokens: 100
          max_tokens: 1000000
          value: 0.3
      stop_on_error: true
  - ref: my-ai-prompt-template-policy
    name: my-ai-prompt-template-policy
    display_name: my-ai-prompt-template-policy
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    type: ai-prompt-template
    enabled: true
    global: false
    config:
      templates:
        - name: summarize
          template: |-
            {
              "messages": [
                {
                  "role": "user",
                  "content": "Summarize the following in one sentence: <LLMLINGUA>{{document}}</LLMLINGUA>"
                }
              ]
            }
  - ref: my-file-log
    name: my-file-log
    display_name: my-file-log
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    type: file-log
    enabled: true
    global: false
    config:
      path: /tmp/compressor.json
ai_gateway_models:
  - ref: my-gpt-4o
    name: my-gpt-4o
    display_name: my-gpt-4o
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    type: model
    enabled: true
    formats: [{ type: openai }]
    config:
      route:
        paths:
          - /
        model:
          body_param: model
          values:
            - my-gpt-4o
    capabilities: [generate]
    policies:
      - !ref my-ai-prompt-template-policy#name
      - !ref my-ai-prompt-compressor-policy#name
      - !ref my-file-log#name
    targets:
      - name: gpt-4o
        provider: generic-openai
        config:
          type: openai
EOF

Copied!

Clients send only the {{document}} value, so the tags always surround the client input and never the instruction. Prompts under 20 tokens aren’t compressed. Prompts of 20 to 100 tokens are compressed to 80% of their length, and longer prompts to 30%.

If your AI Gateway data plane doesn’t run in Docker, set compressor_url to http://localhost:8080 instead.

Validate configuration

Send a short prompt, then a long document through the template, and compare the usage.prompt_tokens value in each response.

A prompt under 20 tokens isn’t compressed, so usage.prompt_tokens stays small and unchanged:

curl -X POST "$KONNECT_PROXY_URL/chat/completions" \
     --no-progress-meter --fail-with-body  \
     -H "Content-Type: application/json"\
     -H "Authorization: $OPENAI_AUTH_HEADER" \
     --json '{
       "model": "my-gpt-4o",
       "messages": [
         {
           "role": "user",
           "content": "What is DNS?"
         }
       ]
     }'

Copied!

The response reports "prompt_tokens": 11.

Reference the template and pass the document as a property. The template wraps the document in <LLMLINGUA> tags, so the Policy compresses the document and leaves the instruction as is:

curl -X POST "$KONNECT_PROXY_URL/chat/completions" \
     --no-progress-meter --fail-with-body  \
     -H "Content-Type: application/json"\
     -H "Authorization: $OPENAI_AUTH_HEADER" \
     --json '{
       "model": "my-gpt-4o",
       "messages": "{template://summarize}",
       "properties": {
         "document": "Kong Gateway is an API gateway that sits in front of upstream services and handles authentication, rate limiting, logging, and traffic control through a plugin architecture. Teams deploy it as a single entry point for their APIs, where it routes requests to the correct service, enforces security policies, and records detailed analytics for every call. Plugins extend the gateway without changes to the upstream services, so teams can add new capabilities such as caching, request transformation, and observability on their own schedule. The control plane distributes configuration to data planes that run close to the services, which keeps latency low and lets each environment scale independently of the others. Operators manage the same gateway across cloud, on-premises, and Kubernetes environments with one consistent configuration model."
       }
     }'

Copied!

Without compression, the document is about 150 tokens. The response reports roughly 60 prompt_tokens, and the model still returns an accurate summary. See Check compression metrics for the exact counts.

Check compression metrics

The File Log Policy writes each request to /tmp/compressor.json, and the AI Prompt Compressor Policy records its results in the ai.compressor object of every entry. For the full list of fields, see the audit log reference.

Read the log from inside your AI Gateway Docker container to show the compression metrics of the most recent request:

docker exec kong-ai-quickstart-gateway cat /tmp/compressor.json \
  | jq -s 'map(.ai.compressor | select(.)) | last'

Copied!

The following output for the long document shows the token counts before and after compression:

{
  "compress_items": [
    {
      "information": "Compression was performed and saved 103 tokens",
      "original_token_count": 147,
      "compress_token_count": 44,
      "msg_id": 1,
      "compressor_model": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
      "compress_type": "rate",
      "compress_value": 0.3,
      "save_token_count": 103
    }
  ],
  "duration": 681,
  "compress_token_count": 44,
  "save_token_count": 103,
  "compressed_message_count": 1,
  "compress_type": "rate",
  "provider": "kong",
  "original_token_count": 147
}

The top-level original_token_count, compress_token_count, and save_token_count fields total the tokens across all compressed messages. The prompt shrank from 147 to 44 tokens, a saving of 103 tokens. After the short prompt, the same command reports save_token_count: 0 and an information message that explains why compression was skipped.

Cleanup

Remove the AI Prompt Compressor service

docker rm -f compress-service

Copied!

Clean up AI Gateway resources

To clean up all AI Gateway resources created in this guide, run:

curl -Ls https://get.konghq.com/ai | bash -s -- -d

Copied!

Reduce prompt size with the AI Prompt Compressor Policy

Prerequisites

AI Gateway running

kongctl v1.26.0+

OpenAI API key

AI Prompt Compressor service

Create the AI Model Provider, AI Model, and AI Policies

Validate configuration

Check compression metrics

Cleanup

Remove the AI Prompt Compressor service

Clean up AI Gateway resources

Help us make these docs great!

Still need help?