Send one request to each endpoint to make sure that both endpoints work. Then run the demo script. The demo script sends an agent conversation with a large tool result to both endpoints and compares the input tokens.
Send a short request to /uncompressed. The model field is the name of the AI Model on that path:
curl -s -X POST "$KONNECT_PROXY_URL/uncompressed/chat/completions" \
-H "Content-Type: application/json" \
--json '{"model":"uncompressed","messages":[{"role":"user","content":"Reply with exactly: pong"}]}'
The response is an OpenAI chat completion. The following example shows the main fields of the response:
{
"model": "gpt-4.1-mini-2025-04-14",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "pong"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 12,
"completion_tokens": 1,
"total_tokens": 13
}
}
The model field in the response shows gpt-4.1-mini-2025-04-14 instead of gpt-4.1-mini. This difference is expected. OpenAI serves gpt-4.1-mini from a dated snapshot and returns the name of the snapshot. The OpenAI model page lists gpt-4.1-mini-2025-04-14 as the default snapshot. To see the model that Kong sent the request to, add -i to the curl command. The X-Kong-LLM-Model response header shows openai/gpt-4.1-mini.
Send the same request to /compressed. Kong sends this request to Headroom before it sends the request to OpenAI. The x-session-id header assigns a Headroom session to the conversation:
curl -s -X POST "$KONNECT_PROXY_URL/compressed/chat/completions" \
-H "Content-Type: application/json" \
-H "x-session-id: readme-example" \
--json '{"model":"compressed","messages":[{"role":"user","content":"Reply with exactly: pong"}]}'
{
"model": "gpt-4.1-mini-2025-04-14",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "pong"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 12,
"completion_tokens": 1,
"total_tokens": 13
}
}
Both responses report 12 input tokens in prompt_tokens, because a one-line message has nothing to compress. In the terminal where Headroom runs, a log line with path=/v1/compress appears for the request to /compressed. No log line appears for the request to /uncompressed.
If the model value doesn’t match the AI Model on the path, Kong returns 500. For example, Kong returns the following error when you send "model":"gpt-4.1-mini" to /compressed:
{
"message":"failed the initial dns/balancer resolve for 'ai-gateway.upstream.local' with: dns server error: 2 server failure",
"request_id":"3b7338565458882c60517341c9a8504f"
}
The demo script sends one agent conversation to both endpoints and compares the number of input tokens that OpenAI counts for each request. Input tokens are the tokens of everything that the model reads for a request, which is the whole conversation. OpenAI bills for every input token and reports the count as usage.prompt_tokens in each response. Output tokens are the tokens of the model’s answer, reported as usage.completion_tokens. Headroom compresses only the input.
The demo conversation is the conversation that an agent sends to the model after the agent runs a tool. The agent adds the tool result to the conversation. Then the agent sends the whole conversation with every later request. In this demo, the tool is get_logs, and its result is 200 sample log records that the script generates. The conversation contains the following four messages:
|
Message
|
Role
|
Content
|
|
1
|
user
|
Pull the last 200 log entries for our services.
|
|
2
|
assistant
|
A tool call: run get_logs with a limit of 200.
|
|
3
|
tool
|
The tool result: 200 log records as JSON, 25,540 characters.
|
|
4
|
user
|
Which log entries are errors? List each id, service, and message.
|
In step 1, the script sends the four messages to /uncompressed. Then the script sends the same four messages to /compressed:
sequenceDiagram
participant D as demo.py
participant K as Kong AI Gateway
participant H as Headroom
participant O as OpenAI
D->>K: POST /uncompressed with messages 1 to 4
K->>O: messages 1 to 4, unchanged
O-->>K: answer (10,272 input tokens)
K-->>D: answer
D->>K: POST /compressed with the same messages 1 to 4
K->>H: messages 1 to 4
Note over H: Rewrites the 200 log records in message 3 as a table.<br/>Leaves messages 1, 2, and 4 unchanged.
H-->>K: messages 1 to 4, with the table in message 3
K->>O: messages 1 to 4, with the table in message 3
O-->>K: answer (5,145 input tokens)
K-->>D: answerBoth requests contain the same conversation. On /compressed, OpenAI receives the log records as a table, so OpenAI counts about half as many input tokens. The copy of the conversation in the script doesn’t change.
Steps 2 and 3 cover two more cases:
- Step 2 asks a follow-up question in the same conversation. Like an agent, the script sends the whole conversation again, including the original 200 log records, the answer from step 1, and the new question. The request has the same
x-session-id header as step 1. So Headroom reuses the table from step 1 and doesn’t compress the records again.
- Step 3 starts a new conversation with one short question and no tool result. Headroom has nothing to compress, so OpenAI counts the same number of input tokens on both endpoints.
The script doesn’t need your OpenAI API key, because Kong adds the key from the AI Model Provider. The OpenAI SDK requires an api_key value, so the script passes a placeholder.
Save the script as demo.py:
"""Compare input tokens with and without Headroom compression on Kong AI Gateway.
The script plays the part of an agent that has used a tool named get_logs.
get_logs isn't a real tool: the script generates 200 log records as its result.
The script sends the same conversation to two endpoints and prints how many input
tokens OpenAI counted for each one. Input tokens are everything the model reads for a
request, the whole conversation; OpenAI reports them as usage.prompt_tokens.
/uncompressed forwards the conversation to OpenAI unchanged.
/compressed sends the conversation to Headroom first, then forwards Headroom's version.
It runs three steps:
Step 1 The conversation right after the tool ran: the user's request, the
assistant's tool call, the tool result (200 log records in JSON), and a
question about the records.
Expected: /compressed sends about half the input tokens, and its answer
lists the three error records.
Step 2 The next request in the same conversation, with the same session ID.
Expected: the savings carry over to the next turn, and the answer is correct.
Step 3 A new conversation with one short question and no tool result.
Expected: the same input tokens on both endpoints.
Run:
export PROXY_URL="$KONNECT_PROXY_URL"
python demo.py
"""
import json
import os
import random
import sys
import uuid
from openai import OpenAI
PROXY_URL = os.getenv("PROXY_URL", "http://localhost:8000")
USE_COLOR = sys.stdout.isatty() and "NO_COLOR" not in os.environ
def bold(text):
return f"\033[1m{text}\033[0m" if USE_COLOR else text
def ask(endpoint, messages, session_id):
"""Send the messages to /uncompressed or /compressed. Returns the response and Kong's latency."""
# Kong adds the OpenAI API key from the AI Model Provider, so this api_key is never used.
client = OpenAI(base_url=f"{PROXY_URL}/{endpoint}", api_key="unused")
raw = client.chat.completions.with_raw_response.create(
model=endpoint, # the AI Model's name, which matches its path
messages=messages,
temperature=0,
extra_headers={"x-session-id": session_id}, # Headroom keeps one session per ID
)
return raw.parse(), raw.headers.get("x-kong-proxy-latency", "?")
def run_step(title, messages, session_id):
"""Send the same messages to both endpoints and print the results. Returns the /compressed answer."""
print(f"\n{bold(title)}")
print(f" {'endpoint':<15}{'input tokens':>12}{'cached':>9}{'Kong time':>12}")
results = {}
for endpoint in ("uncompressed", "compressed"):
response, kong_ms = ask(endpoint, messages, session_id)
results[endpoint] = response
# OpenAI reports input tokens as prompt_tokens, and the cached part of them separately.
usage = response.usage
cached = usage.prompt_tokens_details.cached_tokens if usage.prompt_tokens_details else 0
print(f" {'/' + endpoint:<15}{usage.prompt_tokens:>12,}{cached:>9,}{kong_ms:>9} ms")
before = results["uncompressed"].usage.prompt_tokens
after = results["compressed"].usage.prompt_tokens
summary = f"Saved on /compressed: {before - after:,} input tokens ({(before - after) / before:.0%} fewer)"
print(f" {bold(summary)}")
answer = " ".join(results["compressed"].choices[0].message.content.split())
print(f" Answer from /compressed: {answer}")
return answer
# The get_logs tool result. The tool isn't real: the script generates 200 log records
# with the same six fields. Three of them are errors.
random.seed(7)
logs = []
for i in range(200):
logs.append({
"id": i,
"ts": f"2026-10-18T09:{i % 60:02d}:{i * 7 % 60:02d}Z",
"service": random.choice(["checkout", "auth", "search", "payments"]),
"level": "INFO",
"latency_ms": random.randint(18, 240),
"msg": "request completed",
})
logs[37].update(service="payments", level="ERROR", msg="card processor returned 502")
logs[118].update(service="checkout", level="ERROR", msg="upstream timeout after 30000ms")
logs[164].update(service="auth", level="ERROR", msg="connection reset by peer")
tool_result = json.dumps(logs)
# Step 1: the conversation right after the tool ran. The user asks for logs, the
# assistant calls get_logs, the tool result follows, and the user asks about it.
session_id = f"demo-{uuid.uuid4().hex[:8]}"
conversation = [
{"role": "user", "content": "Pull the last 200 log entries for our services."},
{"role": "assistant", "content": None, "tool_calls": [{
"id": "call_1", "type": "function",
"function": {"name": "get_logs", "arguments": '{"limit": 200}'},
}]},
{"role": "tool", "tool_call_id": "call_1", "content": tool_result},
{"role": "user", "content": "Which log entries are errors? List each id, service, and message."},
]
title = f"Step 1: conversation with a tool result of 200 log records ({len(tool_result):,} characters of JSON)"
answer = run_step(title, conversation, session_id)
errors = ", ".join(f"id {r['id']} {r['service']}" for r in logs if r["level"] == "ERROR")
print(f" Expected: {errors}")
# Step 2: the next request in the same conversation. The script sends every message
# again, the original JSON included, plus the answer and a new question.
conversation += [
{"role": "assistant", "content": answer},
{"role": "user", "content": "What is the latency_ms of entry 150, and which service logged it?"},
]
run_step("Step 2: follow-up question in the same conversation", conversation, session_id)
print(f" Expected: latency_ms {logs[150]['latency_ms']}, service {logs[150]['service']}")
# Step 3: a new conversation with one short message.
question = [{"role": "user", "content": "In one sentence, what does a tool call return to an agent?"}]
run_step("Step 3: new conversation with one short question", question, f"demo-{uuid.uuid4().hex[:8]}")
In the Python virtual environment from the prerequisites, set PROXY_URL to the data plane URL and run the script:
export PROXY_URL="$KONNECT_PROXY_URL"
python demo.py
For each step, the script prints one line per endpoint with the following columns:
input tokens: the number of input tokens that OpenAI counted for the request, from usage.prompt_tokens.cached: the number of input tokens that OpenAI served from its prompt cache.Kong time: the time in milliseconds that Kong spent on the request before Kong sent the request to OpenAI. The script reads this value from the X-Kong-Proxy-Latency response header. On /compressed, this time includes the call to Headroom.
Then the script prints the number of input tokens that /compressed saved and the answer from /compressed. For steps 1 and 2, the script also prints the correct answer for comparison. The following example shows the output:
Step 1: conversation with a tool result of 200 log records (25,540 characters of JSON)
endpoint input tokens cached Kong time
/uncompressed 10,272 0 8 ms
/compressed 5,145 0 112 ms
Saved on /compressed: 5,127 input tokens (50% fewer)
Answer from /compressed: Here are the log entries with errors, including their id, service, and message: - id: 37, service: payments, message: card processor returned 502 - id: 118, service: checkout, message: upstream timeout after 30000ms - id: 164, service: auth, message: connection reset by peer Let me know if you need any further details!
Expected: id 37 payments, id 118 checkout, id 164 auth
Step 2: follow-up question in the same conversation
endpoint input tokens cached Kong time
/uncompressed 10,375 0 8 ms
/compressed 5,248 4,992 24 ms
Saved on /compressed: 5,127 input tokens (49% fewer)
Answer from /compressed: The log entry with id 150 has a latency_ms of 146 and was logged by the search service.
Expected: latency_ms 146, service search
Step 3: new conversation with one short question
endpoint input tokens cached Kong time
/uncompressed 21 0 4 ms
/compressed 21 0 17 ms
Saved on /compressed: 0 input tokens (0% fewer)
Answer from /compressed: A tool call returns the tool's output or response data to the agent for further processing or decision-making.
-
In step 1, OpenAI counted 5,145 input tokens for the /compressed request and 10,272 for the /uncompressed request. The answer from /compressed still listed the three error records. Neither request used the prompt cache, so OpenAI billed 50% less for the input of the /compressed request. Headroom changed only message 3, the tool result. The script sent the tool result as JSON, 25,540 characters:
[{"id": 0, "ts": "2026-10-18T09:00:00Z", "service": "search", "level": "INFO", "latency_ms": 56, "msg": "request completed"}, {"id": 1, "ts": "2026-10-18T09:01:07Z", "service": "payments", "level": "INFO", "latency_ms": 184, "msg": "request completed"}, ...
OpenAI received the following text instead, 12,021 characters:
"[200]{id:int,latency_ms:int,level:string,msg:string,service:string,ts:string}\n0,56,INFO,request completed,search,2026-10-18T09:00:00Z\n1,184,INFO,request completed,payments,2026-10-18T09:01:07Z\n...
Headroom converted the array into a table: a header line with the field names and types, then one line per record, separated by \n. Each field name appears once instead of 200 times. All 200 records are present, including the three errors:
37,189,ERROR,card processor returned 502,payments,2026-10-18T09:37:19Z
118,67,ERROR,upstream timeout after 30000ms,checkout,2026-10-18T09:58:46Z
164,160,ERROR,connection reset by peer,auth,2026-10-18T09:44:08Z
- In step 2, OpenAI counted 5,248 input tokens for the
/compressed request and 10,375 for the /uncompressed request. The answer for entry 150 was correct. The request had the same session ID as step 1, so Headroom reused the table from step 1. As a result, messages 1 to 4 were identical to the messages in the step 1 request to /compressed. OpenAI served 4,992 of the 5,248 input tokens from its prompt cache. OpenAI doesn’t guarantee a cache hit on either endpoint. In this run, the /uncompressed request got no cache hit.
- In step 3, OpenAI counted 21 input tokens on both endpoints. Headroom doesn’t change the latest user message, and the request had nothing else to compress.
- Kong spent 112 ms on the first
/compressed request and 8 ms on the /uncompressed request. The difference includes the Headroom call. On the other /compressed requests, Kong spent 17 to 24 ms.
If you run the script again, step 1 can also show cached input tokens. The second run sends the same conversation as the first run, and OpenAI keeps cached prompts for up to an hour.
Open Konnect and find the AI Gateway named prompt-compression-with-headroom-and-kong-ai-gateway-recipe. The AI Gateway contains the AI Model Provider headroom-provider, the AI Prompt Compressor Policy headroom-compressor, and the AI Models compressed and uncompressed.
To list the AI Models with their paths and Policies, send the following request to the Konnect API:
curl -s "$KONNECT_CONTROL_PLANE_URL/v1/ai-gateways/$AI_GATEWAY_ID/models" \
-H "Authorization: Bearer $KONNECT_TOKEN" \
| jq -r '.data[] | "\(.name) path=\(.config.route.paths[0]) policies=\(.policies)"'
uncompressed path=/uncompressed policies=[]
compressed path=/compressed policies=["headroom-compressor"]
Kong AI Gateway records token consumption and request latency for every AI Model. For traffic analysis across every AI Gateway, go to the Observability menu in Konnect.