demo.py sends a chat completion with an invalid key (expecting 401), a request with a spoofed x-user-id header (expecting Kong to drop it), and three chat turns split across two Phoenix Sessions using x-correlation-id. Tracing doesn’t change any HTTP response, so the script can’t confirm what Phoenix rendered. Open Phoenix after the run to see the traces.
The demo passes the API key via default_headers because the OpenAI SDK reserves api_key for the Authorization: Bearer header. To let clients pass the key through api_key directly, attach a Pre-Function Policy that copies the Bearer token to the apikey header server-side.
Create the demo script:
cat <<'EOF' > demo.py
"""LLM tracing with Arize Phoenix demo. See README for context.
Sends chat completions through Kong AI Gateway's `/llm-tracing-arize-phoenix`
Model and prints what each request proves about the tracing setup:
- An invalid `apikey` is rejected with 401 before any provider call (key-auth).
- Two turns share one `x-correlation-id` so they land in the same Phoenix
Session; a third omits the header so Kong falls back to the request id
(Request Transformer Advanced + the Post-Function Session read).
- A client-supplied `x-user-id` is dropped; the Consumer identity
(`phoenix-client`) is what Phoenix shows as `user.id`.
- Token usage and the resolved model come back as Kong response headers
(`x-kong-llm-model`, `x-kong-llm-tokens-*`), the same numbers the
`gen_ai.*` span attributes carry into Phoenix.
This script cannot confirm what Phoenix rendered. Post-Function writes
OpenInference attributes onto Kong's spans; whether Sessions, the user
column, Input/Output, and Messages populated correctly is only visible in
the Phoenix UI. Run `python demo.py`, then open the project named by
`PHOENIX_PROJECT_NAME` in Phoenix and confirm what the printed Session
ID and turns look like there.
Requires: `pip install -r requirements.txt`. Expects `PROXY_URL` (default
`http://localhost:8000`) and `CHAT_MODEL` (default `gpt-4o-mini`) as needed.
"""
import os
import sys
import time
import uuid
from openai import APIStatusError, OpenAI
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
PROXY_URL = os.getenv("PROXY_URL", "http://localhost:8000")
CHAT_MODEL = os.getenv("CHAT_MODEL", "gpt-4o-mini")
# Matches the Consumer credential's CONSUMER_API_KEY value from the Apply section.
PHOENIX_API_KEY = os.getenv("PHOENIX_API_KEY", "demo-api-key")
PHOENIX_PROJECT_NAME = os.getenv("PHOENIX_PROJECT_NAME", "kong-ai-gateway")
SESSION_ID = os.getenv("SESSION_ID", f"phoenix-demo-{uuid.uuid4().hex[:12]}")
# ANSI color codes. Disabled when stdout isn't a TTY or NO_COLOR is set.
_USE_COLOR = sys.stdout.isatty() and "NO_COLOR" not in os.environ
def _c(code: str, s: str) -> str:
return f"\033[{code}m{s}\033[0m" if _USE_COLOR else s
BOLD = lambda s: _c("1", s)
DIM = lambda s: _c("2", s)
GREEN = lambda s: _c("32", s)
BLUE = lambda s: _c("34", s)
CYAN = lambda s: _c("36", s)
RED = lambda s: _c("31", s)
MAGENTA = lambda s: _c("35", s)
# ---------------------------------------------------------------------------
# Kong-pointed client (key-auth via apikey header in default_headers)
# ---------------------------------------------------------------------------
#
# The OpenAI SDK reserves api_key for the Authorization: Bearer header. Kong
# reads the recipe's key-auth credential from the apikey header instead, so
# it's passed through default_headers. x-correlation-id is set per call
# below so each turn can vary it.
def make_client(correlation_id=None, api_key=PHOENIX_API_KEY):
headers = {"apikey": api_key}
if correlation_id is not None:
headers["x-correlation-id"] = correlation_id
return OpenAI(
base_url=f"{PROXY_URL}/llm-tracing-arize-phoenix",
api_key="placeholder",
default_headers=headers,
)
# ---------------------------------------------------------------------------
# Negative path: invalid key-auth credential
# ---------------------------------------------------------------------------
def check_auth_boundary():
"""Send a request with an invalid key, expecting Kong to return 401.
Confirms key-auth is enforced before any upstream call, so an
unauthenticated caller never reaches OpenAI or produces a trace.
"""
bad_client = make_client(api_key="not-a-real-key")
print(f" Attempting apikey={RED('not-a-real-key')!r}")
try:
bad_client.chat.completions.create(
model=CHAT_MODEL,
messages=[{"role": "user", "content": "ping"}],
)
except APIStatusError as exc:
print(f" {GREEN(BOLD('[BLOCKED]'))} {RED(BOLD(str(exc.status_code)))} {exc.message[:80]}")
return exc.status_code
print(f" {RED(BOLD('[AUTH]'))} unexpected: invalid key was accepted")
return None
# ---------------------------------------------------------------------------
# Client-supplied identity is dropped
# ---------------------------------------------------------------------------
def send_spoofed_user_header():
"""Send a request with a client-chosen x-user-id header.
Request Transformer Advanced removes x-user-id before the request
reaches the AI Model or Post-Function, so this value never becomes
Phoenix's user.id. The Consumer identity (phoenix-client) does.
"""
client = make_client(correlation_id=SESSION_ID)
completion = client.chat.completions.with_raw_response.create(
model=CHAT_MODEL,
messages=[{"role": "user", "content": "Who am I talking to?"}],
extra_headers={"x-user-id": "admin"},
)
parsed = completion.parse()
print(f" Sent header {DIM('x-user-id: admin')} (client-supplied, dropped by Kong)")
print(f" {CYAN(BOLD('[LLM]'))} \"{parsed.choices[0].message.content}\"")
print(f" {DIM('Phoenix user.id will read the Consumer identity instead: ')}{BLUE(BOLD('phoenix-client'))}")
# ---------------------------------------------------------------------------
# Session grouping via x-correlation-id
# ---------------------------------------------------------------------------
def chat_turn(client, message, label):
start = time.perf_counter()
raw = client.chat.completions.with_raw_response.create(
model=CHAT_MODEL,
messages=[{"role": "user", "content": message}],
)
elapsed = time.perf_counter() - start
completion = raw.parse()
reply = completion.choices[0].message.content
usage = completion.usage
kong_model = raw.headers.get("x-kong-llm-model", CHAT_MODEL)
print(f"\n{BOLD(label)}")
print(f" {CYAN(BOLD('[LLM]'))} \"{reply}\"")
stats = (
f"Model: {kong_model} "
f"Tokens: {usage.prompt_tokens} in / {usage.completion_tokens} out "
f"({elapsed:.3f}s)"
)
print(f" {DIM(stats)}")
def run_session_grouping_demo():
"""Two turns share SESSION_ID; a third omits x-correlation-id entirely.
The first two land in the same Phoenix Session. The third has no
x-correlation-id, so Post-Function falls back to
kong.request.get_id() and appears as its own Session.
"""
session_client = make_client(correlation_id=SESSION_ID)
chat_turn(session_client, "What is Kong AI Gateway?", "Turn 1 (session: " + SESSION_ID + ")")
chat_turn(session_client, "Summarize that in five words.", "Turn 2 (session: " + SESSION_ID + ")")
no_session_client = make_client(correlation_id=None)
chat_turn(no_session_client, "What is Arize Phoenix?", "Turn 3 (no x-correlation-id -> falls back to request id)")
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
if __name__ == "__main__":
print(BOLD("LLM Tracing with Arize Phoenix Demo"))
print("=" * 60)
print(f"Kong Proxy: {PROXY_URL}")
print(f"Chat Model: {CYAN(BOLD(CHAT_MODEL))}")
print(f"Session ID: {BLUE(BOLD(SESSION_ID))}")
print(f"\n{DIM('Checking key-auth boundary...')}")
check_auth_boundary()
print(f"\n{DIM('Checking that a client-supplied x-user-id is dropped...')}")
send_spoofed_user_header()
print(f"\n{DIM('Running session-grouping turns...')}")
run_session_grouping_demo()
print()
print("=" * 60)
print(f"{BOLD('Done.')} Open Phoenix -> project {MAGENTA(BOLD(PHOENIX_PROJECT_NAME))} -> Sessions -> {BLUE(BOLD(SESSION_ID))}")
print("Confirm: user column reads 'phoenix-client', Input/Output are populated,")
print("and the LLM span's Messages view shows role/content bubbles for each turn.")
sys.exit(0)
EOF
Run it:
Example output:
LLM Tracing with Arize Phoenix Demo
============================================================
Kong Proxy: http://localhost:8000
Chat Model: gpt-4o-mini
Session ID: phoenix-demo-a1b2c3d4e5f6
Checking key-auth boundary...
Attempting apikey='not-a-real-key'
[BLOCKED] 401 Unauthorized
Checking that a client-supplied x-user-id is dropped...
Sent header x-user-id: admin (client-supplied, dropped by Kong)
[LLM] "I don't have information identifying who you are..."
Phoenix user.id will read the Consumer identity instead: phoenix-client
Running session-grouping turns...
Turn 1 (session: phoenix-demo-a1b2c3d4e5f6)
[LLM] "Kong AI Gateway is a layer that manages, secures, and observes traffic to LLM providers."
Model: gpt-4o-mini Tokens: 14 in / 19 out (0.812s)
Turn 2 (session: phoenix-demo-a1b2c3d4e5f6)
[LLM] "A managed proxy for LLM traffic."
Model: gpt-4o-mini Tokens: 38 in / 8 out (0.431s)
Turn 3 (no x-correlation-id -> falls back to request id)
[LLM] "Arize Phoenix is an open-source LLM observability and evaluation platform."
Model: gpt-4o-mini Tokens: 13 in / 15 out (0.734s)
============================================================
Done. Open Phoenix -> project kong-ai-gateway -> Sessions -> phoenix-demo-a1b2c3d4e5f6
Confirm: user column reads 'phoenix-client', Input/Output are populated,
and the LLM span's Messages view shows role/content bubbles for each turn.
- The first call sent
apikey: not-a-real-key. The Key Auth AI Auth Strategy rejected it with 401 before the request reached the AI Model or OpenAI, so Kong never created a span for it.
- The second call sent
x-user-id: admin alongside a valid key. Request Transformer Advanced dropped that header before Post-Function ran, so the root span’s user.id reads the AI Consumer identity phoenix-client, not the client-supplied value.
- The next two calls reused
SESSION_ID as x-correlation-id. Post-Function reads that header into session.id on the root span, so both turns group into one Phoenix Session.
- The final call omitted
x-correlation-id. Post-Function fell back to kong.request.get_id(), so this turn lands in its own Session.
- Every call’s response headers carried
x-kong-llm-model and the OpenAI SDK’s usage object carried token counts. Those same numbers are what gen_ai.usage.* attributes carry into the Phoenix LLM span.
A request/response pair looks like this over the wire:
{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "What is Kong AI Gateway?"}]
}
{
"id": "chatcmpl-...",
"choices": [
{
"message": {
"role": "assistant",
"content": "Kong AI Gateway is a layer that manages, secures, and observes traffic to LLM providers."
}
}
],
"usage": {"prompt_tokens": 14, "completion_tokens": 19, "total_tokens": 33}
}
Post-Function writes the request body verbatim onto input.value and the response body onto output.value, and walks messages[] / choices[].message into the llm.input_messages.* / llm.output_messages.* keys Phoenix Messages reads.
Open the project named in PHOENIX_PROJECT_NAME (default kong-ai-gateway).
You should see:
- A Kong root span with kind CHAIN, user
phoenix-client, two Sessions (one grouping turns 1 and 2, one holding turn 3 alone), and JSON Input/Output on the spans table
- An LLM child span for the AI Model hop, with model and token counts from
gen_ai.*
- Messages on the LLM span (role and content bubbles), not only a Raw JSON blob
- Span status OK for HTTP 200, and ERROR on the auth-rejected call if it ever produced a span
If the root row shows -- for Input/Output, the listed span is missing input.value / output.value. Confirm Post-Function is attached after the header-normalize policy and that request buffering is enabled.
If Messages is empty, confirm you opened the LLM span, not the CHAIN root. CHAIN ignores llm.input_messages.*.
Open Konnect and find the AI Gateway named llm-tracing-arize-phoenix-recipe. The recipe created an AI Model Provider holding your OpenAI credentials, an AI Model (llm-tracing-arize-phoenix-model) with the Request Transformer Advanced, OpenTelemetry, and Post-Function Policies attached, a Key Auth AI Auth Strategy, and an AI Consumer (phoenix-client) with a demo API key credential, all scoped to this recipe by the kongctl namespace applied above.
For platform-wide traffic analysis across every AI Gateway, open the Observability L1 menu in Konnect.