Kong authenticates the request on the public AI Model. The Datakit Policy on that AI Model estimates the prompt’s token count, calls the small or the large AI Model on the same data plane, and returns that response to the client.
sequenceDiagram
participant C as Client
participant K as Kong AI Gateway (/custom-llm-routing)
participant T as Small or large AI Model
participant O as OpenAI
C->>K: POST /custom-llm-routing (apikey, model auto)
activate K
K->>K: Key Auth AI Auth Strategy validates apikey (else 401)
K->>K: Datakit Policy estimates tokens and picks small or large
K->>T: Datakit call to /custom-llm-routing-small or /custom-llm-routing-large
activate T
T->>T: Key Auth AI Auth Strategy validates the forwarded apikey (else 401)
T->>O: Request with the AI Model Provider's credentials
activate O
O-->>T: Completion
deactivate O
T-->>K: OpenAI-format response
deactivate T
K-->>C: Datakit exit returns the response with X-LLM-Tier and X-Prompt-Est-Tokens
deactivate KKong selects the AI Model for a request before any AI Policy runs, so a policy that rewrites the model field can’t move a request to another AI Model. The Datakit Policy works around this: it calls the other AI Models over HTTP and returns their response with an exit node.
Authentication is an AI Auth Strategy, not an AI Policy. The Key Auth AI Auth Strategy reads the apikey header and matches the key against the AI Consumer credentials. On a match, Kong attaches the AI Consumer to the request. The AI Consumer is the identity that rate limits, ACLs, and analytics use. All three AI Models reference this AI Auth Strategy in access.auth_strategies, so the Datakit Policy forwards the client’s key on each internal call.
The following settings configure the Key Auth AI Auth Strategy and the AI Consumer:
|
Setting
|
Purpose
|
key_names: [apikey]
|
Kong reads the key from the apikey header. Key Auth compares the whole header value, so it can’t read a key from an Authorization: Bearer header. The note in Run the demo shows the workaround.
|
hide_credentials: false
|
Kong keeps the apikey header on the request, because the Datakit Policy reads it and forwards it on the internal calls. With true, Kong removes the key before the Datakit Policy runs, and the internal calls fail with 401. OpenAI also receives the apikey header, but the AI Model Provider replaces the Authorization header with its own credentials.
|
credentials[].api_key
|
The AI Consumer’s key. The field is write-only, so it takes a !secret reference to CONSUMER_API_KEY, even for a demo value.
|
Kong adds X-Consumer-Username and related headers to the upstream request, not to the response. Per-consumer usage appears in Konnect analytics.
The Datakit Policy is attached to the public AI Model and runs the routing workflow as a graph of nodes. Its call nodes send requests to the data plane’s own proxy listener at http://localhost:8000, at each internal AI Model’s path plus /chat/completions, so each internal call goes through the same authentication as a client request.
The following nodes make up the workflow:
|
Setting
|
Purpose
|
EXTRACT_PROMPT, BODY_SMALL, BODY_LARGE
|
jq nodes that build each internal request body: the client’s messages array and the name of the AI Model to call. The internal AI Models have no config.route.model, so each one accepts only its own name in model. The other fields of the client request, for example temperature or max_tokens, aren’t forwarded.
|
EXTRACT_AUTH
|
A jq node that copies the apikey header from the request headers, so the internal calls authenticate as the same AI Consumer.
|
ESTIMATE_TIER
|
A jq node that counts the characters in each message’s content: the string, or the text of each part when content is an array. It estimates tokens as ceil(characters / 4), with a minimum of 1, and picks large at 80 estimated tokens or more. jq’s length counts Unicode code points, not bytes. The estimate isn’t a tokenizer count. For exact counts, use a tokenizer such as OpenAI’s tiktoken.
|
BUILD_DIAG_HEADERS
|
A jq node that turns the estimate into the X-LLM-Tier and X-Prompt-Est-Tokens response headers.
|
IS_LARGE, ROUTE_DECISION
|
IS_LARGE turns the tier into a boolean. The branch node runs CALL_LARGE and RESPOND_LARGE when it’s true, and CALL_SMALL and RESPOND_SMALL otherwise.
|
CALL_SMALL, CALL_LARGE
|
call nodes that send BODY_SMALL or BODY_LARGE and the key to /custom-llm-routing-small/chat/completions or /custom-llm-routing-large/chat/completions. Without the /chat/completions operation path, no route matches the call.
|
RESPOND_SMALL, RESPOND_LARGE
|
exit nodes that return the internal AI Model’s response body, with the headers from BUILD_DIAG_HEADERS, to the client. They don’t pass on the internal AI Model’s response headers. AI Gateway then doesn’t call the public AI Model’s own target.
|
The AI Model Provider holds the OpenAI API key and adds it to each request to OpenAI. Clients call the public AI Model, custom-llm-routing-chat, on /custom-llm-routing. The small and large AI Models exist for the Datakit Policy, but a client with a valid key can also call them directly, because the internal calls use the client’s own key.
The following settings configure the AI Model Provider and the AI Models:
|
Setting
|
Purpose
|
config.auth (AI Model Provider)
|
The OpenAI API key, from OPENAI_TOKEN. Kong adds it to each request to OpenAI. The field is write-only, so Konnect never returns it.
|
targets[].name (small and large AI Models)
|
The upstream models, from CHAT_MODEL_1 and CHAT_MODEL_2. Change them and re-apply to swap models without changing clients.
|
config.route.model (public AI Model)
|
Kong reads the model field of the request body (body_param: model) and matches it against auto. Another value matches no AI Model and returns 500 with failed the initial dns/balancer resolve for 'ai-gateway.upstream.local'.
|
policies (public AI Model)
|
Attaches the Datakit Policy to the public AI Model only. The internal calls don’t run the workflow again.
|
targets (public AI Model)
|
An AI Model needs at least one target. The exit nodes return before Kong uses it.
|
config.route.paths
|
/custom-llm-routing for the public AI Model, and /custom-llm-routing-small and /custom-llm-routing-large for the internal ones. The generate capability adds /chat/completions to each path.
|
config.response_streaming: deny
|
Clients can’t set stream=true on any of the three AI Models. The Datakit Policy needs the complete response body to return it.
|
config.max_request_body_size: 10485760
|
Accepts request bodies up to 10 MB, instead of the 8 MB default.
|
config.route.protocols: [http, https]
|
Accepts HTTP, so you can call the local data plane at http://localhost:8000, and so the Datakit Policy can call the internal AI Models over HTTP.
|
config.route.strip_path: true
|
Kong removes the route path before it forwards the request upstream.
|
config.logging.payloads: true
|
Kong logs request and response payloads. The default is false.
|
Production credentials: This cookbook stores the AI Consumer’s API key and the OpenAI API key in environment variables for simplicity. In production, use AI Vaults to reference them from a secret manager, for example AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, or Azure Key Vault.
The Datakit Policy adds the following headers to each routed response:
|
Header
|
Description
|
X-LLM-Tier
|
The tier that the Datakit Policy picked: small or large.
|
X-Prompt-Est-Tokens
|
The estimated token count of the prompt, as ceil(characters / 4).
|