Custom LLM Routing

Minimum Version
AI Gateway - 2.0
Incompatible with
on-prem

Overview

Route each chat request to a small or a large OpenAI model based on the size of its prompt, so that short prompts don’t pay large-model prices and clients never choose a model. You’ll set up:

  • One /custom-llm-routing endpoint that accepts only requests with a valid AI Consumer API key
  • A workflow in AI Gateway that estimates the prompt’s token count and sends the prompt to the small or the large model
  • Response headers that show the estimate and the chosen tier

Prerequisites

This tutorial uses Kong Konnect. The AI Gateway quickstart script provisions a recipe-scoped AI Gateway and local data plane.

  1. Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
  2. Export your token. You reuse the same token later for kongctl commands:

    export KONNECT_TOKEN='YOUR_KONNECT_PAT'
  3. Set the recipe-scoped AI Gateway name and run the quickstart script:

    export KONNECT_CONTROL_PLANE_NAME='custom-llm-routing-recipe'
    curl -Ls https://get.konghq.com/ai | bash -s -- -k $KONNECT_TOKEN

    This provisions an AI Gateway named custom-llm-routing-recipe, a local data plane connected to it, and prints export lines for the rest of the session vars, including AI_GATEWAY_ID. Paste those into your shell when prompted.

This tutorial uses kongctl to manage AI Gateway configuration.

  1. Install kongctl from the kongctl documentation.
  2. Verify it’s installed:

    kongctl version

Export the credentials of the provider that the AI Models call. The provider’s tab in Apply the Kong configuration reads these variables.

The demo script requires Python 3.11 or later. Set up an isolated environment:

python3 -m venv .venv
source .venv/bin/activate
pip install 'openai>=1.0.0'

What this cookbook solves

Most teams send every prompt to one model, or let each client choose a model, which causes the following problems:

Problem

Solution

A large model charges the same per-token price for a one-line question as for a long document, so short prompts cost more than they need to. The Datakit Policy estimates the token count of each prompt and sends prompts under 80 estimated tokens to the small AI Model.
Long inputs, for example documents or long conversations, need a more capable model than short questions do. Prompts of 80 estimated tokens or more go to the large AI Model.
When clients choose the model, each team repeats the same size rule, and nothing stops a caller from always choosing the large model. Clients send model: auto to one endpoint. AI Gateway chooses the model, and the X-LLM-Tier and X-Prompt-Est-Tokens response headers show the decision.
Each service holds a copy of the OpenAI API key. The AI Model Provider keeps the key in Kong. Clients hold an AI Consumer key, and the Key Auth AI Auth Strategy rejects unknown keys with 401 before any upstream call.

Try it out

Apply the configuration, then run a demo script that sends a short prompt, a long prompt, and one request with an invalid API key to /custom-llm-routing.

Apply the Kong configuration

The configuration creates an AI Model Provider with your OpenAI API key, a Key Auth AI Auth Strategy, an AI Consumer with a demo API key, a Datakit Policy, and three AI Models: the public /custom-llm-routing AI Model and the small and large AI Models that it calls.

Adopt the quickstart AI Gateway into a kongctl namespace. Adoption adds the KONGCTL-namespace label to the AI Gateway:

kongctl adopt ai-gateway "${KONNECT_CONTROL_PLANE_NAME}" \
  --namespace "${KONNECT_CONTROL_PLANE_NAME}" \
  --pat "${KONNECT_TOKEN}"

Export the demo AI Consumer’s API key. The credential’s api_key field is write-only, so its value must come from an environment variable, even for a fixed demo key:

export CONSUMER_API_KEY='demo-api-key'

Select your provider. The tab exports CHAT_MODEL_1, the model of the small AI Model, and CHAT_MODEL_2, the model of the large AI Model, then applies the configuration. Provider credentials come from the prerequisites.

Run the demo

The script sends hi and a prompt of 400 characters, both with model: auto, then one request with an invalid API key. In the output, look for the tier and upstream model on each [ROUTED TO] line, and for the [BLOCKED] 401 line.

The demo passes the API key in default_headers because the OpenAI SDK always sends api_key as an Authorization: Bearer header. To let clients use api_key directly, add a global Pre-Function Policy (global: true) that copies the Bearer token to the apikey header. A global Pre-Function Policy runs before authentication, while one attached to the AI Model runs after it.

Create the demo script:

cat <<'EOF' > demo.py
"""Custom LLM routing demo for Kong AI Gateway.

Shows three things:
  1. A short prompt goes to the small model. The Datakit Policy estimates the
     prompt size and returns it in the X-Prompt-Est-Tokens and X-LLM-Tier headers.
  2. A long prompt goes to the large model. Both requests send model="auto",
     the alias of the public AI Model, so the client never chooses the model.
  3. A request with an invalid key gets 401 from Kong before any upstream call.

Expected output: a [ROUTED TO] line with tier=small for the short prompt and
tier=large for the long prompt, then a [BLOCKED] 401 line for the invalid key.

Run: python demo.py (set PROXY_URL if the data plane is not at http://localhost:8000,
and CONSUMER_API_KEY if you changed the demo key).
"""

import os
import sys

from openai import APIStatusError, OpenAI

PROXY_URL = os.getenv("PROXY_URL", "http://localhost:8000")
API_KEY = os.getenv("CONSUMER_API_KEY", "demo-api-key")
THRESHOLD = 80

_USE_COLOR = sys.stdout.isatty() and "NO_COLOR" not in os.environ


def _c(code: str, s: str) -> str:
    return f"\033[{code}m{s}\033[0m" if _USE_COLOR else s


def BOLD(s: str) -> str:
    return _c("1", s)


def DIM(s: str) -> str:
    return _c("2", s)


def GREEN(s: str) -> str:
    return _c("32", s)


def CYAN(s: str) -> str:
    return _c("36", s)


def RED(s: str) -> str:
    return _c("31", s)


def make_client(api_key: str) -> OpenAI:
    """Construct an OpenAI client that sends the given API key in the apikey header."""
    return OpenAI(
        base_url=f"{PROXY_URL}/custom-llm-routing",
        api_key="unused",
        default_headers={"apikey": api_key},
    )


def est_tokens(content: str) -> int:
    """Estimate tokens the way the Datakit Policy does: ceil(characters / 4), at least 1."""
    return max(1, (len(content) + 3) // 4)


def send(client: OpenAI, content: str):
    return client.chat.completions.with_raw_response.create(
        model="auto",
        messages=[{"role": "user", "content": content}],
    )


def call(client: OpenAI, label: str, content: str) -> None:
    """Send one chat request and print the tier and model that answered."""
    tokens = est_tokens(content)
    expected = "large" if tokens >= THRESHOLD else "small"
    print(f"\n{BOLD('[REQUEST]')} {label}")
    print(f"  {DIM(f'est_tokens={tokens} threshold={THRESHOLD} expected_tier={expected}')}")
    print(f"  {DIM(f'model=auto prompt_preview={content[:48]!r}')}")

    try:
        raw = send(client, content)
    except APIStatusError as e:
        print(f"{RED(BOLD('[ERROR]'))} {RED(BOLD(str(e.status_code)))} {e.message}")
        raise SystemExit(1) from e

    completion = raw.parse()
    tier = raw.headers.get("x-llm-tier", ".")
    est = raw.headers.get("x-prompt-est-tokens", ".")
    answer = (completion.choices[0].message.content or "")[:120]

    print(f"[RESPONSE] {DIM(answer)}")
    print(
        f"{GREEN(BOLD('[ROUTED TO]'))} tier={GREEN(BOLD(tier))} "
        f"est_tokens={est} upstream model={CYAN(BOLD(completion.model))}"
    )

    if tier != expected:
        print(f"{RED(BOLD('[MISMATCH]'))} expected tier={expected!r}, got={tier!r}")
        raise SystemExit(1)


def call_invalid_key() -> None:
    """Send one request with a key that no AI Consumer has."""
    bad_key = "not-a-real-key"
    print(f"\nTrying apikey={bad_key!r}")
    try:
        send(make_client(bad_key), "hi")
    except APIStatusError as e:
        print(f"{RED(BOLD('[BLOCKED]'))} {RED(BOLD(str(e.status_code)))}")
        return
    print(f"{RED(BOLD('[ALLOWED]'))} Kong accepted the invalid key")
    raise SystemExit(1)


def section(title: str) -> None:
    bar = "=" * 70
    print(f"\n{bar}\n{BOLD(title)}\n{bar}")


def main() -> None:
    client = make_client(API_KEY)

    section("1. Short prompt routes to the small model")
    call(client, "short", "hi")

    section("2. Long prompt routes to the large model")
    call(client, "long", "x" * 400)

    section("3. Invalid API key. Kong rejects before any upstream call")
    call_invalid_key()

    section("Done.")


if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        sys.exit(130)
EOF

Run it:

python demo.py

Example output:

======================================================================
1. Short prompt routes to the small model
======================================================================

[REQUEST] short
  est_tokens=1 threshold=80 expected_tier=small
  model=auto prompt_preview='hi'
[RESPONSE] Hello! How can I assist you today?
[ROUTED TO] tier=small est_tokens=1 upstream model=gpt-4o-mini-2024-07-18

======================================================================
2. Long prompt routes to the large model
======================================================================

[REQUEST] long
  est_tokens=100 threshold=80 expected_tier=large
  model=auto prompt_preview='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
[RESPONSE] It looks like your message contains a long string of "x" characters. Could you please provide more details or let me kno
[ROUTED TO] tier=large est_tokens=100 upstream model=gpt-4o-2024-08-06

======================================================================
3. Invalid API key. Kong rejects before any upstream call
======================================================================

Trying apikey='not-a-real-key'
[BLOCKED] 401

======================================================================
Done.
======================================================================

Check the results

The output confirms the following:

Behavior

Evidence

Routing by prompt size hi is estimated at 1 token and gets tier=small. The 400-character prompt is estimated at 100 tokens and gets tier=large. The upstream model is the model that OpenAI reports in the response body, so each prompt reached a different model.
The client doesn’t choose the model Both requests send model: auto, the alias of the public AI Model. The Datakit Policy sends each internal call the name of the AI Model that it calls, and that AI Model’s target sets the upstream model.
Authentication before routing The invalid key gets 401. Kong rejected the request before the Datakit Policy ran, so the request reached neither internal AI Model nor OpenAI.

To review the created entities, and per-consumer usage, open the custom-llm-routing-recipe AI Gateway in Konnect. For traffic across all your AI Gateways, use the Observability menu.

How it works

Kong authenticates the request on the public AI Model. The Datakit Policy on that AI Model estimates the prompt’s token count, calls the small or the large AI Model on the same data plane, and returns that response to the client.

sequenceDiagram
    participant C as Client
    participant K as Kong AI Gateway (/custom-llm-routing)
    participant T as Small or large AI Model
    participant O as OpenAI

    C->>K: POST /custom-llm-routing (apikey, model auto)
    activate K
    K->>K: Key Auth AI Auth Strategy validates apikey (else 401)
    K->>K: Datakit Policy estimates tokens and picks small or large
    K->>T: Datakit call to /custom-llm-routing-small or /custom-llm-routing-large
    activate T
    T->>T: Key Auth AI Auth Strategy validates the forwarded apikey (else 401)
    T->>O: Request with the AI Model Provider's credentials
    activate O
    O-->>T: Completion
    deactivate O
    T-->>K: OpenAI-format response
    deactivate T
    K-->>C: Datakit exit returns the response with X-LLM-Tier and X-Prompt-Est-Tokens
    deactivate K

Kong selects the AI Model for a request before any AI Policy runs, so a policy that rewrites the model field can’t move a request to another AI Model. The Datakit Policy works around this: it calls the other AI Models over HTTP and returns their response with an exit node.

Key Auth AI Auth Strategy and AI Consumer

Authentication is an AI Auth Strategy, not an AI Policy. The Key Auth AI Auth Strategy reads the apikey header and matches the key against the AI Consumer credentials. On a match, Kong attaches the AI Consumer to the request. The AI Consumer is the identity that rate limits, ACLs, and analytics use. All three AI Models reference this AI Auth Strategy in access.auth_strategies, so the Datakit Policy forwards the client’s key on each internal call.

The following settings configure the Key Auth AI Auth Strategy and the AI Consumer:

Setting

Purpose

key_names: [apikey] Kong reads the key from the apikey header. Key Auth compares the whole header value, so it can’t read a key from an Authorization: Bearer header. The note in Run the demo shows the workaround.
hide_credentials: false Kong keeps the apikey header on the request, because the Datakit Policy reads it and forwards it on the internal calls. With true, Kong removes the key before the Datakit Policy runs, and the internal calls fail with 401. OpenAI also receives the apikey header, but the AI Model Provider replaces the Authorization header with its own credentials.
credentials[].api_key The AI Consumer’s key. The field is write-only, so it takes a !secret reference to CONSUMER_API_KEY, even for a demo value.

Kong adds X-Consumer-Username and related headers to the upstream request, not to the response. Per-consumer usage appears in Konnect analytics.

Datakit Policy

The Datakit Policy is attached to the public AI Model and runs the routing workflow as a graph of nodes. Its call nodes send requests to the data plane’s own proxy listener at http://localhost:8000, at each internal AI Model’s path plus /chat/completions, so each internal call goes through the same authentication as a client request.

The following nodes make up the workflow:

Setting

Purpose

EXTRACT_PROMPT, BODY_SMALL, BODY_LARGE jq nodes that build each internal request body: the client’s messages array and the name of the AI Model to call. The internal AI Models have no config.route.model, so each one accepts only its own name in model. The other fields of the client request, for example temperature or max_tokens, aren’t forwarded.
EXTRACT_AUTH A jq node that copies the apikey header from the request headers, so the internal calls authenticate as the same AI Consumer.
ESTIMATE_TIER A jq node that counts the characters in each message’s content: the string, or the text of each part when content is an array. It estimates tokens as ceil(characters / 4), with a minimum of 1, and picks large at 80 estimated tokens or more. jq’s length counts Unicode code points, not bytes. The estimate isn’t a tokenizer count. For exact counts, use a tokenizer such as OpenAI’s tiktoken.
BUILD_DIAG_HEADERS A jq node that turns the estimate into the X-LLM-Tier and X-Prompt-Est-Tokens response headers.
IS_LARGE, ROUTE_DECISION IS_LARGE turns the tier into a boolean. The branch node runs CALL_LARGE and RESPOND_LARGE when it’s true, and CALL_SMALL and RESPOND_SMALL otherwise.
CALL_SMALL, CALL_LARGE call nodes that send BODY_SMALL or BODY_LARGE and the key to /custom-llm-routing-small/chat/completions or /custom-llm-routing-large/chat/completions. Without the /chat/completions operation path, no route matches the call.
RESPOND_SMALL, RESPOND_LARGE exit nodes that return the internal AI Model’s response body, with the headers from BUILD_DIAG_HEADERS, to the client. They don’t pass on the internal AI Model’s response headers. AI Gateway then doesn’t call the public AI Model’s own target.

AI Model Provider and AI Models

The AI Model Provider holds the OpenAI API key and adds it to each request to OpenAI. Clients call the public AI Model, custom-llm-routing-chat, on /custom-llm-routing. The small and large AI Models exist for the Datakit Policy, but a client with a valid key can also call them directly, because the internal calls use the client’s own key.

The following settings configure the AI Model Provider and the AI Models:

Setting

Purpose

config.auth (AI Model Provider) The OpenAI API key, from OPENAI_TOKEN. Kong adds it to each request to OpenAI. The field is write-only, so Konnect never returns it.
targets[].name (small and large AI Models) The upstream models, from CHAT_MODEL_1 and CHAT_MODEL_2. Change them and re-apply to swap models without changing clients.
config.route.model (public AI Model) Kong reads the model field of the request body (body_param: model) and matches it against auto. Another value matches no AI Model and returns 500 with failed the initial dns/balancer resolve for 'ai-gateway.upstream.local'.
policies (public AI Model) Attaches the Datakit Policy to the public AI Model only. The internal calls don’t run the workflow again.
targets (public AI Model) An AI Model needs at least one target. The exit nodes return before Kong uses it.
config.route.paths /custom-llm-routing for the public AI Model, and /custom-llm-routing-small and /custom-llm-routing-large for the internal ones. The generate capability adds /chat/completions to each path.
config.response_streaming: deny Clients can’t set stream=true on any of the three AI Models. The Datakit Policy needs the complete response body to return it.
config.max_request_body_size: 10485760 Accepts request bodies up to 10 MB, instead of the 8 MB default.
config.route.protocols: [http, https] Accepts HTTP, so you can call the local data plane at http://localhost:8000, and so the Datakit Policy can call the internal AI Models over HTTP.
config.route.strip_path: true Kong removes the route path before it forwards the request upstream.
config.logging.payloads: true Kong logs request and response payloads. The default is false.

Production credentials: This cookbook stores the AI Consumer’s API key and the OpenAI API key in environment variables for simplicity. In production, use AI Vaults to reference them from a secret manager, for example AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, or Azure Key Vault.

Response headers

The Datakit Policy adds the following headers to each routed response:

Header

Description

X-LLM-Tier The tier that the Datakit Policy picked: small or large.
X-Prompt-Est-Tokens The estimated token count of the prompt, as ceil(characters / 4).

Variations and next steps

You can change or extend this configuration in the following ways:

Variation

How

Swap models Change CHAT_MODEL_1 or CHAT_MODEL_2 and re-apply the configuration from Apply the Kong configuration. Clients keep sending model: auto.
Change the threshold Change 80 in the ESTIMATE_TIER node and re-apply the configuration. Set THRESHOLD in demo.py to the same value, because the demo checks the tier against it.
Route on another request value Change the ESTIMATE_TIER node to pick the tier from another value, for example a keyword in the prompt, or a header that you add as an input from request.headers. The branch, call, and exit nodes don’t change.
Forward other request parameters Build BODY_SMALL and BODY_LARGE from request.body with .model set to the AI Model name, instead of only from messages, so that fields such as temperature and max_tokens reach OpenAI.
Route by prompt complexity Replace the size estimate with a classifier call to a small model. See the model-based routing cookbook for the same call, branch, and exit pattern with a classifier.
Add per-consumer token limits Attach the AI Rate Limiting Advanced Policy to the small and large AI Models, which make the upstream calls. This cookbook hasn’t confirmed separate budgets for each AI Consumer. See the LLM cost optimization cookbook for an example.

Cleanup

The quickstart created an AI Gateway for this recipe only, so this teardown removes only this recipe’s configuration. Tear down the local data plane and delete the AI Gateway from Konnect:

export KONNECT_CONTROL_PLANE_NAME='custom-llm-routing-recipe' && curl -Ls https://get.konghq.com/ai | bash -s -- -d -k $KONNECT_TOKEN

Leave the virtual environment and remove the files and environment variables that the recipe created:

deactivate
rm -rf .venv demo.py
unset CONSUMER_API_KEY CHAT_MODEL_1 CHAT_MODEL_2

If you created the OpenAI API key only for this recipe, delete it in the OpenAI dashboard.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!