Use LiteLLM with AI Proxy with Kong AI Gateway
You can configure LiteLLM to to use your AI Gateway Route by replacing the base_url parameter in the LiteLLM API call with your Kong Gateway proxy URL.
Prerequisites
Kong Konnect
This is a Konnect tutorial and requires a Konnect personal access token.
-
Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
-
Export your token to an environment variable:
export KONNECT_TOKEN='YOUR_KONNECT_PAT'Copied! -
Run the quickstart script to automatically provision a Control Plane and Data Plane, and configure your environment:
curl -Ls https://get.konghq.com/quickstart | bash -s -- -k $KONNECT_TOKEN --deck-outputCopied!This sets up a Konnect Control Plane named
quickstart, provisions a local Data Plane, and prints out the following environment variable exports:export DECK_KONNECT_TOKEN=$KONNECT_TOKEN export DECK_KONNECT_CONTROL_PLANE_NAME=quickstart export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com export KONNECT_PROXY_URL='http://localhost:8000'Copied!Copy and paste these into your terminal to configure your session.
Kong Gateway running
This tutorial requires Kong Gateway Enterprise. If you don’t have Kong Gateway set up yet, you can use the quickstart script with an enterprise license to get an instance of Kong Gateway running almost instantly.
-
Export your license to an environment variable:
export KONG_LICENSE_DATA='LICENSE-CONTENTS-GO-HERE'Copied! -
Run the quickstart script:
curl -Ls https://get.konghq.com/quickstart | bash -s -- -e KONG_LICENSE_DATACopied!Once Kong Gateway is ready, you will see the following message:
Kong Gateway Ready
decK v1.43+
decK is a CLI tool for managing Kong Gateway declaratively with state files. To complete this tutorial, install decK version 1.43 or later.
This guide uses deck gateway apply, which directly applies entity configuration to your Gateway instance.
We recommend upgrading your decK installation to take advantage of this tool.
You can check your current decK version with deck version.
Required entities
For this tutorial, you’ll need Kong Gateway entities, like Gateway Services and Routes, pre-configured. These entities are essential for Kong Gateway to function but installing them isn’t the focus of this guide. Follow these steps to pre-configure them:
-
Run the following command:
echo ' _format_version: "3.0" services: - name: example-service url: http://httpbin.konghq.com/anything routes: - name: example-route paths: - "/anything" service: name: example-service ' | deck gateway apply -Copied!
To learn more about entities, you can read our entities documentation.
OpenAI
This tutorial uses OpenAI:
- Create an OpenAI account.
- Get an API key.
- Create a decK variable with the API key:
export DECK_OPENAI_API_KEY="YOUR OPENAI API KEY"
export DECK_OPENAI_API_KEY="YOUR OPENAI API KEY"
Configure the AI Proxy plugin
Enable the AI Proxy plugin with your OpenAI API key and model details to route LiteLLM OpenAI-compatible requests through Kong AI Gateway. In this example, we’ll use the gpt-4.1 model from OpenAI:
echo '
_format_version: "3.0"
plugins:
- name: ai-proxy
config:
route_type: llm/v1/chat
auth:
header_name: Authorization
header_value: Bearer ${{ env "DECK_OPENAI_API_KEY" }}
model:
provider: openai
name: gpt-4.1
' | deck gateway apply -
Add authentication
To secure access to your Route, create a Consumer and set up an authentication plugin:
LiteLLM expects authentication as an
Authorizationheader with a value starting withBearer. You can use plugins like OAuth 2.0 Authentication or OpenID Connect to generate Bearer tokens. In this example, for testing purposes, we’ll recreate this pattern using the Key Authentication plugin.
echo '
_format_version: "3.0"
plugins:
- name: key-auth
route: example-route
config:
key_names:
- Authorization
consumers:
- username: ai-user
keyauth_credentials:
- key: Bearer my-api-key
' | deck gateway apply -
Install LiteLLM
Install the LiteLLM Python SDK:
Create a LiteLLM script
Use the following command to create a file named app.py containing a LiteLLM Python script:
cat <<EOF > app.py
import litellm
kong_url = "http://127.0.0.1:8000"
kong_route = "anything"
response = litellm.completion(
model="gpt-4.1",
messages=[{"role": "user", "content": "What are you?"}],
api_key="my-api-key",
base_url=f"{kong_url}/{kong_route}"
)
print(f"$ ChainAnswer:> {response['choices'][0]['message']['content']}")
EOF
cat <<EOF > app.py
import litellm
import os
kong_url = os.environ['KONNECT_PROXY_URL']
kong_route = "anything"
response = litellm.completion(
model="gpt-4.1",
messages=[{"role": "user", "content": "What are you?"}],
api_key="my-api-key",
base_url=f"{kong_url}/{kong_route}"
)
print(f"$ ChainAnswer:> {response['choices'][0]['message']['content']}")
EOF
With the base_url parameter, we can override the OpenAI base URL that LiteLLM uses by default with the URL to our Kong Gateway Route. This allows proxying requests and applying Kong Gateway plugins while still using LiteLLM’s API interface.
In the api_key parameter, we’ll add the API key we created, without the Bearer prefix, which LiteLLM adds automatically in the request header.
Validate
Run your script to validate that LiteLLM can access the Route:
python3 ./app.py
The response should look like this:
ChainAnswer:> I'm an artificial intelligence (AI) assistant created by OpenAI. I'm designed to help answer questions, provide information, write content, and assist with a wide variety of tasks through natural conversation. You can think of me as a type of intelligent computer program that uses language models to understand and respond to your messages. If you have any questions or need help with something, just let me know!
Cleanup
Clean up Konnect environment
If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.
Destroy the Kong Gateway container
curl -Ls https://get.konghq.com/quickstart | bash -s -- -d