OpenAI SDK: Multi-deployment chat routingv3.6+
Use separate Routes to map Azure OpenAI SDK requests to specific deployments of GPT-3.5 and GPT-4.
Using OpenAI SDK with the AI Proxy plugin, you can configure multiple Routes in Kong Gateway to represent different Azure OpenAI deployments.
Each Route maps a unique path segment (such as azure-gpt-3-5 or azure-gpt-4) to the corresponding deployment ID and model name.
This setup allows you to use a single Azure-compatible OpenAI SDK client to switch between deployments by changing only the base URL.
For example:
client = OpenAI(
base_url="http://127.0.0.1:8000/openai/deployments/azure-gpt-3-5"
)
Or:
client = OpenAI(
base_url="http://127.0.0.1:8000/openai/deployments/azure-gpt-4"
)
Kong Gateway reads the deployment path, maps it to the appropriate Azure deployment ID and model, and handles authentication automatically.
For this configuration to work properly, you need two Routes with the following configuration:
routes: - name: azure-chat-gpt-3-5 paths: - "~/openai/deployments/azure-gpt-3-5/chat/completions$" methods: - POSTand:
routes: - name: azure-chat-gpt-4 paths: - "~/openai/deployments/azure-gpt-4/chat/completions$" methods: - POST
Prerequisites
- Azure account
Environment variables
-
AZURE_API_KEY: The API key to authenticate requests to Azure.
Add this section to your declarative configuration file:
_format_version: "3.0"
plugins:
- name: ai-proxy
config:
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: "${azure_key}"
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-35-turbo
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-3-5
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: "${azure_key}"
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-4
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-4
Make the following request:
curl -i -X POST http://localhost:8001/plugins/ \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '
{
"name": "ai-proxy",
"config": [
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-35-turbo",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-3-5"
}
}
},
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-4",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-4"
}
}
}
]
}
'
Make the following request:
curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/plugins/ \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "ai-proxy",
"config": [
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-35-turbo",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-3-5"
}
}
},
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-4",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-4"
}
}
}
]
}
'
Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
controlPlaneId: Theidof the control plane. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
See the Konnect API reference to learn about region-specific URLs and personal access tokens.
echo "
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
name: ai-proxy
namespace: kong
annotations:
kubernetes.io/ingress.class: kong
labels:
global: 'true'
config:
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: '${azure_key}'
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-35-turbo
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-3-5
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: '${azure_key}'
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-4
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-4
plugin: ai-proxy
" | kubectl apply -f -
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}
Add the following to your Terraform configuration to create a Konnect Gateway Plugin:
resource "konnect_gateway_plugin_ai_proxy" "my_ai_proxy" {
enabled = true
config = [
{
route_type = "llm/v1/chat"
auth = {
header_name = "api-key"
header_value = "${azure_key}"
}
logging = {
log_statistics = true
log_payloads = false
}
model = {
provider = "azure"
name = "gpt-35-turbo"
options = {
azure_instance = "my-openai-instace"
azure_deployment_id = "my-gpt-3-5"
}
}
},
{
route_type = "llm/v1/chat"
auth = {
header_name = "api-key"
header_value = "${azure_key}"
}
logging = {
log_statistics = true
log_payloads = false
}
model = {
provider = "azure"
name = "gpt-4"
options = {
azure_instance = "my-openai-instace"
azure_deployment_id = "my-gpt-4"
}
}
} ]
control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
}
This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "azure_api_key" {
type = string
}
Add this section to your declarative configuration file:
_format_version: "3.0"
plugins:
- name: ai-proxy
service: serviceName|Id
config:
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: "${azure_key}"
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-35-turbo
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-3-5
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: "${azure_key}"
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-4
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-4
Make sure to replace the following placeholders with your own values:
-
serviceName|Id: Theidornameof the service the plugin configuration will target.
Make the following request:
curl -i -X POST http://localhost:8001/services/{serviceName|Id}/plugins/ \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '
{
"name": "ai-proxy",
"config": [
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-35-turbo",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-3-5"
}
}
},
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-4",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-4"
}
}
}
]
}
'
Make sure to replace the following placeholders with your own values:
-
serviceName|Id: Theidornameof the service the plugin configuration will target.
Make the following request:
curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/services/{serviceId}/plugins/ \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "ai-proxy",
"config": [
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-35-turbo",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-3-5"
}
}
},
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-4",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-4"
}
}
}
]
}
'
Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
controlPlaneId: Theidof the control plane. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
serviceId: Theidof the service the plugin configuration will target.
See the Konnect API reference to learn about region-specific URLs and personal access tokens.
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: ai-proxy
namespace: kong
annotations:
kubernetes.io/ingress.class: kong
config:
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: '${azure_key}'
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-35-turbo
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-3-5
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: '${azure_key}'
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-4
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-4
plugin: ai-proxy
" | kubectl apply -f -
Next, apply the KongPlugin resource by annotating the service resource:
kubectl annotate -n kong service SERVICE_NAME konghq.com/plugins=ai-proxy
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}
Add the following to your Terraform configuration to create a Konnect Gateway Plugin:
resource "konnect_gateway_plugin_ai_proxy" "my_ai_proxy" {
enabled = true
config = [
{
route_type = "llm/v1/chat"
auth = {
header_name = "api-key"
header_value = "${azure_key}"
}
logging = {
log_statistics = true
log_payloads = false
}
model = {
provider = "azure"
name = "gpt-35-turbo"
options = {
azure_instance = "my-openai-instace"
azure_deployment_id = "my-gpt-3-5"
}
}
},
{
route_type = "llm/v1/chat"
auth = {
header_name = "api-key"
header_value = "${azure_key}"
}
logging = {
log_statistics = true
log_payloads = false
}
model = {
provider = "azure"
name = "gpt-4"
options = {
azure_instance = "my-openai-instace"
azure_deployment_id = "my-gpt-4"
}
}
} ]
control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
service = {
id = konnect_gateway_service.my_service.id
}
}
This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "azure_api_key" {
type = string
}
Add this section to your declarative configuration file:
_format_version: "3.0"
plugins:
- name: ai-proxy
route: routeName|Id
config:
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: "${azure_key}"
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-35-turbo
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-3-5
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: "${azure_key}"
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-4
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-4
Make sure to replace the following placeholders with your own values:
-
routeName|Id: Theidornameof the route the plugin configuration will target.
Make the following request:
curl -i -X POST http://localhost:8001/routes/{routeName|Id}/plugins/ \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '
{
"name": "ai-proxy",
"config": [
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-35-turbo",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-3-5"
}
}
},
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-4",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-4"
}
}
}
]
}
'
Make sure to replace the following placeholders with your own values:
-
routeName|Id: Theidornameof the route the plugin configuration will target.
Make the following request:
curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/routes/{routeId}/plugins/ \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "ai-proxy",
"config": [
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-35-turbo",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-3-5"
}
}
},
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-4",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-4"
}
}
}
]
}
'
Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
controlPlaneId: Theidof the control plane. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
routeId: Theidof the route the plugin configuration will target.
See the Konnect API reference to learn about region-specific URLs and personal access tokens.
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: ai-proxy
namespace: kong
annotations:
kubernetes.io/ingress.class: kong
config:
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: '${azure_key}'
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-35-turbo
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-3-5
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: '${azure_key}'
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-4
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-4
plugin: ai-proxy
" | kubectl apply -f -
Next, apply the KongPlugin resource by annotating the httproute or ingress resource:
kubectl annotate -n kong httproute konghq.com/plugins=ai-proxy
kubectl annotate -n kong ingress konghq.com/plugins=ai-proxy
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}
Add the following to your Terraform configuration to create a Konnect Gateway Plugin:
resource "konnect_gateway_plugin_ai_proxy" "my_ai_proxy" {
enabled = true
config = [
{
route_type = "llm/v1/chat"
auth = {
header_name = "api-key"
header_value = "${azure_key}"
}
logging = {
log_statistics = true
log_payloads = false
}
model = {
provider = "azure"
name = "gpt-35-turbo"
options = {
azure_instance = "my-openai-instace"
azure_deployment_id = "my-gpt-3-5"
}
}
},
{
route_type = "llm/v1/chat"
auth = {
header_name = "api-key"
header_value = "${azure_key}"
}
logging = {
log_statistics = true
log_payloads = false
}
model = {
provider = "azure"
name = "gpt-4"
options = {
azure_instance = "my-openai-instace"
azure_deployment_id = "my-gpt-4"
}
}
} ]
control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
route = {
id = konnect_gateway_route.my_route.id
}
}
This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "azure_api_key" {
type = string
}
Add this section to your declarative configuration file:
_format_version: "3.0"
plugins:
- name: ai-proxy
consumer: consumerName|Id
config:
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: "${azure_key}"
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-35-turbo
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-3-5
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: "${azure_key}"
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-4
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-4
Make sure to replace the following placeholders with your own values:
-
consumerName|Id: Theidornameof the consumer the plugin configuration will target.
Make the following request:
curl -i -X POST http://localhost:8001/consumers/{consumerName|Id}/plugins/ \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '
{
"name": "ai-proxy",
"config": [
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-35-turbo",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-3-5"
}
}
},
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-4",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-4"
}
}
}
]
}
'
Make sure to replace the following placeholders with your own values:
-
consumerName|Id: Theidornameof the consumer the plugin configuration will target.
Make the following request:
curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/consumers/{consumerId}/plugins/ \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "ai-proxy",
"config": [
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-35-turbo",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-3-5"
}
}
},
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-4",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-4"
}
}
}
]
}
'
Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
controlPlaneId: Theidof the control plane. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
consumerId: Theidof the consumer the plugin configuration will target.
See the Konnect API reference to learn about region-specific URLs and personal access tokens.
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: ai-proxy
namespace: kong
annotations:
kubernetes.io/ingress.class: kong
config:
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: '${azure_key}'
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-35-turbo
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-3-5
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: '${azure_key}'
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-4
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-4
plugin: ai-proxy
" | kubectl apply -f -
Next, apply the KongPlugin resource by annotating the KongConsumer resource:
kubectl annotate -n kong CONSUMER_NAME konghq.com/plugins=ai-proxy
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}
Add the following to your Terraform configuration to create a Konnect Gateway Plugin:
resource "konnect_gateway_plugin_ai_proxy" "my_ai_proxy" {
enabled = true
config = [
{
route_type = "llm/v1/chat"
auth = {
header_name = "api-key"
header_value = "${azure_key}"
}
logging = {
log_statistics = true
log_payloads = false
}
model = {
provider = "azure"
name = "gpt-35-turbo"
options = {
azure_instance = "my-openai-instace"
azure_deployment_id = "my-gpt-3-5"
}
}
},
{
route_type = "llm/v1/chat"
auth = {
header_name = "api-key"
header_value = "${azure_key}"
}
logging = {
log_statistics = true
log_payloads = false
}
model = {
provider = "azure"
name = "gpt-4"
options = {
azure_instance = "my-openai-instace"
azure_deployment_id = "my-gpt-4"
}
}
} ]
control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
consumer = {
id = konnect_gateway_consumer.my_consumer.id
}
}
This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "azure_api_key" {
type = string
}
Add this section to your declarative configuration file:
_format_version: "3.0"
plugins:
- name: ai-proxy
consumer_group: consumerGroupName|Id
config:
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: "${azure_key}"
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-35-turbo
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-3-5
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: "${azure_key}"
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-4
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-4
Make sure to replace the following placeholders with your own values:
-
consumerGroupName|Id: Theidornameof the consumer group the plugin configuration will target.
Make the following request:
curl -i -X POST http://localhost:8001/consumer_groups/{consumerGroupName|Id}/plugins/ \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '
{
"name": "ai-proxy",
"config": [
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-35-turbo",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-3-5"
}
}
},
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-4",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-4"
}
}
}
]
}
'
Make sure to replace the following placeholders with your own values:
-
consumerGroupName|Id: Theidornameof the consumer group the plugin configuration will target.
Make the following request:
curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/consumer_groups/{consumerGroupId}/plugins/ \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "ai-proxy",
"config": [
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-35-turbo",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-3-5"
}
}
},
{
"route_type": "llm/v1/chat",
"auth": {
"header_name": "api-key",
"header_value": "${azure_key}"
},
"logging": {
"log_statistics": true,
"log_payloads": false
},
"model": {
"provider": "azure",
"name": "gpt-4",
"options": {
"azure_instance": "my-openai-instace",
"azure_deployment_id": "my-gpt-4"
}
}
}
]
}
'
Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
controlPlaneId: Theidof the control plane. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
consumerGroupId: Theidof the consumer group the plugin configuration will target.
See the Konnect API reference to learn about region-specific URLs and personal access tokens.
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: ai-proxy
namespace: kong
annotations:
kubernetes.io/ingress.class: kong
config:
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: '${azure_key}'
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-35-turbo
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-3-5
- route_type: llm/v1/chat
auth:
header_name: api-key
header_value: '${azure_key}'
logging:
log_statistics: true
log_payloads: false
model:
provider: azure
name: gpt-4
options:
azure_instance: my-openai-instace
azure_deployment_id: my-gpt-4
plugin: ai-proxy
" | kubectl apply -f -
Next, apply the KongPlugin resource by annotating the KongConsumerGroup resource:
kubectl annotate -n kong CONSUMERGROUP_NAME konghq.com/plugins=ai-proxy
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}
Add the following to your Terraform configuration to create a Konnect Gateway Plugin:
resource "konnect_gateway_plugin_ai_proxy" "my_ai_proxy" {
enabled = true
config = [
{
route_type = "llm/v1/chat"
auth = {
header_name = "api-key"
header_value = "${azure_key}"
}
logging = {
log_statistics = true
log_payloads = false
}
model = {
provider = "azure"
name = "gpt-35-turbo"
options = {
azure_instance = "my-openai-instace"
azure_deployment_id = "my-gpt-3-5"
}
}
},
{
route_type = "llm/v1/chat"
auth = {
header_name = "api-key"
header_value = "${azure_key}"
}
logging = {
log_statistics = true
log_payloads = false
}
model = {
provider = "azure"
name = "gpt-4"
options = {
azure_instance = "my-openai-instace"
azure_deployment_id = "my-gpt-4"
}
}
} ]
control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
consumer_group = {
id = konnect_gateway_consumer_group.my_consumer_group.id
}
}
This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "azure_api_key" {
type = string
}