Home / How-to Guides
Edit
Report

Route Azure OpenAI SDK requests to specific deployments with multiple routes

Uses: Kong Gateway AI Gateway deck
Deployment Platform
Related Documentation
Gateway Documentation decK Documentation
Tags
#ai #openai #azure #ai-sdks
Related Resources
AI Gateway
AI Proxy Advanced
AI Proxy Advanced: Multi-deployment chat routing example
Minimum Version
Kong Gateway - 3.6
TL;DR

Create a Route for each Azure deployment with a path that matches the SDK’s URL pattern, then configure AI Proxy Advanced on each Route with the corresponding deployment ID. The SDK switches between deployments by changing the base URL.

Prerequisites

Kong Konnect

This is a Konnect tutorial and requires a Konnect personal access token.

  1. Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.

  2. Export your token to an environment variable:

     export KONNECT_TOKEN='YOUR_KONNECT_PAT'

  3. Run the quickstart script to automatically provision a Control Plane and Data Plane, and configure your environment:

     curl -Ls https://get.konghq.com/quickstart | bash -s -- -k $KONNECT_TOKEN --deck-output

    This sets up a Konnect Control Plane named quickstart, provisions a local Data Plane, and prints out the following environment variable exports:

     export DECK_KONNECT_TOKEN=$KONNECT_TOKEN
 export DECK_KONNECT_CONTROL_PLANE_NAME=quickstart
 export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
 export KONNECT_PROXY_URL='http://localhost:8000'

    Copy and paste these into your terminal to configure your session.

Kong Gateway running

This tutorial requires Kong Gateway Enterprise. If you don’t have Kong Gateway set up yet, you can use the quickstart script with an enterprise license to get an instance of Kong Gateway running almost instantly.

  1. Export your license to an environment variable:

     export KONG_LICENSE_DATA='LICENSE-CONTENTS-GO-HERE'

  2. Run the quickstart script:

    curl -Ls https://get.konghq.com/quickstart | bash -s -- -e KONG_LICENSE_DATA

    Once Kong Gateway is ready, you will see the following message:

     Kong Gateway Ready

decK   v1.43+

decK is a CLI tool for managing Kong Gateway declaratively with state files. To complete this tutorial, install decK version 1.43 or later.

This guide uses deck gateway apply, which directly applies entity configuration to your Gateway instance. We recommend upgrading your decK installation to take advantage of this tool.

You can check your current decK version with deck version.

Required entities

For this tutorial, you’ll need Kong Gateway entities, like Gateway Services and Routes, pre-configured. These entities are essential for Kong Gateway to function but installing them isn’t the focus of this guide. Follow these steps to pre-configure them:

  1. Run the following command:

    echo '
_format_version: "3.0"
services:
  - name: azure-openai-service
    url: http://localhost:8000
routes:
  - name: azure-gpt-4o
    paths:
    - "~/openai/deployments/gpt-4o/chat/completions$"
    methods:
    - POST
    service:
      name: azure-openai-service
  - name: azure-gpt-4-1-mini
    paths:
    - "~/openai/deployments/gpt-4.1-mini/chat/completions$"
    methods:
    - POST
    service:
      name: azure-openai-service
' | deck gateway apply -

To learn more about entities, you can read our entities documentation.

Azure OpenAI service

This tutorial uses Azure OpenAI service. Use the following steps to configure it:

  1. Create an Azure account.
  2. In the Azure Portal, click Create a resource.
    1. Search for Azure OpenAI and select Azure OpenAI Service.
    2. Configure your Azure resource.
    3. Once created, export the following environment variable: 
       export DECK_AZURE_INSTANCE_NAME='YOUR AZURE RESOURCE NAME'
  3. Once you’ve created your Azure resource, go to Azure AI foundry and do the following:
    1. In the My assets subgroup in the main sidebar, click Models and deployments and click Deploy model.
    2. Once deployed, export the following environment variables: 
       export DECK_AZURE_OPENAI_API_KEY='YOUR AZURE OPENAI MODEL API KEY'
 export DECK_AZURE_DEPLOYMENT_ID='YOUR AZURE OPENAI DEPLOYMENT NAME'

Python

To complete this tutorial, you’ll need Python (version 3.7 or later) and pip installed on your machine. You can verify it by running:

python3
python3 -m pip --version

  1. Create a virtual env:

    python3 -m venv myenv

  2. Activate it:

    source myenv/bin/activate

OpenAI SDK

Install the OpenAI SDK:

pip install openai

The Azure OpenAI SDK constructs request URLs in the format https://{azure_instance}.openai.azure.com/openai/deployments/{deployment_id}/chat/completions. Each deployment has its own URL path.

You can map each deployment to a separate Kong Gateway Route with its own AI Proxy Advanced configuration. The SDK switches between deployments by pointing azure_endpoint at Kong Gateway and changing the model parameter. Kong Gateway matches the request to the correct Route and forwards it to the corresponding Azure deployment. When the SDK sends a request with model="gpt-4o", the AzureOpenAI client constructs the path /openai/deployments/gpt-4o/chat/completions, which matches the first Route. Requests with model="gpt-4.1-mini" match the second Route.

This approach gives you explicit control over each deployment’s configuration, such as different auth keys, model options, or logging settings per deployment.

Configure AI Proxy Advanced for the GPT-4o Route

Configure AI Proxy Advanced on the azure-gpt-4o Route to target the gpt-4o deployment:

echo '
_format_version: "3.0"
plugins:
  - name: ai-proxy-advanced
    route: azure-gpt-4o
    config:
      targets:
      - route_type: llm/v1/chat
        auth:
          header_name: api-key
          header_value: "${{ env "DECK_AZURE_OPENAI_API_KEY" }}"
        model:
          provider: azure
          name: gpt-4o
          options:
            azure_instance: "${{ env "DECK_AZURE_INSTANCE_NAME" }}"
            azure_deployment_id: gpt-4o
' | deck gateway apply -

Configure AI Proxy Advanced for the GPT-4.1-mini Route

Configure AI Proxy Advanced on the azure-gpt-4-1-mini Route to target the gpt-4.1-mini deployment:

echo '
_format_version: "3.0"
plugins:
  - name: ai-proxy-advanced
    route: azure-gpt-4-1-mini
    config:
      targets:
      - route_type: llm/v1/chat
        auth:
          header_name: api-key
          header_value: "${{ env "DECK_AZURE_OPENAI_API_KEY" }}"
        model:
          provider: azure
          name: gpt-4.1-mini
          options:
            azure_instance: "${{ env "DECK_AZURE_INSTANCE_NAME" }}"
            azure_deployment_id: gpt-4.1-mini
' | deck gateway apply -

Validate

Create a test script that sends requests to both deployments through Kong Gateway. The AzureOpenAI client constructs the correct URL path for each deployment based on the model parameter:

cat <<EOF > test_azure_multi_route.py
from openai import AzureOpenAI

client = AzureOpenAI(
    api_key="test",
    azure_endpoint="http://localhost:8000",
    api_version="2025-01-01-preview"
)

for model in ["gpt-4o", "gpt-4.1-mini"]:
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "What model are you? Reply with only your model name."}]
    )
    print(f"Requested: {model}, Got: {response.model}")
EOF

cat <<EOF > test_azure_multi_route.py
from openai import AzureOpenAI
import os

client = AzureOpenAI(
    api_key="test",
    azure_endpoint=os.environ['KONNECT_PROXY_URL'],
    api_version="2025-01-01-preview"
)

for model in ["gpt-4o", "gpt-4.1-mini"]:
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "What model are you? Reply with only your model name."}]
    )
    print(f"Requested: {model}, Got: {response.model}")
EOF

Run the script:

python test_azure_multi_route.py

You should see each request routed to the corresponding Azure deployment, confirming that each Route maps to a different model.

Cleanup

Clean up Konnect environment

If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.

Destroy the Kong Gateway container

 
curl -Ls https://get.konghq.com/quickstart | bash -s -- -d
Something wrong?
Report an Issue | Edit this Page

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!
Contribute

Still need help

Ask in our Forum
Contact Support