Home / How-to Guides
Edit
Report

Strip the model field from OpenAI SDK requests

Uses: Kong Gateway AI Gateway deck
Deployment Platform
Related Documentation
Gateway Documentation decK Documentation
Tags
#ai #openai #ai-sdks
Related Resources
AI Gateway
AI Proxy Advanced
Pre-function
Minimum Version
Kong Gateway - 3.6
TL;DR

Add a Pre-function plugin that strips the model field from the request body before AI Proxy Advanced processes it. This lets the gateway control model selection through its balancer configuration.

Prerequisites

Kong Konnect

This is a Konnect tutorial and requires a Konnect personal access token.

  1. Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.

  2. Export your token to an environment variable:

     export KONNECT_TOKEN='YOUR_KONNECT_PAT'

  3. Run the quickstart script to automatically provision a Control Plane and Data Plane, and configure your environment:

     curl -Ls https://get.konghq.com/quickstart | bash -s -- -k $KONNECT_TOKEN --deck-output

    This sets up a Konnect Control Plane named quickstart, provisions a local Data Plane, and prints out the following environment variable exports:

     export DECK_KONNECT_TOKEN=$KONNECT_TOKEN
 export DECK_KONNECT_CONTROL_PLANE_NAME=quickstart
 export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
 export KONNECT_PROXY_URL='http://localhost:8000'

    Copy and paste these into your terminal to configure your session.

Kong Gateway running

This tutorial requires Kong Gateway Enterprise. If you don’t have Kong Gateway set up yet, you can use the quickstart script with an enterprise license to get an instance of Kong Gateway running almost instantly.

  1. Export your license to an environment variable:

     export KONG_LICENSE_DATA='LICENSE-CONTENTS-GO-HERE'

  2. Run the quickstart script:

    curl -Ls https://get.konghq.com/quickstart | bash -s -- -e KONG_LICENSE_DATA

    Once Kong Gateway is ready, you will see the following message:

     Kong Gateway Ready

decK   v1.43+

decK is a CLI tool for managing Kong Gateway declaratively with state files. To complete this tutorial, install decK version 1.43 or later.

This guide uses deck gateway apply, which directly applies entity configuration to your Gateway instance. We recommend upgrading your decK installation to take advantage of this tool.

You can check your current decK version with deck version.

Required entities

For this tutorial, you’ll need Kong Gateway entities, like Gateway Services and Routes, pre-configured. These entities are essential for Kong Gateway to function but installing them isn’t the focus of this guide. Follow these steps to pre-configure them:

  1. Run the following command:

    echo '
_format_version: "3.0"
services:
  - name: example-service
    url: http://httpbin.konghq.com/anything
routes:
  - name: example-route
    paths:
    - "/anything"
    service:
      name: example-service
' | deck gateway apply -

To learn more about entities, you can read our entities documentation.

OpenAI

This tutorial uses OpenAI:

  1. Create an OpenAI account.
  2. Get an API key.

  3. Create a decK variable with the API key:

    export DECK_OPENAI_API_KEY='YOUR OPENAI API KEY'

OpenAI-compatible SDKs always set the model field in the request body. This is a required parameter and can’t be omitted.

AI Proxy Advanced validates the body model against the plugin-configured model. If they don’t match, the plugin rejects the request with 400 Bad Request: cannot use own model - must be: <configured-model>. When load balancing across multiple models, the balancer may route to a target that doesn’t match the SDK’s model value, which triggers this error.

The fix is to use the Pre-function plugin to strip the model field from the request body before AI Proxy Advanced processes it.

Configure the Pre-function plugin

First, let’s configure the Pre-function plugin to removes the model field from the JSON request body to the LLM provider:

echo '
_format_version: "3.0"
plugins:
  - name: pre-function
    config:
      access:
      - |-
        local req_body = kong.request.get_body()
        req_body["model"] = nil
        kong.service.request.set_body(req_body)
' | deck gateway apply -

Configure the AI Proxy Advanced plugin

Now, let’s let’s configure AI Proxy Advanced with multiple targets to different OpenAI models. The balancer selects which target handles each request, independent of whatever model the SDK originally specified:

echo '
_format_version: "3.0"
plugins:
  - name: ai-proxy-advanced
    config:
      balancer:
        algorithm: round-robin
        retries: 3
      targets:
      - route_type: llm/v1/chat
        auth:
          header_name: Authorization
          header_value: Bearer ${{ env "DECK_OPENAI_API_KEY" }}
        model:
          provider: openai
          name: gpt-4o
          options:
            max_tokens: 512
            temperature: 1.0
      - route_type: llm/v1/chat
        auth:
          header_name: Authorization
          header_value: Bearer ${{ env "DECK_OPENAI_API_KEY" }}
        model:
          provider: openai
          name: gpt-4o-mini
          options:
            max_tokens: 512
            temperature: 1.0
' | deck gateway apply -

Create a test script

Now, let’s create a test script. Even though the SDK sends model="gpt-4o" in the body, the Pre-function plugin strips it. AI Proxy Advanced’s balancer decides which model actually handles the request:

cat <<EOF > test_strip_model.py
from openai import OpenAI

kong_url = "http://localhost:8000"
kong_route = "anything"

client = OpenAI(
    api_key="test",
    base_url=f"{kong_url}/{kong_route}"
)

for i in range(4):
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "What model are you? Reply with only your model name."}]
    )
    print(f"Request {i+1}: {response.model}")
EOF

cat <<EOF > test_strip_model.py
from openai import OpenAI
import os

kong_url = os.environ['KONNECT_PROXY_URL']
kong_route = "anything"

client = OpenAI(
    api_key="test",
    base_url=f"{kong_url}/{kong_route}"
)

for i in range(4):
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "What model are you? Reply with only your model name."}]
    )
    print(f"Request {i+1}: {response.model}")
EOF

Validate the configuration

Now we can run the script created in the previous step:

python test_strip_model.py

With round-robin balancing and two targets, you should see the response.model value alternate between gpt-4o and gpt-4o-mini across the four requests, confirming that the gateway controls model selection regardless of what the SDK sends.

Cleanup

Clean up Konnect environment

If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.

Destroy the Kong Gateway container

 
curl -Ls https://get.konghq.com/quickstart | bash -s -- -d
Something wrong?
Report an Issue | Edit this Page

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!
Contribute

Still need help

Ask in our Forum
Contact Support