AI Request Transformer Policy

The AI Request Transformer Policy uses a configured LLM service to transform a client request body before proxying the request upstream.

This Policy supports llm/v1/chat requests and can be tuned by setting the parameters in the config.llm block.

The AI Request Transformer Policy runs before all of the AI prompt Policies, allowing it to also transform requests before sending them to a different LLM.

Known failure mode: Chaining AI Request Transformer with the AI Gateway

Chaining AI Request Transformer with other AI Gateway operations may fail for some upstream providers, even though the same setup works correctly with other providers.

The reason is that the AI Request Transformer Policy forwards raw model output, and if the LLM service model does not produce strict JSON, the proxy chain cannot function correctly. This is not a bug in AI Gateway but a limitation of LLM behavior.

 
sequenceDiagram
    autonumber
    participant client as Client
    participant kong as AI Gateway
    participant ai as AI service
    participant backend as Upstream LLM
    activate client
    activate kong
    client->>kong: Sends a request
    deactivate client
    activate ai
    kong->>ai: Sends client's request for transformation
    ai->>kong: Transforms request
    deactivate ai
    activate backend
    kong->>backend: Sends transformed request to backend
    backend->>kong: Returns response to AI Gateway
    deactivate backend
    activate ai
    kong->>ai: Sends response to AI service
    ai->>kong: Transforms response
    deactivate ai
    activate client
    kong->>client: Returns transformed response to client
    deactivate kong
    deactivate client
  

Figure 1: The diagram shows the journey of a consumer’s request through AI Gateway to the upstream LLM provider, where it is transformed by Kong’s AI Request Transformer and AI Response Transformer Policies using an AI service.

The AI Gateway admin sets up an llm configuration block and a prompt for the LLM service used to transform requests.

The prompt becomes the system message in the LLM chat request, and prepares the LLM with transformation instructions for the incoming client request body.

1. The client makes an HTTP(S) call.
2. The AI Gateway creates a request to the LLM service using client’s request body as the user message in the LLM chat request, and then sends it to the configured LLM service to be transformed.
3. The LLM service returns a response assistant message, containing the transformed user message. This is subsequently set as the request body for the upstream LLM provider.
4. The AI Gateway sends the transformed request to the upstream LLM provider.
5. The the upstream LLM provider returns a response to AI Gateway.
6. The AI Gateway sends the response to the AI Response Transformer policy (as in figure 1), or directly to the client.