AI Request Transformer

The AI Request Transformer plugin uses a configured LLM service to transform a client request body before proxying the request upstream.

This plugin supports the same llm/v1/chat requests and providers as the AI Proxy plugin.

It also uses the same configuration and tuning parameters as the AI Proxy plugin, under the config.llm block.

The AI Request Transformer plugin runs before all of the AI prompt plugins and the AI Proxy plugin, allowing it to also transform requests before sending them to a different LLM.

Known failure mode: AI Request Transformer with AI Proxy

Chaining AI Request Transformer with AI Proxy or AI Proxy Advanced may fail for some providers, even though the same setup works with others.

The reason is that the AI Request Transformer plugin forwards raw model output, and if the model does not produce strict JSON, the proxy chain cannot function correctly. This is not a bug in Kong AI Gateway but a limitation of LLM behavior.

 
sequenceDiagram
    autonumber
    participant client as Client
    participant kong as Kong Gateway
    participant ai as AI LLM service
    participant backend as Backend service
    activate client
    activate kong
    client->>kong: Sends a request
    deactivate client
    activate ai
    kong->>ai: Sends client's request for transformation
    ai->>kong: Transforms request
    deactivate ai
    activate backend
    kong->>backend: Sends transformed request to backend
    backend->>kong: Returns response to Kong Gateway
    deactivate backend
    activate ai
    kong->>ai: Sends response to AI service
    ai->>kong: Transforms response
    deactivate ai
    activate client
    kong->>client: Returns transformed response to client
    deactivate kong
    deactivate client
  

Figure 1: The diagram shows the journey of a consumer’s request through Kong Gateway to the backend service, where it is transformed by both an AI LLM service and Kong’s AI Request Transformer and the AI Response Transformer plugins.

1. The Kong Gateway admin sets up an llm configuration block.
2. The Kong Gateway admin sets up a prompt. The prompt becomes the system message in the LLM chat request, and prepares the LLM with transformation instructions for the incoming client request body.
3. The client makes an HTTP(S) call.
4. Before proxying the client’s request to the backend, Kong Gateway sets the entire request body as the user message in the LLM chat request, and then sends it to the configured LLM service.
5. The LLM service returns a response assistant message, which is subsequently set as the upstream request body.
6. The Kong Gateway sends the transformed request to the AI LLM service.
7. The AI LLM service returns a response to Kong Gateway.
8. The Kong Gateway sends the transformed response to the client.