AI Response Transformer Policy

The AI Response Transformer Policy uses a configured LLM service to transform the upstream LLM provider’s response before returning it to the client.

It can also terminate or otherwise nullify the response if it fails a compliance or formatting check from the configured LLM service.

This Policy supports llm/v1/chat requests and can be tuned by setting the parameters in the config.llm block.

The AI Response Transformer Policy runs after AI Gateway proxies to the upstream LLM service through an AI Model, allowing it to transform responses from any upstream LLM.

 
sequenceDiagram
    autonumber
    participant client as Client
    participant kong as AI Gateway
    participant backend as Upstream LLM
    participant ai as AI service
    activate client
    activate kong
    client->>kong: Sends a prompt
    deactivate client
    activate backend
    kong->>backend: Forwards the prompt
    backend->>kong: Returns the response to AI Gateway
    deactivate backend
    activate ai
    kong->>ai: Sends the response for transformation
    ai->>kong: Returns the transformed response
    deactivate ai
    activate client
    kong->>client: Returns the transformed response to the client
    deactivate kong
    deactivate client
  

Figure 1: The diagram shows the journey of a consumer’s prompt through AI Gateway to an Upstream LLM that sends a response, the response is transformed by the AI Response Transformer Policy using an AI service.

The AI Gateway admin sets up an llm configuration block and a prompt for the LLM service used to transform requests.

The prompt becomes the system message in the LLM chat request, and provides transformation instructions to the LLM for the returning upstream response body.

1. The client makes an HTTP(S) call.
2. The AI Gateway sends the transformed request to the upstream LLM provider or uses the AI Request Transformer Policy.
3. The the upstream LLM provider returns a response to AI Gateway.
4. The AI Response Transformer policy sets the entire response body as the user message in an LLM chat request, then sends it to the configured LLM service.
5. The LLM service returns a response assistant message, which is subsequently set as the upstream response body.
6. The Policy returns early (kong.response.exit) and can handle gzip or chunked requests, similar to the Forward Proxy policy.

Adjusting response headers, status codes, and body

You can additionally instruct the LLM service to respond in the following format, which lets you adjust the response headers, response status code, and response body:

{
  "headers":
    {
      "new-header": "new-value"
    }
}

If the parse_llm_response_json_instructions parameter is set to true, AI Gateway will parse these instructions and set the specified response headers, response status code, and replacement response body. This lets you change specific headers such as Content-Type, or throw errors from the LLM.