Debug Kong Gateway requests

Uses: Kong Gateway

Enable request debugging

Request debugging is enabled by default and has the following configurations in kong.conf:

request_debug = on
request_debug_token = <random>

The debug token prevents feature abuse because only authorized personnel can issue debug requests. By default, Kong Gateway generates a token randomly when it starts, restarts, and reloads, but you can also define a token explicitly with the request_debug_token.

You can find the debug token in the following locations:

Kong Gateway error log: The debug token is logged in the error log (notice level) when Kong Gateway starts, restarts, or reloads. The log line will have the [request-debug] prefix to aid in searching.
Filesystem: The debug token is also stored in a file at /.request_debug_token and updated when Kong Gateway starts, restarts, or reloads.

Debug request configuration

To debug a request, add the following request headers:

At a minimum, you should set the X-Kong-Request-Debug header.
If the requests originate from anywhere other than the loopback addresses, you also need the X-Kong-Request-Debug-Token header.

X-Kong-Request-Debug header

If the X-Kong-Request-Debug header is set to *, timing information will be collected and exported for the current request.

You can also specify a comma-separated list of filters to reduce the scope of the time information that is collected. Each filter specifies which phase to collect timing information from. The following filters are supported:

rewrite
access
balancer
response
header_filter
body_filter
upstream
log

If this header isn’t present or contains an unknown value, timing information will not be collected for the current request.

X-Kong-Request-Debug-Log header

If the X-Kong-Request-Debug-Log header is set to true, timing information will also be logged in the Kong Gateway error log with a log level of notice. By default, the X-Kong-Request-Debug-Log header is set to false. The log line will have the [request-debug] prefix to aid in searching.

X-Kong-Request-Debug-Token header

The X-Kong-Request-Debug-Token is the token used to authenticate the client and make debug request. This prevents debugging abuse. Debug requests originating from loopback addresses don’t require this header.

X-Kong-Request-Id header v3.5+

The X-Kong-Request-Id header contains a unique identifier for each client request. This is enabled by default both upstream and downstream. This unique ID helps in matching specific requests to their corresponding error logs, which is useful for debugging. If Kong Gateway returns an error by calling the PDK kong.response.error, the request ID will also be included in the response body generated by Kong Gateway. In addition, any Kong Gateway error log generated will contain the same request ID, with the format: request_id: xxx.

The log line produced by the debug header as well as the debug response header contains the request ID. You can use this to search for the debug header output using a log viewer UI. This is especially useful when the debug output is too long to fit in the response header. You can see an example log line that includes the request ID in the Truncation for large debugging output debug request example.

Debug request example

The following is an example debug request:

curl http://localhost:8000/example \
    -H "X-Kong-Request-Debug: *" \
    -H "X-Kong-Request-Debug-Token: $TOKEN"

Here’s an example of the output of the response header:

{
    "child": {
        "rewrite": {
            "total_time": 0
        },
        "access": {
            "child": {
                "dns": {
                    "child": {
                        "example.com": {
                            "child": {
                                "resolve": {
                                    "total_time": 0,
                                    "cache_hit": true
                                }
                            },
                            "total_time": 0
                        }
                    },
                    "total_time": 0
                },
                "plugins": {
                    "child": {
                        "rate-limiting": {
                            "child": {
                                "176928d4-0949-47c8-8114-19cac8f86aab": {
                                    "child": {
                                        "redis": {
                                            "total_time": 1,
                                            "child": {
                                                "connections": {
                                                    "child": {
                                                        "tcp://localhost:6379": {
                                                            "child": {
                                                                "connect": {
                                                                    "child": {
                                                                        "dns": {
                                                                            "child": {
                                                                                "localhost": {
                                                                                    "child": {
                                                                                        "resolve": {
                                                                                            "total_time": 0,
                                                                                            "cache_hit": true
                                                                                        }
                                                                                    },
                                                                                    "total_time": 0
                                                                                }
                                                                            },
                                                                            "total_time": 0
                                                                        }
                                                                    },
                                                                    "total_time": 0
                                                                }
                                                            },
                                                            "total_time": 0
                                                        }
                                                    },
                                                    "total_time": 0
                                                }
                                            }
                                        }
                                    },
                                    "total_time": 2
                                }
                            },
                            "total_time": 2
                        },
                        "request-transformer": {
                            "child": {
                                "cfd2d953-ad82-453c-9979-b7573f52c226": {
                                    "total_time": 0
                                }
                            },
                            "total_time": 0
                        }
                    },
                    "total_time": 2
                }
            },
            "total_time": 3
        },
        "log": {
            "child": {
                "plugins": {
                    "child": {
                        "http-log": {
                            "child": {
                                "22906259-2963-4c6d-96a1-6d36d21714e3": {
                                    "total_time": 4
                                }
                            },
                            "total_time": 4
                        }
                    },
                    "total_time": 4
                }
            },
            "total_time": 4
        },
        "header_filter": {
            "child": {
                "plugins": {
                    "child": {
                        "response-transformer": {
                            "child": {
                                "dee98076-a58f-490d-8f7b-8523506bf96d": {
                                    "total_time": 1
                                }
                            },
                            "total_time": 1
                        }
                    },
                    "total_time": 1
                }
            },
            "total_time": 1
        },
        "body_filter": {
            "child": {
                "plugins": {
                    "child": {
                        "response-transformer": {
                            "child": {
                                "dee98076-a58f-490d-8f7b-8523506bf96d": {
                                    "total_time": 0
                                }
                            },
                            "total_time": 1
                        }
                    },
                    "total_time": 1
                }
            },
            "total_time": 1
        },
        "balancer": {
            "total_time": 0
        },
        "upstream": {
            "total_time": 152,
            "child": {
                "time_to_first_byte": {
                    "total_time": 151
                },
                "streaming": {
                    "total_time": 1
                }
            }
        }
    },
    "request_id": "0208903e83001d216bee5435dbc5ed25"
}

If you analyze the example debug output, you can see that:

The unit of total_time is millisecond.
The DNS resolution for example.host was cached, which is why it’s so fast in the example.
The upstream took 100ms in this request:
- The elapsed time from Kong Gateway sending the request to the upstream to Kong Gateway receiving the first byte is 20ms.
- The elapsed time from the first byte to the last byte from the upstream is 80ms.

You can also filter the debug output:

curl http://localhost:8000/example \
    -H "X-Kong-Request-Debug: upstream" \
    -H "X-Kong-Request-Debug-Token: $TOKEN"

This will return something like this:

{
  "request_id": "a1a1530f8ddb6f6f2462916ae002b715",
  "child": {
    "upstream": {
      "total_time": 363,
      "child": {
        "time_to_first_byte": {
          "total_time": 363
        }
      }
    }
  }
}

Truncation for large debugging output

The downstream system may impose a size restriction on response headers, leading Kong Gateway to truncate the X-Kong-Request-Output if it exceeds 2KB. This truncated output will be unconditionally logged in the error_log.

curl http://localhost:8000/large_debugging_output \
    -H "X-Kong-Request-Debug: *" \
    -H "X-Kong-Request-Debug-Token: $TOKEN"

This will return something like the following:

{
  "request_id": "60ca0a4f8e5e936c43692f49b27d2932",
  "truncated": true,
  "message": "Output is truncated, please check the error_log for full output by filtering with the request_id."
}

Debug request outputs that exceed 3KB are split into multiple parts with the request_id as an identifier.

Note: The debug output doesn’t have a consistent pattern and may change in the future. It wasn’t designed to be processed by automated tools. Rather, it was intended for human readability.