File Log plugin output mixed with Kong Gateway log entries

Uses: Kong Gateway
TL;DR

Why does the output of the File Log plugin get mixed with Kong Gateway log entries?

Logs interleave because the Linux kernel can’t guarantee atomicity for write() calls larger than the PIPE_BUF limit (4096 bytes), and this limit can’t be changed. Use custom_fields_by_lua to remove unneeded fields from the File Log output and keep entries under the limit.

When running in Kubernetes, some of Kong Gateway’s logs are directed to /dev/stdout. When the File Log plugin is also configured to write to /dev/stdout, the output can get mixed with Kong Gateway log entries.

In containerized environments, the Docker or Kubernetes host node collects logs via a PIPE that every container outputs to /dev/stdout.

When writing data through a PIPE, the data must fit within the PIPE buffer, which is usually 4096 bytes. When writing data larger than 4 KB through a PIPE, the Linux kernel can’t ensure the atomicity of the write() syscall.

This explains why the interleaving occurs for logs larger than 4 KB. There’s no config setting to increase PIPE_BUF directly, as it’s hard-coded in the kernel.

The File Log plugin uses write() directly to output to a file. This is a blocking I/O operation, which can affect performance, and there’s no locking mechanism in /dev/stdout.

As a workaround, we recommend removing any unneeded response headers or data from the File Log output using the custom_fields_by_lua field.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!