Kong Gateway tracing

Uses: Kong Gateway

OpenTelemetry tracing is now the standard for distributed tracing. Use the OpenTelemetry tracing guidance on this page to set up tracing in your environment.

Core tracing instrumentations

To use tracing instrumentations, you must enable a plugin that uses Kong’s Tracing API:

Kong Gateway provides a set of core instrumentations for tracing, these can be configured in the tracing_instrumentations configuration in kong.conf:

Parameter Description
tracing_instrumentations Default: off

Comma-separated list of tracing instrumentations this node should load. By default, no instrumentations are enabled.

Valid values for this setting are:

  • off: do not enable instrumentations.
  • request: only enable request-level instrumentations.
  • all: enable all the following instrumentations.
  • db_query: trace database queries.
  • dns_query: trace DNS queries.
  • router: trace router execution, including router rebuilding.
  • http_client: trace OpenResty HTTP client requests.
  • balancer: trace balancer retries.
  • plugin_rewrite: trace plugin iterator execution with rewrite phase.
  • plugin_access: trace plugin iterator execution with access phase.
  • plugin_header_filter: trace plugin iterator execution with header_filter phase.

Note: In the current implementation, tracing instrumentations are not enabled in stream mode.

Header propagation

The tracing API supports propagating the following headers:

The tracing API detects the propagation format from the headers, and uses the appropriate format to propagate the span. If no appropriate format is found, it falls back to the default format, which can be user-specified.

The propagation API works for both the OpenTelemetry plugin and the Zipkin plugin.

Headers

The headers parameter in kong.conf lists supported tracing headers:

Parameter Description
headers Default: server_tokens, latency_tokens, X-Kong-Request-Id

Comma-separated list of headers Kong should inject in client responses.

Accepted values are:

  • Server: Injects Server: kong/x.y.z on Kong-produced responses (e.g., Admin API, rejected requests from auth plugin).
  • Via: Injects Via: kong/x.y.z for successfully proxied requests.
  • X-Kong-Proxy-Latency: Time taken (in milliseconds) by Kong to process a request and run all plugins before proxying the request upstream.
  • X-Kong-Response-Latency: Time taken (in milliseconds) by Kong to produce a response in case of, e.g., a plugin short-circuiting the request, or in case of an error.
  • X-Kong-Upstream-Latency: Time taken (in milliseconds) by the upstream service to send response headers.
  • X-Kong-Admin-Latency: Time taken (in milliseconds) by Kong to process an Admin API request.
  • X-Kong-Upstream-Status: The HTTP status code returned by the upstream service. This is particularly useful for clients to distinguish upstream statuses if the response is rewritten by a plugin.
  • X-Kong-Request-Id: Unique identifier of the request.
  • server_tokens: Same as specifying both Server and Via.
  • latency_tokens: Same as specifying X-Kong-Proxy-Latency, X-Kong-Response-Latency, X-Kong-Admin-Latency, and X-Kong-Upstream-Latency.

In addition to these, this value can be set to off, which prevents Kong from injecting any of the above headers. Note that this does not prevent plugins from injecting headers of their own.

Example: headers = via, latency_tokens

X-Kong-Request-Id header v3.5+

The X-Kong-Request-Id header is enabled by default and provides a unique ID for every client request, both upstream and downstream. This ID is especially useful for debugging, as it links specific requests to their corresponding error logs.

When Kong Gateway returns an error using the PDK function kong.response.error, the request ID is included in both the response body and the error logs, formatted as request_id: xxx.

The same ID appears in the debug header and debug response header, allowing you to trace requests in a log viewer UI. This is especially useful when the debug output is too long to fit in the response header.

Tracing API

The tracing API is available under the kong.tracing namespace and follows the OpenTelemetry API specification. This specification defines how to use the API to instrument your module.

If you’re already familiar with the OpenTelemetry API, you’ll find the kong.tracing API intuitive and consistent with those standards.

Using the tracing API, you can configure your module with the following operations:

Create a tracer

Kong Gateway uses a global tracer internally to instrument the core modules and plugins.

By default, the tracer is a NoopTracer. The tracer is first initialized when the tracing_instrumentations configuration is enabled.

You can create a new tracer manually, or use the global tracer instance:

local tracer

-- Create a new tracer
tracer = kong.tracing.new("custom-tracer")

-- Use the global tracer
tracer = kong.tracing

Sampling traces

Configure the sampling rate of a tracer:

local tracer = kong.tracing.new("custom-tracer", {
  -- Set the sampling rate to 0.1
  sampling_rate = 0.1,
})

A sampling_rate of 0.1 means that 1 of every 10 requests will be traced. A rate of 1 means that all requests will be traced.

Create a span

A span represents a single operation within a trace. Spans can be nested to form trace trees. Each trace contains a root span, which typically describes the entire operation and, optionally, one or more sub-spans for its sub-operations.

local tracer = kong.tracing

local span = tracer:start_span("my-span")

The span properties can be set by passing a table to the start_span method:

local span = tracer:start_span("my-span", {
  start_time_ns = ngx.now() * 1e9, -- override the start time
  span_kind = 2, -- SPAN_KIND
                  -- UNSPECIFIED: 0
                  -- INTERNAL: 1
                  -- SERVER: 2
                  -- CLIENT: 3
                  -- PRODUCER: 4
                  -- CONSUMER: 5
  should_sample = true, -- by setting it to `true` to ignore the sampling decision
})

Make sure to end the span when you are done:

span:finish() -- ends the span

Note: The span table will be cleared and put into the table pool after the span is finished. Don’t use it after the span is finished.

Get or set the active span

The active span is the span that is currently being executed.

To avoid overheads, the active span is manually set by calling the set_active_span method. When you finish a span, the active span becomes the parent of the finished span.

Set or get the active span:

local tracer = kong.tracing
local span = tracer:start_span("my-span")
tracer.set_active_span(span)

local active_span = tracer.active_span() -- returns the active span

Scope

The tracers are scoped to a specific context by a namespace key.

To get the active span for a specific namespace, you can use the following:

-- get global tracer's active span, and set it as the parent of new created span
local global_tracer = kong.tracing
local tracer = kong.tracing.new("custom-tracer")

local root_span = global_tracer.active_span()
local span = tracer.start_span("my-span", {
  parent = root_span
})

Set the span attributes

The attributes of a span are a map of key-value pairs and can be set by passing a table to the set_attributes method.

local span = tracer:start_span("my-span")

The OpenTelemetry specification defines the general semantic attributes. You can use it to describe the span. It could also be meaningful to visualize the span in a UI.

span:set_attribute("key", "value")

The following are defined semantic conventions for spans:

  • General: General semantic attributes that may be used in describing different kinds of operations.
  • HTTP: For HTTP client and server spans.
  • Database: For SQL and NoSQL client call spans.
  • RPC/RMI: For remote procedure call (e.g., gRPC) spans.
  • Messaging: For messaging systems (queues, publish/subscribe, etc.) spans.
  • FaaS: For Function as a Service (e.g., AWS Lambda) spans.
  • Exceptions: For recording exceptions associated with a span.
  • Compatibility: For spans generated by compatibility components, e.g. OpenTracing Shim layer.

Set the span events

The events of a span are time-series events that can be set by passing a table to the add_event method:

local span = kong.tracing:start_span("my-span")
span:add_event("my-event", {
  -- attributes
  ["key"] = "value",
})

Record error message

The event can also be used to record error messages:

local span = kong.tracing:start_span("my-span")
span:record_error("my-error-message")

-- or (same as above)
span:add_event("exception", {
  ["exception.message"] = "my-error-message",
})

Set the span status

The status of a span is a status code and can be set by passing a table to the set_status method:

local span = kong.tracing:start_span("my-span")

-- Status codes:
-- - `0` unset
-- - `1` ok
-- - `2` error
span:set_status(2)

Release the span (optional)

The spans are stored in a pool, and can be released by calling the release method:

local span = kong.tracing:start_span("my-span")
span:release()

By default, the span will be released after the Nginx request ends.

Visualize the trace

Because the traces are compatible with OpenTelemetry, they can be natively visualized through any OpenTelemetry UI.

See the OpenTelemetry plugin to learn how to visualize the traces.

Write a custom trace exporter

Kong Gateway bundled the OpenTelemetry plugin in core with a implementation of OTLP/HTTP, but you can still write your own exporter at scale.

To write a custom trace exporter, you must gather the spans. The spans are stored in the tracer’s buffer. The buffer is a queue of spans that are waiting to be sent to the backend.

You can access the buffer and process the span using the span_processor function:

-- Use the global tracer
local tracer = kong.tracing

-- Process the span
local span_processor = function(span)
    -- clone the span so it can be processed after the original one is cleared
    local span_dup = table.clone(span)
    -- you can transform the span, add tags, etc. to other specific data structures
end

The span_processor function should be called in the log phase of the plugin.

See Github for an example of a custom trace exporter.

FAQs

Granular Tracing was removed from Kong Gateway starting in 3.7, and configurations like tracing = on are no longer available. Instead, use the OpenTelemetry tracing (tracing_instrumentations) described on this page.

Something wrong?

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!