Kong Gateway tracing

OpenTelemetry tracing is now the standard for distributed tracing. Use the OpenTelemetry tracing guidance on this page to set up tracing in your environment.

To use tracing instrumentations, you must enable a plugin that uses Kong’s Tracing API:

• OpenTelemetry plugin
• Zipkin plugin

Kong Gateway provides a set of core instrumentations for tracing, these can be configured in the tracing_instrumentations configuration in kong.conf:

The tracing API supports propagating the following headers:

• w3c
• b3, b3-single
• jaeger
• ot

The tracing API detects the propagation format from the headers, and uses the appropriate format to propagate the span. If no appropriate format is found, it falls back to the default format, which can be user-specified.

The propagation API works for both the OpenTelemetry plugin and the Zipkin plugin.

The headers parameter in kong.conf lists supported tracing headers:

X-Kong-Request-Id header v3.5+

The X-Kong-Request-Id header is enabled by default and provides a unique ID for every client request, both upstream and downstream. This ID is especially useful for debugging, as it links specific requests to their corresponding error logs.

When Kong Gateway returns an error using the PDK function kong.response.error, the request ID is included in both the response body and the error logs, formatted as request_id: xxx.

The same ID appears in the debug header and debug response header, allowing you to trace requests in a log viewer UI. This is especially useful when the debug output is too long to fit in the response header.

The tracing API is available under the kong.tracing namespace and follows the OpenTelemetry API specification. This specification defines how to use the API to instrument your module.

If you’re already familiar with the OpenTelemetry API, you’ll find the kong.tracing API intuitive and consistent with those standards.

Using the tracing API, you can configure your module with the following operations:

• Span
• Attributes

Create a tracer

Kong Gateway uses a global tracer internally to instrument the core modules and plugins.

By default, the tracer is a NoopTracer. The tracer is first initialized when the tracing_instrumentations configuration is enabled.

You can create a new tracer manually, or use the global tracer instance:

local tracer

-- Create a new tracer
tracer = kong.tracing.new("custom-tracer")

-- Use the global tracer
tracer = kong.tracing

Sampling traces

Configure the sampling rate of a tracer:

local tracer = kong.tracing.new("custom-tracer", {
  -- Set the sampling rate to 0.1
  sampling_rate = 0.1,
})

A sampling_rate of 0.1 means that 1 of every 10 requests will be traced. A rate of 1 means that all requests will be traced.

Create a span

A span represents a single operation within a trace. Spans can be nested to form trace trees. Each trace contains a root span, which typically describes the entire operation and, optionally, one or more sub-spans for its sub-operations.

local tracer = kong.tracing

local span = tracer:start_span("my-span")

The span properties can be set by passing a table to the start_span method:

local span = tracer:start_span("my-span", {
  start_time_ns = ngx.now() * 1e9, -- override the start time
  span_kind = 2, -- SPAN_KIND
                  -- UNSPECIFIED: 0
                  -- INTERNAL: 1
                  -- SERVER: 2
                  -- CLIENT: 3
                  -- PRODUCER: 4
                  -- CONSUMER: 5
  should_sample = true, -- by setting it to `true` to ignore the sampling decision
})

Make sure to end the span when you are done:

span:finish() -- ends the span

Note: The span table will be cleared and put into the table pool after the span is finished. Don’t use it after the span is finished.

Get or set the active span

The active span is the span that is currently being executed.

To avoid overheads, the active span is manually set by calling the set_active_span method. When you finish a span, the active span becomes the parent of the finished span.

Set or get the active span:

local tracer = kong.tracing
local span = tracer:start_span("my-span")
tracer.set_active_span(span)

local active_span = tracer.active_span() -- returns the active span

Scope

The tracers are scoped to a specific context by a namespace key.

To get the active span for a specific namespace, you can use the following:

-- get global tracer's active span, and set it as the parent of new created span
local global_tracer = kong.tracing
local tracer = kong.tracing.new("custom-tracer")

local root_span = global_tracer.active_span()
local span = tracer.start_span("my-span", {
  parent = root_span
})

Set the span attributes

The attributes of a span are a map of key-value pairs and can be set by passing a table to the set_attributes method.

local span = tracer:start_span("my-span")

The OpenTelemetry specification defines the general semantic attributes. You can use it to describe the span. It could also be meaningful to visualize the span in a UI.

span:set_attribute("key", "value")

The following are defined semantic conventions for spans:

• General: General semantic attributes that may be used in describing different kinds of operations.
• HTTP: For HTTP client and server spans.
• Database: For SQL and NoSQL client call spans.
• RPC/RMI: For remote procedure call (e.g., gRPC) spans.
• Messaging: For messaging systems (queues, publish/subscribe, etc.) spans.
• FaaS: For Function as a Service (e.g., AWS Lambda) spans.
• Exceptions: For recording exceptions associated with a span.
• Compatibility: For spans generated by compatibility components, e.g. OpenTracing Shim layer.

Set the span events

The events of a span are time-series events that can be set by passing a table to the add_event method:

local span = kong.tracing:start_span("my-span")
span:add_event("my-event", {
  -- attributes
  ["key"] = "value",
})

Record error message

The event can also be used to record error messages:

local span = kong.tracing:start_span("my-span")
span:record_error("my-error-message")

-- or (same as above)
span:add_event("exception", {
  ["exception.message"] = "my-error-message",
})

Set the span status

The status of a span is a status code and can be set by passing a table to the set_status method:

local span = kong.tracing:start_span("my-span")

-- Status codes:
-- - `0` unset
-- - `1` ok
-- - `2` error
span:set_status(2)

Release the span (optional)

The spans are stored in a pool, and can be released by calling the release method:

local span = kong.tracing:start_span("my-span")
span:release()

By default, the span will be released after the Nginx request ends.

Visualize the trace

Because the traces are compatible with OpenTelemetry, they can be natively visualized through any OpenTelemetry UI.

See the OpenTelemetry plugin to learn how to visualize the traces.

Kong Gateway bundled the OpenTelemetry plugin in core with a implementation of OTLP/HTTP, but you can still write your own exporter at scale.

To write a custom trace exporter, you must gather the spans. The spans are stored in the tracer’s buffer. The buffer is a queue of spans that are waiting to be sent to the backend.

You can access the buffer and process the span using the span_processor function:

-- Use the global tracer
local tracer = kong.tracing

-- Process the span
local span_processor = function(span)
    -- clone the span so it can be processed after the original one is cleared
    local span_dup = table.clone(span)
    -- you can transform the span, add tags, etc. to other specific data structures
end

The span_processor function should be called in the log phase of the plugin.

See Github for an example of a custom trace exporter.