OpenTelemetry tracing is now the standard for distributed tracing. Use the OpenTelemetry tracing guidance on this page to set up tracing in your environment.
Kong Gateway tracing
Core tracing instrumentations
To use tracing instrumentations, you must enable a plugin that uses Kong’s Tracing API:
Kong Gateway provides a set of core instrumentations for tracing, these can be configured in the tracing_instrumentations
configuration in kong.conf
:
Parameter | Description |
---|---|
tracing_instrumentations
Default: off
|
Comma-separated list of tracing instrumentations this node should load. By default, no instrumentations are enabled. Valid values for this setting are:
Note: In the current implementation, tracing instrumentations are not enabled in stream mode. |
Header propagation
The tracing API supports propagating the following headers:
w3c
b3
,b3-single
jaeger
ot
The tracing API detects the propagation format from the headers, and uses the appropriate format to propagate the span. If no appropriate format is found, it falls back to the default format, which can be user-specified.
The propagation API works for both the OpenTelemetry plugin and the Zipkin plugin.
Headers
The headers parameter in kong.conf
lists supported tracing headers:
Parameter | Description |
---|---|
headers
Default: server_tokens, latency_tokens, X-Kong-Request-Id
|
Comma-separated list of headers Kong should inject in client responses. Accepted values are:
In addition to these, this value can be set
to Example: |
X-Kong-Request-Id header v3.5+
The X-Kong-Request-Id
header is enabled by default and provides a unique ID for every client request, both upstream and downstream. This ID is especially useful for debugging, as it links specific requests to their corresponding error logs.
When Kong Gateway returns an error using the PDK function kong.response.error
, the request ID is included in both the response body and the error logs, formatted as request_id: xxx
.
The same ID appears in the debug header and debug response header, allowing you to trace requests in a log viewer UI. This is especially useful when the debug output is too long to fit in the response header.
Tracing API
The tracing API is available under the kong.tracing
namespace and follows the OpenTelemetry API specification. This specification defines how to use the API to instrument your module.
If you’re already familiar with the OpenTelemetry API, you’ll find the kong.tracing
API intuitive and consistent with those standards.
Using the tracing API, you can configure your module with the following operations:
Create a tracer
Kong Gateway uses a global tracer internally to instrument the core modules and plugins.
By default, the tracer is a NoopTracer. The tracer is first initialized when the tracing_instrumentations
configuration is enabled.
You can create a new tracer manually, or use the global tracer instance:
local tracer
-- Create a new tracer
tracer = kong.tracing.new("custom-tracer")
-- Use the global tracer
tracer = kong.tracing
Sampling traces
Configure the sampling rate of a tracer:
local tracer = kong.tracing.new("custom-tracer", {
-- Set the sampling rate to 0.1
sampling_rate = 0.1,
})
A sampling_rate
of 0.1
means that 1 of every 10 requests will be traced. A rate of 1
means that all requests will be traced.
Create a span
A span represents a single operation within a trace. Spans can be nested to form trace trees. Each trace contains a root span, which typically describes the entire operation and, optionally, one or more sub-spans for its sub-operations.
local tracer = kong.tracing
local span = tracer:start_span("my-span")
The span properties can be set by passing a table to the start_span
method:
local span = tracer:start_span("my-span", {
start_time_ns = ngx.now() * 1e9, -- override the start time
span_kind = 2, -- SPAN_KIND
-- UNSPECIFIED: 0
-- INTERNAL: 1
-- SERVER: 2
-- CLIENT: 3
-- PRODUCER: 4
-- CONSUMER: 5
should_sample = true, -- by setting it to `true` to ignore the sampling decision
})
Make sure to end the span when you are done:
span:finish() -- ends the span
Note: The span table will be cleared and put into the table pool after the span is finished. Don’t use it after the span is finished.
Get or set the active span
The active span is the span that is currently being executed.
To avoid overheads, the active span is manually set by calling the set_active_span
method.
When you finish a span, the active span becomes the parent of the finished span.
Set or get the active span:
local tracer = kong.tracing
local span = tracer:start_span("my-span")
tracer.set_active_span(span)
local active_span = tracer.active_span() -- returns the active span
Scope
The tracers are scoped to a specific context by a namespace key.
To get the active span for a specific namespace, you can use the following:
-- get global tracer's active span, and set it as the parent of new created span
local global_tracer = kong.tracing
local tracer = kong.tracing.new("custom-tracer")
local root_span = global_tracer.active_span()
local span = tracer.start_span("my-span", {
parent = root_span
})
Set the span attributes
The attributes of a span are a map of key-value pairs
and can be set by passing a table to the set_attributes
method.
local span = tracer:start_span("my-span")
The OpenTelemetry specification defines the general semantic attributes. You can use it to describe the span. It could also be meaningful to visualize the span in a UI.
span:set_attribute("key", "value")
The following are defined semantic conventions for spans:
- General: General semantic attributes that may be used in describing different kinds of operations.
- HTTP: For HTTP client and server spans.
- Database: For SQL and NoSQL client call spans.
- RPC/RMI: For remote procedure call (e.g., gRPC) spans.
- Messaging: For messaging systems (queues, publish/subscribe, etc.) spans.
- FaaS: For Function as a Service (e.g., AWS Lambda) spans.
- Exceptions: For recording exceptions associated with a span.
- Compatibility: For spans generated by compatibility components, e.g. OpenTracing Shim layer.
Set the span events
The events of a span are time-series events that can be set by passing a table to the add_event
method:
local span = kong.tracing:start_span("my-span")
span:add_event("my-event", {
-- attributes
["key"] = "value",
})
Record error message
The event can also be used to record error messages:
local span = kong.tracing:start_span("my-span")
span:record_error("my-error-message")
-- or (same as above)
span:add_event("exception", {
["exception.message"] = "my-error-message",
})
Set the span status
The status of a span is a status code and can be set by passing a table to the set_status
method:
local span = kong.tracing:start_span("my-span")
-- Status codes:
-- - `0` unset
-- - `1` ok
-- - `2` error
span:set_status(2)
Release the span (optional)
The spans are stored in a pool, and can be released by calling the release
method:
local span = kong.tracing:start_span("my-span")
span:release()
By default, the span will be released after the Nginx request ends.
Visualize the trace
Because the traces are compatible with OpenTelemetry, they can be natively visualized through any OpenTelemetry UI.
See the OpenTelemetry plugin to learn how to visualize the traces.
Write a custom trace exporter
Kong Gateway bundled the OpenTelemetry plugin in core with a implementation of OTLP/HTTP, but you can still write your own exporter at scale.
To write a custom trace exporter, you must gather the spans. The spans are stored in the tracer’s buffer. The buffer is a queue of spans that are waiting to be sent to the backend.
You can access the buffer and process the span using the span_processor
function:
-- Use the global tracer
local tracer = kong.tracing
-- Process the span
local span_processor = function(span)
-- clone the span so it can be processed after the original one is cleared
local span_dup = table.clone(span)
-- you can transform the span, add tags, etc. to other specific data structures
end
The span_processor
function should be called in the log
phase of the plugin.
See Github for an example of a custom trace exporter.
FAQs
When did OpenTelemetry replace the Granular Tracing feature?
Granular Tracing was removed from Kong Gateway starting in 3.7,
and configurations like tracing = on
are no longer available. Instead, use the
OpenTelemetry tracing (tracing_instrumentations
) described on this page.