Using Debugger for Datakit

Related Documentation
Minimum Version
Kong Gateway - 3.15
Incompatible with
on-prem

When you run a Debugger session on a Kong Gateway instance that uses the Datakit plugin, Debugger captures tracing data for each Datakit node in the pipeline. This gives you visibility into how data flows through your Datakit configuration, which nodes ran, what values they received and produced, and where errors or skips occurred.

Datakit tracing provides debugging detail without exposing sensitive data. Kong Gateway tracks values as they move through the Datakit workflow and applies redaction and sanitization rules before sending trace data to Konnect.

Enable tracing event collection

To collect Datakit tracing events, start a Konnect Debugger session with body payload capture enabled. If header-only payload capture is enabled, node spans will be recorded but tracing events won’t be uploaded.

Datakit node spans

For each Datakit node that executes, Debugger creates a span named kong.datakit.node.NODE_NAME, where NODE_NAME is the name you assigned to the node in your Datakit configuration.

Node spans capture metadata about node execution but don’t contain node input or output values. Values are only captured in the separate Datakit tracing event payload.

If a node is skipped due to branch routing rules, it won’t create a span because execution hasn’t started. Information about skipped nodes is available in Datakit tracing events.

Each span includes the following attributes:

Attribute

Description

Values

proxy.kong.datakit.node.type The type of the Datakit node. Depends on the node, for example, jq, call, or jwt_decode. See the Datakit nodes reference for all supported node types.
proxy.kong.datakit.node.status The final status of the node after execution. complete, fail, or cancel

Tracing events

Tracing events capture the full lifecycle of each node, including input and output values. Each event corresponds to a specific point in a node’s execution and includes the data the node was working with at that point.

Note: Large values may be omitted from tracing events if they exceed the capture size limit.

Each tracing event includes the following fields:

Field

Description

name The name of the Datakit node.
type The type of the Datakit node (for example, jq, call, jwt_decode).
action The lifecycle action that generated this event. See Actions.
timestamp The time the event occurred.
value The input or output value associated with this action.

The run action maps to input values, while the complete, fail, cancel actions map to output values.
error The error message, if the action is fail.

Actions

The action field describes what happened to the node at the moment the event was recorded.

Action

Value captured

Description

run Input value The node started running. The event includes the input value the node received.
complete Output value The node finished successfully. The event includes the output value the node produced.
fail Partial output value (if any) The node encountered an error. The event includes an error message and any partial output.
skip Routing value The node was skipped due to branch routing. The event shows the routing value that caused the skip.
cancel None The node was canceled before it completed.
schedule Input value call nodes only. The node was scheduled for an async HTTP request.
dispatch Request data call nodes only. The async HTTP request was dispatched.
resume Response data call nodes only. The async HTTP response was received and the node resumed execution.

Redaction and sanitization

Datakit values can carry arbitrary content, including Vault secrets, authentication headers, personally identifiable information (PII), response bodies, and transformed values. Datakit applies redaction and sanitization rules based on the source of the node’s input. Tracing values are sanitized on the data plane before being transmitted to Konnect.

The same custom masking rules you configure for general Debugger payload capture also apply to Datakit tracing events.

Vault-derived values

Any value that comes from a Vault secret is always replaced with ********, regardless of other sanitization rules. If a node’s output is derived from a Vault-resolved value, the entire output is redacted.

Header sanitization

When a Datakit value is a headers map (a structured map of header name-value pairs), it’s sanitized using header-name rules. The following headers are masked by default:

  • authorization
  • api-key
  • x-api-key
  • x-consumer-username
  • x-consumer-custom-id
  • x-consumer-id
  • x-credential-identifier

Header context is lost when a value passes through a transformation node like jq, json_to_xml, or xml_to_json. After transformation, the value no longer carries header context and is sanitized using body rules instead.

Body and structured value sanitization

String values are sanitized using body rules (regex or JSONPath). Structured values (maps and objects) are sanitized using body rules with JSONPath expressions applied to the structure. When a node’s output is assembled from individual field-level inputs, each field is sanitized independently according to the rules that apply to its source.

Node-specific sanitization behavior

The sanitization applied to a node’s tracing event values depends on the node type and the source of its inputs. In summary:

  • Values that come from request data use sanitizer rules configured for request data.
  • Values that come from response data use sanitizer rules configured for response data.
  • Values that combine request and response data can use both request and response sanitizer rules.

The following table breaks down sanitization behavior by node:

Node type

Input or output

Sanitization applied

request Headers Header-name rules
request Body, query parameters Body rules
response All values Based on connected inputs
service_request All values Based on connected inputs
service_response All values Based on connected inputs
vault Output Always ******** (vault-derived)
call Output headers Response header-name rules
call Output body Response body rules
static Output Only if the value matches a configured rule
property GET Output Only if the value matches a configured rule
property SET Output Based on input values
jq Output Body rules (header context lost after transformation)
json_to_xml Output Body rules (header context lost after transformation)
xml_to_json Output Body rules (header context lost after transformation)
jwt_decode Input JWT Always redacted
jwt_decode JWT signature Always redacted
jwt_decode JWT header and payload claims Body rules
jwt_verify Input token and key Always redacted
jwt_verify Output claims and header Body rules
jwt_sign Input key Always redacted
jwt_sign Output token Always redacted
jwt_sign Input claims and headers Body rules
cache GET Output Only if the value matches a configured rule
cache SET Cached data Based on input data source
cache SET Generated values Body rules
branch Routing values Body rules
exit Response body Based on input sources
exit Response headers Based on input sources

Custom sanitization rules

Custom payload sanitization rules configured for Debugger also apply to Datakit tracing values. Custom body rules can use regex or JSONPath expressions (RFC 9535).

For object inputs and outputs, each field is sanitized independently. When a node’s output is assembled from field-level inputs (for example, request.headers connected to jq.headers), Datakit preserves per-field sanitization for that output. Transformed $self outputs, like the full output of a jq node, are sanitized as a single value.

Write JSONPath rules relative to the value as it appears in the tracing event. For example, jwt_decode output is an object with header, payload, and signature fields:

{
  "header": {
    "typ": "JWT",
    "alg": "HS256"
  },
  "payload": {
    "sub": "user-123",
    "email": "alice@example.com"
  },
  "signature": "..."
}
  • To redact payload.sub, use $.payload.sub.
  • To redact header.typ, use $.header.typ.

If the jwt_decode output is then passed through a jq node that reshapes it, write your JSONPath rule relative to the jq output structure instead.

To redact any field named sub regardless of its position in the structure, use the recursive deep-scan JSONPath rule $..sub.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!