Troubleshooting with Konnect Debugger

Konnect Debugger provides a connected debugging experience and real-time trace-level visibility into API traffic, enabling you to:

• Troubleshoot issues: Investigate and resolve problems during deployments or incidents with targeted, on-demand traces.
• Understand request lifecycle: Visualize exactly what happened during a specific request, including order and duration plugin execution. See Debugger spans for a list of spans captured.
• Improve performance and reliability: Use deep insights to fine-tune configurations and resolve bottlenecks.

Kong Konnect’s Debugger provides exclusive, in-depth insights not available through third-party telemetry tools. The detailed traces captured during a live session are unique to Kong and offer unparalleled visibility into system behavior.

Debugger works best for investigating specific issues, and is not recommended for continuous monitoring. Use it for investigating problems like:

• API errors
• Authentication failures
• Plugin execution problems
• Unexpected routing behavior
• Performance degradation
• Upstream connectivity issues

To get the most out of a session, we recommend keeping the following practices in mind:

• Start with a clear objective. A session without a specific question to answer generates unnecessary telemetry and makes root-cause analysis harder.
• Use the smallest scope possible. Before you start, narrow the session to a specific Route, Service, Consumer, or request attribute. Scope to an entire control plane only when you need to, and use advanced sampling rules.

Konnect Debugger allows you to capture traces and logs.

Traces

Traces provide a visual representation of the request and response lifecycle, offering a comprehensive overview of Kong’s request processing pipeline.

The debugger helps capture OpenTelemetry-compatible traces for all requests matching the sampling criteria. The detailed spans are captured for the entire request/response lifecycle. These traces can be visualized with Konnect’s built-in span viewer with no additional instrumentation or telemetry tools. For a complete list of available spans and their meanings, see Debugger spans.

Key Highlights

• Traces can be generated for a service or per route
• Refined traces can be generated for all requests matching a sampling criteria
• Sampling criteria can be defined with simple expressions language, for example: http.method == GET
• Trace sessions are retained for up to 15 days
• Traces can be visualized in Konnect’s built in trace viewer

To ensure consistency and interoperability, tracing adheres to OpenTelemetry naming conventions for spans and attributes, wherever possible.

Logs

For deeper insights, logs can be captured along with traces. When initiating a debug session, administrators can choose to capture logs. Detailed Kong Gateway logs are captured for the duration of the session. These logs are then correlated with traces using trace_id and span_id providing a comprehensive and drill-down view of logs generated during specific trace or span.

Traces captured during a debug session can be visualized in debugger’s built-in trace viewer. The trace viewer displays Summary, Spans and Logs view. You can gain instant insights with the summary view while the spans and logs view help you to dive deeper. You can also download captured traces.

All Konnect users can upload a trace .json or .zip file (for example, one downloaded from a debug session) to view by navigating to Observability > Debugger and clicking Upload trace. This lets users view traces even if they don’t have permission to create or start debug sessions.

Summary view

Summary view helps you visualize the entire API request-response flow in a single glance. This view provides a concise overview of critical latency metrics and a transaction map. The lifecycle map includes the different phases of Kong Gateway and the plugins executed by Kong Gateway on both the request and the response along with the times spent in each phase.

Use the summary view to quickly understand the end-to-end API flow, identify performance bottlenecks, and optimize your API strategy:

Spans view

The spans view gives you unparalleled visibility into Kong Gateway’s internal workings. This detailed view breaks down into individual spans, providing a comprehensive understanding of:

• Kong Gateway’s internal processes and phases
• Plugin execution and performance
• Request and response handling

For detailed definitions of each span, see Debugger spans.

Use the spans view to troubleshoot issues, optimize performance, and refine your configuration:

Logs view

The logs view gives you a drill-down view of all the logs generated during specific debug session. All the spans in the trace are correlated using trace_id and span_id. The logs can be filtered on log level and spans. Logs are displayed in reverse chronological order. Konnect encrypts all the logs that are ingested. You can further ensure complete privacy and control by using customer-managed encryption keys (CMEK).

Use the logs view to quickly troubleshoot and pinpoint issues:

Analyze with KAi

When viewing a trace, you can click Analyze with KAi to send the trace to KAi. The trace summary, spans, and logs are automatically applied as context. KAi can help identify errors, determine their probable root cause, and answer questions about the request, such as:

• Why is my API returning 5xx errors?
• Why is the OIDC plugin execution failing?
• Why is the /flights API slow?
• What is the plugin execution order for this route?
• Why does my hybrid gateway have connectivity issues with the upstream?

In critical scenarios, having access to payload details can help identify and pinpoint failures. Payload capture lets you configure a debug session to capture headers and/or body for requests and responses.

Payload capture records the following:

• The client request and response
• v3.14+ The upstream request and response
• v3.14+ Headers and body before and after each plugin runs, including custom plugins
• v3.14+ The contents of calls Kong Gateway makes to third-party dependencies during request processing, such as Redis for rate limiting, an identity provider for token validation, or DNS for resolution
• v3.14+ Gzip-encoded bodies (Content-Encoding: gzip or x-gzip), automatically decompressed so they appear as readable text

You can use this information to compare the sent request with the received request, and the sent response with the received response, to identify where a failure occurred.

Due to the nature of this telemetry, you must explicitly opt in by signing the Advanced Features Addendum. Contact your organization admin or your Kong representative to get started.

Prerequisites

• Your organization has opted in to the Debugger payload capture feature and signed the Advanced Features Addendum.

• Your data plane nodes are deployed with the following telemetry endpoints:

   KONG_CLUSTER_CONTROL_PLANE=xxx.us.cp.konghq.com:443
   KONG_CLUSTER_SERVER_NAME=xxx.us.cp.konghq.com
   KONG_CLUSTER_TELEMETRY_ENDPOINT=xxx.us.tp.konghq.com:443
   KONG_CLUSTER_TELEMETRY_SERVER_NAME=xxx.us.tp.konghq.com

  Copied!
• Your firewall rules allow traffic to the telemetry endpoints.

Payload collection and sanitization

When you start a debug session with payload capture enabled, the Debugger captures headers and/or body for all requests matching the sampling criteria. Sampling filters and sanitization run on the data plane before any data is transmitted to Konnect.

The log sanitizer scrubs transactions and redacts sensitive data like credit card numbers from the payload. Authentication and identity headers (for example, Authorization, API key header values, and Consumer ID header fields) are also masked by default.

Note: The log sanitizer uses the Luhn algorithm, a well-known algorithm to validate credit card numbers, International Mobile Equipment Identity (IMEI) numbers, and other sensitive numerical data. Matched characters are replaced with *.

Custom masking rules v3.14+

You can define custom payload masking rules to target specific sensitive data in your requests and responses. Custom rules let you redact data in both headers and body content.

Header rules

Header masking rules let you redact the value of specific headers by name.

Body rules

Body masking rules support two strategies:

• JSONPath (RFC 9535): Target specific fields in JSON payloads using standard JSONPath expressions. This includes support for dot notation ($.field), bracket notation, wildcards ([*]), recursive descent ($..), array slicing, and filter expressions.
• Regex (PCRE): Match and redact patterns in the raw body content using PCRE-compatible regular expressions.

Matched content is replaced with *.

Note: Custom masking rules are applied in addition to the built-in credit card redaction. The built-in Luhn algorithm-based redaction is always active and cannot be disabled.

AI Gateway v3.14+

When a Route has AI plugins configured, payload capture shows the standard payloads in addition to the AI Gateway details view that shows the prompt at each stage of the pipeline:

• What the client sent
• What each AI plugin did to the prompt, for example PII sanitization or content filtering
• The final prompt sent to the model provider, after all plugins have run
• What the model returned, and how any response plugins transformed it

Use this to confirm guardrails fired, policies were enforced, and transformations produced the prompt you expected. You can also diagnose why a semantic cache hit or missed, or inspect what a guardrail service returned.

Payload ingestion, storage, and retention

By default, Konnect encrypts captured payloads with a default encryption key provisioned for your org. You can also configure Konnect to use customer-managed encryption keys (CMEK). Konnect supports symmetric key encryption and integrates with AWS Key Management Service (KMS).

Debug sessions with payload data are retained for up to 3 days, after which they are purged from Konnect.

By default, logs are automatically encrypted using encryption keys that are owned and managed by Konnect. However if you have a specific compliance and regulatory requirements related to the keys that protect your data, you can use the customer-managed encryption keys. This ensures that sensitive data are secured for each organization with their own key and nobody, including Konnect, has access to that data. For more information about how to create and manage CMEK keys, see Customer-Managed Encryption Keys (CMEK).

To begin using the Debugger, ensure the following requirements are met:

• Your data plane nodes are running Kong Gateway version 3.9.1 or later.
• Logs require Kong Gateway version 3.11.0 or later.
• You need Debug Session Creator, Control Plane Admin, or Org Admin permissions to create debug sessions.
• Your Konnect data planes are hosted using self-managed hybrid, Dedicated Cloud Gateways, or serverless gateways. Kong Ingress Controller or Kong Event Gateway Gateways aren’t currently supported.
• For version 3.9.x only: set the following environment variables in kong.conf:
  • KONG_CLUSTER_RPC=on
  • KONG_ACTIVE_TRACING=on

From version 3.10 and later, these environment variables are enabled by default and no manual configuration is required.

1. In the Konnect sidebar, click Observability.
2. In the Observability sidebar, click Debugger.
3. Click New session.
4. Define the sampling criteria.
5. Click Start Session.

Once the session starts, traces will be captured for requests that match the rule. Click a trace to view it in the span viewer.

You can also start a debug session from the overview page of a control plane, Gateway Service, or Route by clicking the Actions dropdown menu and clicking Start Debugging. Additionally, you can use KAi to debug as well.

Each session can be configured to run for a time between 10 seconds and 30 minutes. Sessions are retained for up to 15 days.

For details on defining sampling rules, see Sampling rules.

Sampling rules help you capture only relevant traffic. Requests that match the defined criteria are included in the session. There are two types:

• Basic sampling rules: Filter by Route or Service.
• Advanced sampling rules: Use expressions for fine-grained filtering.

For example, to capture all requests with a 503 response:

http.response.status_code==503

The list of all possible sampling expressions are captured here Sampling Rules

A sample trace is shown below. By inspecting the spans, you can see that the bulk of the latency occurs in the pre-function plugin during the access phase.

The following scenarios show how to scope a session for a specific problem and what the captured trace data can tell you.