When deploying Observability on Kong Mesh you may run into a situation where tracing is not gathering data as you expected. What are some common pitfalls and ways to troubleshoot this further?
Kong Mesh: Observability/tracing troubleshooting steps
Uses:
Kong Gateway
TL;DR
What are common pitfalls when troubleshooting Kong Mesh Observability/tracing, and how do I diagnose them?
If tracing isn’t collecting the expected data, gather the Envoy config dump, Kong Mesh debug logs, the Mesh resource, and the TrafficTrace (or MeshTrace) policy, then verify the tracing backend and policy are configured correctly and check the logs for errors.
In the config dump, confirm the expected listeners are present — if the protocol isn’t explicitly configured per service (HTTP, HTTP2, or gRPC), jaeger will only show up under config.bootstrap.cluster.name and not as a listener.
Solution
Note: TrafficTrace, referenced below, is deprecated in favor of MeshTrace. For new setups, use MeshTrace instead. The troubleshooting technique described here remains valid for either policy.
Prerequisites:
We will be referencing the kuma-demo app.
We will also assume you’ve followed this guide.
Files to gather:
-
Config Dump from envoy sidecar for app that is communicating with Jaeger (in the demo app we can use the frontend):
kubectl port-forward svc/frontend > -n kuma-demo 9901:9901 2>1 &Access the UI through localhost:9901 and gather the
config_dump. -
Kong Mesh debug logs
-
Mesh deployment file
k get mesh <mesh name> -o yaml > meshdata.yaml -
Traffic Trace Policy
k get traffictrace -o yaml > TTData.yaml
Troubleshooting steps:
- Verify Tracing backend is configured correctly
- Verify Traffic Trace policy is configured correctly
- Verify any errors in the logs.
Debugging the config dump can be intimidating as there is a lot of different data to review. One key thing to look for would be to verify if the expected Listeners are deployed.
Example:
Tracing is only supported over HTTP, HTTP2, gRPC protocols. So you must explicitly specify the protocol for each service.
If the protocol is not configured, then the config dump will only reference jaeger under config.bootstrap.cluster.name.
"name": "tracing:jaeger-collector"In a working scenario, jaeger would be referenced as listeners as well.