Send OpenTelemetry data to Grafana Cloud

We need to add a connection to OpenTelemetry in Grafana to generate the configuration we need.

1. Log in to Grafana and launch your Grafana Cloud stack.
2. In the sidebar, click Connections > Add new connection.
3. Click OpenTelemetry Collector.
4. In the Access Policy token name field, enter a name for the token, “kong-otel” for example.

5. Click Create token.

   From the configuration generated by Grafana, export the values under extensions.basicauth/grafana_cloud.client_auth to your environment:

   export GRAFANA_USERNAME='username value'
   export GRAFANA_PASSWORD='password value'
   

   Copied!

   Then export the URL under exporters.otlphttp/grafana_cloud.endpoint:

   export GRAFANA_ENDPOINT='endpoint value'
   

   Copied!

6. In a terminal, create a configuration file for OpenTelemetry Collector:

   nano otel-config.yaml
   

   Copied!

   Add the following content to the file and save it:

   extensions:
     basicauth/grafana_cloud:
       client_auth:
         username: ${env:GRAFANA_USERNAME}
         password: ${env:GRAFANA_PASSWORD}
     health_check:
     pprof:
       endpoint: 0.0.0.0:1777
     zpages:
       endpoint: 0.0.0.0:55679
      
   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:4317
         http:
           endpoint: 0.0.0.0:4318
      
     prometheus:
       config:
         scrape_configs:
         - job_name: 'otel-collector'
           scrape_interval: 10s
           static_configs:
           - targets: ['0.0.0.0:8888']
      
     jaeger:
       protocols:
         grpc:
           endpoint: 0.0.0.0:14250
         thrift_binary:
           endpoint: 0.0.0.0:6832
         thrift_compact:
           endpoint: 0.0.0.0:6831
         thrift_http:
           endpoint: 0.0.0.0:14268
      
     zipkin:
       endpoint: 0.0.0.0:9411
      
   processors:
     batch:
      
   exporters:
     otlphttp/grafana_cloud:
       endpoint: ${env:GRAFANA_ENDPOINT}
       auth:
         authenticator: basicauth/grafana_cloud
     debug:
       verbosity: detailed
      
   service:
      
     pipelines:
      
       traces:
         receivers: [otlp, jaeger, zipkin]
         processors: [batch]
         exporters: [debug,otlphttp/grafana_cloud]
      
       metrics:
         receivers: [otlp, prometheus]
         processors: [batch]
         exporters: [debug,otlphttp/grafana_cloud]
      
       logs:
         receivers: [otlp]
         processors: [batch]
         exporters: [debug,otlphttp/grafana_cloud]
      
     extensions: [health_check, pprof, zpages, basicauth/grafana_cloud]
   

   Copied!

   This configuration is the default OpenTelemetry Collector configuration, with the addition of the Grafana exporter.

7. In Grafana Cloud, in the Service name field, enter a name. In this example, we’ll use “kong-dev”.
8. In your terminal, export the following environment variables:

   export OTEL_SERVICE_NAME='kong-dev'
   export OTEL_RESOURCE_ATTRIBUTES='deployment.environment=production'
   export OTEL_EXPORTER_OTLP_ENDPOINT='http://localhost:4318'
   export OTEL_EXPORTER_OTLP_PROTOCOL='http/protobuf'
   

   Copied!

Now that we have the OpenTelemetry Collector configuration, we can deploy it. Use the opentelemetry-collector-contrib image and apply the configuration and environment variables that we defined in the previous step:

docker run \
    -p 4318:4318 \
    -e OTEL_SERVICE_NAME \
    -e OTEL_RESOURCE_ATTRIBUTES \
    -e OTEL_EXPORTER_OTLP_ENDPOINT \
    -e OTEL_EXPORTER_OTLP_PROTOCOL \
    -e GRAFANA_USERNAME \
    -e GRAFANA_PASSWORD \
    -e GRAFANA_ENDPOINT \
    -v $(pwd)/otel-config.yaml:/etc/otelcol-contrib/config.yaml \
    otel/opentelemetry-collector-contrib:0.141.0

You should already see some data in the terminal, however, this data will not be sent to Grafana since it doesn’t use the kong-dev service name.

In a new terminal, export the OTEL Collector host:

export DECK_OTEL_HOST=host.docker.internal

Let’s configure the OpenTelemetry plugin to send Kong Gateway metrics, traces, and logs to Grafana using the OpenTelemetry Collector.

Enable the OTEL plugin with the OTEL Collector endpoints settings configured:

echo '
_format_version: "3.0"
plugins:
  - name: opentelemetry
    config:
      traces_endpoint: http://${{ env "DECK_OTEL_HOST" }}:4318/v1/traces
      access_logs_endpoint: http://${{ env "DECK_OTEL_HOST" }}:4318/v1/logs
      logs_endpoint: http://${{ env "DECK_OTEL_HOST" }}:4318/v1/logs
      metrics:
        endpoint: http://${{ env "DECK_OTEL_HOST" }}:4318/v1/metrics
      resource_attributes:
        service.name: kong-dev
' | deck gateway apply -

The config.resource_attributes.service.name value should be the same as the service name defined in Grafana Cloud.

Send a POST request to generate traffic that we can use to validate that OpenTelemetry Collector is receiving the telemetry data:

curl -X POST "$KONNECT_PROXY_URL/anything" \
     --no-progress-meter --fail-with-body  \
     -H "Accept: application/json"\
     -H "Content-Type: application/json"

curl -X POST "http://localhost:8000/anything" \
     --no-progress-meter --fail-with-body  \
     -H "Accept: application/json"\
     -H "Content-Type: application/json"

You should see data in your OpenTelemetry Collector terminal.

Now, go back to the OTEL collector configuration in Grafana Cloud and click Test connection. You should see the following message:

Traces are being ingested properly

1. In the sidebar, click Drilldown.
2. Click the signal you want to see:
   • Metrics
   • Logs
   • Traces

If you don’t see your data, check that:

1. You’re viewing the correct data source. In the Data source dropdown list, select:
   • grafanacloud-<stack-name>-prom for metrics
   • grafanacloud-<stack-name>-logs for logs
   • grafanacloud-<stack-name>-traces for traces
2. You’re viewing the correct time range.

If everything is working as expected, you should see graphs with your data. You can now start building dashboards.