Set up observability for Kong Event Gateway

Run the quickstart script to automatically provision a demo Event Gateway control plane and data plane, and configure your environment for sending metrics and traces:

curl -Ls https://get.konghq.com/event-gateway | bash -s -- \
  -k $KONNECT_TOKEN \
  -N kafka_event_gateway \
  -e "KEG__OBSERVABILITY__OTLP__TRACING__ENABLED=true" \
  -e "KEG__RUNTIME__HEALTH_LISTENER_ADDRESS_PORT=0.0.0.0:8080" \
  -e "OTEL_EXPORTER_OTLP_PROTOCOL=grpc" \
  -e "OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317" \
  -e "OTEL_EXPORTER_OTLP_TIMEOUT=10s" \
  -e "OTEL_SERVICE_NAME=eventgw"

Where you configure the following custom telemetry settings:

This sets up an Event Gateway control plane named event-gateway-quickstart, provisions a local data plane, and prints out the following environment variable export:

export EVENT_GATEWAY_ID=your-gateway-id

Copy and paste this into your terminal to configure your session.

This quickstart script is meant for demo purposes only, therefore it runs locally with most default parameters and a small number of exposed ports. If you want to run Kong Gateway as a part of a production-ready platform, set up your control plane and data planes through the Konnect UI, or using Terraform.

Next, we need to create two configuration files: one for the OTEL collector, and one for Prometheus.

Open a new terminal window and create a directory for the configuration:

mkdir otel && cd otel

Create an OTEL collector configuration file:

cat <<EOF > otel-collector-config.yaml
receivers:
  otlp/keg:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
  prometheus/keg:
    config:
      scrape_configs:
        - job_name: kong
          scrape_interval: 5s
          static_configs:
            - targets: ['event-gateway-quickstart:8080']

processors:
  batch: {}

extensions:
  health_check: {}

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  otlphttp/prometheus:
    endpoint: http://prometheus:9090/api/v1/otlp
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp/keg]
      processors: [batch]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [prometheus/keg]
      processors: [batch]
      exporters: [otlphttp/prometheus]
EOF

Create a Prometheus configuration file:

cat <<EOF > prometheus.yaml
global:
  scrape_interval: 5s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
EOF

Now, let’s run the OTEL collector stack with Jaeger and Prometheus.

Run the following command to save a Docker compose file, making sure that it communicates on the same network as your Kafka cluster and your Event Gateway data plane:

cat <<EOF > docker-compose.yaml

name: otel-stack
networks:
  kafka:
    name: kafka_event_gateway
    external: true

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    networks:
      - kafka
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"  # OTLP gRPC receiver
      - "18639:9090" # Use this for testing the scraping endpoint
  
  jaeger:
    image: jaegertracing/all-in-one:latest
    networks:
      - kafka
    ports:
      - "6831:6831/udp" # UDP port for Jaeger agent
      - "16686:16686"   # Web UI
      - "14268:14268"   # HTTP port for spans

  prometheus:
    image: prom/prometheus:latest
    networks:
      - kafka
    command:
      - '--config.file=/etc/prometheus/prometheus.yaml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.enable-otlp-receiver'   # Enables Prometheus to receive metrics using OTLP
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yaml
    ports:
      - "9090:9090"
EOF

Launch the OTEL stack:

docker compose up -d

Use the following Kafka configuration to access your Kafka resources from the virtual clusters:

cat <<EOF > kafkactl.yaml
contexts:
  direct:
    brokers:
      - localhost:9095
      - localhost:9096
      - localhost:9094
  vc:
    brokers:
      - localhost:19092
EOF

Use the following command to create a backend cluster that connects to the Kafka servers you set up:

BACKEND_CLUSTER_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/backend-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "backend_cluster",
       "bootstrap_servers": [
         "kafka1:9092",
         "kafka2:9092",
         "kafka3:9092"
       ],
       "authentication": {
         "type": "anonymous"
       },
       "tls": {
         "enabled": false
       },
       "insecure_allow_anonymous_virtual_cluster_auth": true
     }' | jq -r ".id"
)

In this example configuration:

• bootstrap_servers: Points the backend cluster to the three bootstrap servers that we launched in the prerequisites.
• authentication and insecure_allow_anonymous_virtual_cluster_auth: For demo purposes, we’re allowing insecure anonymous connections, which means no authentication required.
• tls: TLS is disabled so that we can easily test the connection.

Run the following command to create a new virtual cluster associated with our backend cluster. This will let you route event traffic and apply policies:

VIRTUAL_CLUSTER_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example_virtual_cluster",
       "destination": {
         "id": "'$BACKEND_CLUSTER_ID'"
       },
       "dns_label": "vcluster-1",
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "acl_mode": "passthrough"
     }' | jq -r ".id"
)

In this example:

• authentication: Allows anonymous authentication.
• acl_mode: The setting passthrough means that all clients are allowed and don’t have to match a defined ACL. In a production environment, you would set this to enforce_on_gateway and define an ACL policy.
• name is an internal name for the configuration object, while the dns_label is necessary for SNI routing.

For testing purposes, we’ll use port forwarding to route traffic to the virtual cluster.
In production environments, you should use SNI routing instead.

Run the following command to create a new listener:

LISTENER_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/listeners" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example_listener",
       "addresses": [
         "0.0.0.0"
       ],
       "ports": [
         "19092-19095"
       ]
     }' | jq -r ".id"
)

Create the port mapping policy:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/listeners/$LISTENER_ID/policies" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "type": "forward_to_virtual_cluster",
       "name": "forward",
       "config": {
         "type": "port_mapping",
         "advertised_host": "localhost",
         "destination": {
           "id": "'$VIRTUAL_CLUSTER_ID'"
         }
       }
     }'

Create a topic using the direct context, which is a direct connection to our Kafka cluster:

kafkactl -C kafkactl.yaml --context direct create topic my-test-topic

Then produce a message through the virtual cluster:

kafkactl -C kafkactl.yaml --context vc produce my-test-topic --value="test message"

You should see the following responses:

topic created: my-test-topic
message produced (partition=0	offset=0)

Now that we’ve configured the OTEL collector to scrape the metrics endpoint and send them to Prometheus, let’s look at what data is available in Prometheus. If we go to the query tab, we will see the metrics populated:

1. In your browser, open the Prometheus dashboard at http://localhost:9090/.
2. Search for kong to see the list of available metrics.
3. Let’s look at a sample metric: kong_keg_kafka_backend_roundtrip_duration_seconds_sum.

This tells you how long it took for the Event Gateway to send a request to the backend cluster and receive a response.

Let’s go to Jaeger to see we can see the full trace generated by Event Gateway.

1. Send any command through the virtual cluster, such as list topics:

   kafkactl -C kafkactl.yaml --context vc list topics
   

   Copied!
2. In your browser, open the Jaeger search dashboard at http://localhost:16686/search.
3. Select the eventgw service.
4. Click Find Traces.

Here you can see the full trace generated by Event Gateway for each command. For example, you can click an eventgw: request trace to see the details, including information about the virtual cluster and backend cluster.