Collect metrics with Datadog and the Prometheus plugin

Uses: Kong Gateway decK
TL;DR

Install the Datadog Agent and enable the Prometheus plugin. Configure the Datadog Agent with the Kong Gateway /metrics endpoint and set kong.* for metrics. Restart the Datadog Agent, and send requests to generate metrics. You should see the metrics in Datadog Metrics summary.

Prerequisites

This is a Konnect tutorial and requires a Konnect personal access token.

  1. Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.

  2. Export your token to an environment variable:

     export KONNECT_TOKEN='YOUR_KONNECT_PAT'
    
  3. Run the quickstart script to automatically provision a Control Plane and Data Plane, and configure your environment:

     curl -Ls https://get.konghq.com/quickstart | bash -s -- -k $KONNECT_TOKEN -e KONG_STATUS_LISTEN=0.0.0.0:8100 -p 8100:8100 --deck-output
    

    This sets up a Konnect Control Plane named quickstart, provisions a local Data Plane, and prints out the following environment variable exports:

     export DECK_KONNECT_TOKEN=$KONNECT_TOKEN
     export DECK_KONNECT_CONTROL_PLANE_NAME=quickstart
     export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
     export KONNECT_PROXY_URL='http://localhost:8000'
    

    Copy and paste these into your terminal to configure your session.

This tutorial requires Kong Gateway Enterprise. If you don’t have Kong Gateway set up yet, you can use the quickstart script with an enterprise license to get an instance of Kong Gateway running almost instantly.

  1. Export your license to an environment variable:

     export KONG_LICENSE_DATA='LICENSE-CONTENTS-GO-HERE'
    
  2. Run the quickstart script:

     curl -Ls https://get.konghq.com/quickstart | bash -s -- -e KONG_LICENSE_DATA 
    

    Once Kong Gateway is ready, you will see the following message:

     Kong Gateway Ready
    

decK is a CLI tool for managing Kong Gateway declaratively with state files. To complete this tutorial you will first need to install decK.

For this tutorial, you’ll need Kong Gateway entities, like Gateway Services and Routes, pre-configured. These entities are essential for Kong Gateway to function but installing them isn’t the focus of this guide. Follow these steps to pre-configure them:

  1. Run the following command:

    echo '
    _format_version: "3.0"
    services:
      - name: example-service
        url: http://httpbin.konghq.com/anything
    routes:
      - name: example-route
        paths:
        - "/anything"
        service:
          name: example-service
    ' | deck gateway apply -
    

To learn more about entities, you can read our entities documentation.

For this tutorial, you’ll need to configure the following:

  • A Datadog account
  • Install the Datadog Agent by navigating to the Agent Installation page or Integration > Install agents in the Datadog UI.
  • Your Datadog API key and app key. You can find these in the Datadog UI in Organization settings.

Set the following as environment variables:

export DD_API_KEY='YOUR-API-KEY'
export DD_APP_KEY='YOUR-APPLICATION-KEY'
export DD_SITE_API_URL='YOUR-API-SITE-URL'
  • Some distributions require you to modify the datadog.yaml file and add your API key and Datadog site URL. Ensure this file is configured correctly or Datadog won’t be able to scrape metrics.
  • Your Datadog site API URL varies depending on the region you’re using. For example, for the US5 region, the URL would be https://api.us5.datadoghq.com.

Enable the Prometheus plugin

Before you configure the Datadog Agent to scrape metrics from Kong Gateway, you first need to enable the Prometheus plugin.

The following configuration enables the plugin globally and exports status code metrics, like the total number of HTTP requests:

echo '
_format_version: "3.0"
plugins:
  - name: prometheus
    config:
      status_code_metrics: true
' | deck gateway apply -

Configure the Datadog Agent to collect Kong Gateway metrics

Now that the Prometheus plugin is configured, you can configure the Datadog Agent to scrape Kong Gateway metrics.

Create the conf.yaml file:

touch ./.datadog-agent/conf.d/openmetrics.d/conf.yaml

This command uses the macOS directory location. For other distributions, see Datadog’s Agent configuration directory.

Copy and paste the following configuration in the conf.yaml file:

instances:
  - openmetrics_endpoint: http://localhost:8001/metrics
    namespace: kong
    metrics:
      - kong.*
instances:
  - openmetrics_endpoint: http://localhost:8100/metrics
    namespace: kong
    metrics:
      - kong.*

This configuration pulls all the kong. prefixed metrics from the Kong Gateway metrics endpoint (http://localhost:8001/metrics).

This configuration pulls all the kong. prefixed metrics from the Kong Gateway Status API metrics endpoint (http://localhost:8100/metrics).

Important: If you’re running Kong Gateway and the Datadog Agent in Docker, you’ll need to replace localhost in the config.yaml with the name of the Kong Gateway container. Also, both containers need to be running on the same network for them to communicate.

Restart the Datadog Agent

You must restart the agent to start collecting metrics:

launchctl stop com.datadoghq.agent
launchctl start com.datadoghq.agent

This example is for macOS, see Datadog’s Agent commands documentation for all restart commands.

The Datadog Agent may take a few minutes to restart.

Validate

Now, you can validate that Datadog can scrape Kong Gateway metrics by first sending requests to generate metrics:

curl "$KONNECT_PROXY_URL/anything"
curl "$KONNECT_PROXY_URL/anything"
curl "$KONNECT_PROXY_URL/anything"
curl "$KONNECT_PROXY_URL/anything"
curl "$KONNECT_PROXY_URL/anything"
curl "$KONNECT_PROXY_URL/anything"
curl "$KONNECT_PROXY_URL/anything"
curl "$KONNECT_PROXY_URL/anything"
curl "$KONNECT_PROXY_URL/anything"
curl "$KONNECT_PROXY_URL/anything"
curl "http://localhost:8000/anything"
curl "http://localhost:8000/anything"
curl "http://localhost:8000/anything"
curl "http://localhost:8000/anything"
curl "http://localhost:8000/anything"
curl "http://localhost:8000/anything"
curl "http://localhost:8000/anything"
curl "http://localhost:8000/anything"
curl "http://localhost:8000/anything"
curl "http://localhost:8000/anything"

You can validate that Kong Gateway is sending metrics to Datadog by running the following:

curl -X GET "$DD_SITE_API_URL/api/v1/search?q=kong.kong_http_requests.count" \
-H "Accept: application/json" \
-H "DD-API-KEY: $DD_API_KEY" \
-H "DD-APPLICATION-KEY: $DD_APP_KEY"

You should get the following response:

{"results":{"metrics":["kong.kong_http_requests.count"],"hosts":[]}}

Alternatively, you can navigate to the Metrics Explorer page in the Datadog UI and search for kong.kong_http_requests.count. You should see the 10 requests that you just sent.

Cleanup

To stop collecting metrics, you can uninstall the Datadog Agent.

curl -Ls https://get.konghq.com/quickstart | bash -s -- -d

If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.

Something wrong?

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!