How to send metrics and logs to Datadog in a Kubernetes environment

How to send metrics and logs to Datadog in a Kubernetes environment?

The procedure is described in the Datadog documentation.

1. Create an account in Datadog in a Datadog site, notice that each Datadog site is independent.

   export DD_SITE=datadoghq.eu

   Copied!

2. Create an API key on your Datadog account.

   export DD_API_KEY=123456789123456789123456789

   Copied!

3. Install the Datadog Agent via Helm chart.

   helm install datadog  \
   --set datadog.site=$DD_SITE \
   --set datadog.apiKey=$DD_API_KEY \
   --set datadog.logs.enabled=true \
   --set datadog.logs.containerCollectAll=true \
   --set datadog.kubelet.tlsVerify=false \
   datadog/datadog

   Copied!

   (datadog.kubelet.tlsVerfiy = false is required when running tests in minikube.)

4. Datadog uses OpenMetrics to scrape metrics, we need to run the prometheus plugin in Kong.

   apiVersion: configuration.konghq.com/v1
   kind: KongClusterPlugin
   metadata:
     name: plugin-prometheus
     annotations:
       kubernetes.io/ingress.class: kong
     labels:
       global: "true"
   plugin: prometheus

   Copied!

   kubectl apply -f prometheus-plugin.yaml

   Copied!

5. Add required podAnnotations in the Kong Helm chart to allow the Datadog agent to use Autodiscovery to collect metrics and logs from Kong pods:

   podAnnotations:
     ad.datadoghq.com/proxy.check_names: '["kong"]'
     ad.datadoghq.com/proxy.init_configs: '[{}]'
     ad.datadoghq.com/proxy.instances: '[{"openmetrics_endpoint": "http://%%host%%:8100/metrics"}]'
     ad.datadoghq.com/proxy.logs: '[{"source": "kong", "service": "kong-proxy"}]'
     ad.datadoghq.com/ingress-controller.logs: '[{"source": "kong", "service": "kong-ingress-controller"}]'

   Copied!

6. Make sure that the status endpoint is enabled on your Helm chart.