Verify a Datakit call node's TLS connection using a custom CA

Deployment Platform
Minimum Version
Kong Gateway - 3.16
TL;DR

To make a Datakit call node trust an internal service signed by your own private CA, add the CA to Kong Gateway as a CA Certificate object, then set the Datakit plugin’s ca_certificates field to that object’s UUID. Any call node in the plugin instance verifies its outbound TLS connections against that CA instead of Kong Gateway’s global trusted CA store, so you can leave ssl_verify at its secure default of true.

Prerequisites

This is a Konnect tutorial and requires a Konnect personal access token.

  1. Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.

  2. Export your token to an environment variable:

    export KONNECT_TOKEN='YOUR_KONNECT_PAT'
  3. Run the quickstart script to automatically provision a Control Plane and Data Plane, and configure your environment:

    curl -Ls https://get.konghq.com/quickstart | bash -s -- -k $KONNECT_TOKEN \
         --deck-output

    This sets up a Konnect Control Plane named quickstart, provisions a local Data Plane, and prints out the following environment variable exports:

    export DECK_KONNECT_TOKEN=$KONNECT_TOKEN
    export DECK_KONNECT_CONTROL_PLANE_NAME=quickstart
    export DECK_KONNECT_ADDR=https://us.api.konghq.com
    export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
    export KONNECT_PROXY_URL='http://localhost:8000'

    Copy and paste these into your terminal to configure your session.

This tutorial requires Kong Gateway Enterprise. If you don’t have Kong Gateway set up yet, you can use the quickstart script with an enterprise license to get an instance of Kong Gateway running almost instantly.

  1. Export your license to an environment variable:

     export KONG_LICENSE_DATA='LICENSE-CONTENTS-GO-HERE'
  2. Run the quickstart script:

    curl -Ls https://get.konghq.com/quickstart | bash -s -- -e KONG_LICENSE_DATA 

    Once Kong Gateway is ready, you will see the following message:

     Kong Gateway Ready

To complete this tutorial, install decK. We recommend keeping decK up to date with the latest version (1.66.1).

decK is a CLI tool for managing Kong Gateway declaratively with state files. This guide uses deck gateway apply, which directly applies entity configuration to your Gateway instance.

You can check your current decK version with deck version.

For this tutorial, you’ll need Kong Gateway entities, like Gateway Services and Routes, pre-configured. These entities are essential for Kong Gateway to function but installing them isn’t the focus of this guide. Follow these steps to pre-configure them:

  1. Run the following command:

    echo '
    _format_version: "3.0"
    services:
      - name: example-service
        url: http://httpbin.konghq.com/anything
    routes:
      - name: example-route
        paths:
        - "/anything"
        service:
          name: example-service
        protocols:
        - http
        - https
    ' | deck gateway apply -

To learn more about entities, you can read our entities documentation.

To use the copy, paste, and run the instructions in this how-to, you need the ID of your control plane.

Look it up using DECK_KONNECT_CONTROL_PLANE_NAME, exported in a previous prerequisite, or substitute the name of your own control plane:

CONTROL_PLANE_ID=$(curl -X GET "https://us.api.konghq.com/v2/control-planes?filter%5Bname%5D%5Beq%5D=$DECK_KONNECT_CONTROL_PLANE_NAME" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" | jq -r ".data[0].id"
)

The Datakit plugin’s call node makes outbound HTTPS requests as part of the plugin’s workflow, independent of any Service’s TLS configuration. By default, it verifies the server’s certificate against Kong Gateway’s global trusted CA store (lua_ssl_trusted_certificate).

If a call node needs to reach an internal service signed by a private certificate authority, that global store often isn’t an option:

  • On Konnect Dedicated Cloud Gateways, the global store is managed by Konnect, so you can’t add your own private CAs to it.
  • Editing global Kong Gateway configuration to add a private CA affects every plugin and connection on the node.

This guide shows how to configure the Datakit plugin’s ca_certificates field so a call node verifies its outbound TLS connection against your private CA.

Generate certificates

In this guide, you generate two certificates:

  • A CA certificate, used to sign the internal service’s certificate and to configure trust in the Datakit plugin
  • An internal service certificate, presented by the mock internal service during the TLS handshake

The internal service runs in its own Docker container, published to your host machine. Kong Gateway (which runs in its own container) reaches it via host.docker.internal, Docker’s built-in DNS name for the host machine, so there’s no custom Docker network to create or match.

  1. Create a working directory and change into it:

    mkdir -p ~/datakit-custom-ca && cd ~/datakit-custom-ca
  2. Generate a CA certificate:

    openssl req -new -x509 -nodes -days 365 \
      -subj '/CN=my-private-ca' \
      -keyout ca.key \
      -out ca.crt
  3. Generate a certificate for the internal service, signed by the CA. The subjectAltName must include host.docker.internal, since that’s the hostname the Datakit call node uses to reach your host machine from inside the Kong Gateway container:

    openssl genrsa -out internal-service.key 2048
    
    openssl req -new -key internal-service.key -out internal-service.csr \
      -subj "/CN=host.docker.internal"
    
    cat > internal-service.ext <<EOF
    authorityKeyIdentifier=keyid,issuer
    basicConstraints=CA:FALSE
    keyUsage = digitalSignature, keyEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = DNS:host.docker.internal
    EOF
    
    openssl x509 -req \
      -in internal-service.csr \
      -CA ca.crt -CAkey ca.key -CAcreateserial \
      -out internal-service.crt -days 365 -sha256 -extfile internal-service.ext

Start the internal service

For this guide, we’ll use Nginx to simulate an internal service that only serves HTTPS with the certificate you just generated. In production, you will point this to your real service.

  1. Create a directory for the service and copy the certificates into it:

    mkdir -p ~/datakit-custom-ca/internal-service
    cp ~/datakit-custom-ca/internal-service.crt ~/datakit-custom-ca/internal-service/
    cp ~/datakit-custom-ca/internal-service.key ~/datakit-custom-ca/internal-service/
  2. Create a configuration file named nginx.conf:

    cat <<'EOF' > ~/datakit-custom-ca/internal-service/nginx.conf
    worker_processes auto;
    events {
      worker_connections 1024;
    }
    
    http {
      default_type application/json;
    
      server {
        listen 443 ssl;
        server_name host.docker.internal;
    
        ssl_certificate     /etc/ssl/certs/internal-service.crt;
        ssl_certificate_key /etc/ssl/certs/internal-service.key;
    
        location /author {
          default_type application/json;
          return 200 '{"author":"Example Author"}';
        }
      }
    }
    EOF
  3. Create the Dockerfile:

    cat <<'EOF' > ~/datakit-custom-ca/internal-service/Dockerfile
    FROM nginx:latest
    COPY internal-service.crt /etc/ssl/certs/internal-service.crt
    COPY internal-service.key /etc/ssl/certs/internal-service.key
    COPY nginx.conf           /etc/nginx/nginx.conf
    EXPOSE 443
    CMD ["nginx", "-g", "daemon off;"]
    EOF
  4. Build and start the internal service, publishing its port to your host machine:

    cd ~/datakit-custom-ca/internal-service
    docker build -t internal-service .
    docker run -d --name internal-service -p 9443:443 internal-service
  5. Verify that the service is reachable and presents the expected certificate:

    curl -s --cacert ~/datakit-custom-ca/ca.crt \
      --resolve host.docker.internal:9443:127.0.0.1 \
      https://host.docker.internal:9443/author

    You should receive:

    {"author":"Example Author"}

Add the CA certificate to Kong Gateway

The Datakit plugin uses a Kong Gateway CA Certificate entity to verify the internal service’s TLS certificate.

Navigate back to the working directory, then build the request body from the certificate file:

cd ~/datakit-custom-ca
jq -n --rawfile cert ca.crt '{"cert": $cert}' > ca-cert-body.json

Add the CA certificate and export its ID:

export DECK_CA_CERT_ID=$(curl -s -X POST http://localhost:8001/ca_certificates \
    --json @ca-cert-body.json | jq -r .id)
echo "CA Cert ID: $DECK_CA_CERT_ID"
curl -X POST "https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/core-entities/ca_certificates" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json "$(cat ca-cert-body.json)"

Save the CA cert ID from the response as an environment variable:

export DECK_CA_CERT_ID="ID_OF_CA_CERT"

Configure the Datakit plugin

Using the example-service and example-route from the prerequisites, configure the Datakit plugin with a call node that reaches the internal service over HTTPS, and reference the CA Certificate object in ca_certificates:

echo '
_format_version: "3.0"
plugins:
  - name: datakit
    route: example-route
    config:
      ca_certificates:
      - "${{ env "DECK_CA_CERT_ID" }}"
      nodes:
      - name: AUTHOR
        type: call
        url: https://host.docker.internal:9443/author
        ssl_verify: true
      - name: EXIT
        type: exit
        inputs:
          body: AUTHOR.body
        status: 200
' | deck gateway apply -

In this configuration:

  • ca_certificates: A list of CA Certificate entity UUIDs. This is set at the top level of the plugin’s config, so every call node in this plugin instance shares the same trust store.
  • AUTHOR: A call node with ssl_verify: true (the default) that reaches the internal service. Its TLS certificate is verified against the CA Certificate object referenced in ca_certificates, instead of Kong Gateway’s global trusted CA store.
  • EXIT: Returns the AUTHOR node’s response body directly to the client.

Validate the flow

Send a request through Kong Gateway:

curl -i "$KONNECT_PROXY_URL/anything" \
     --no-progress-meter --fail-with-body 
curl -i "http://localhost:8000/anything" \
     --no-progress-meter --fail-with-body 

You should get an HTTP 200 response with the internal service’s response body:

{"author":"Example Author"}

This confirms that the AUTHOR node’s TLS handshake succeeded using only the private CA referenced in ca_certificates. Kong Gateway’s global trusted CA store was never consulted for this request.

Confirm the CA is being enforced

To see what happens when the referenced CA doesn’t match the internal service’s certificate, update the plugin to point at a CA that didn’t sign it.

Change into the internal-service directory and generate an unrelated CA certificate:

cd internal-service
openssl req -new -x509 -nodes -days 365 \
  -subj '/CN=unrelated-ca' \
  -keyout ~/datakit-custom-ca/unrelated-ca.key \
  -out ~/datakit-custom-ca/unrelated-ca.crt

Navigate back to the datakit-custom-ca directory, then build the request body from the certificate file:

cd ~/datakit-custom-ca
jq -n --rawfile cert unrelated-ca.crt '{"cert": $cert}' > unrelated-ca-cert-body.json

Add it to Kong Gateway and export its ID:

export DECK_WRONG_CA_CERT_ID=$(curl -s -X POST http://localhost:8001/ca_certificates \
    --json @unrelated-ca-cert-body.json | jq -r .id)
echo "Wrong CA Cert ID: $DECK_WRONG_CA_CERT_ID"
curl -X POST "https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/core-entities/ca_certificates" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json "$(cat unrelated-ca-cert-body.json)"

Save the CA cert ID from the response as an environment variable:

export DECK_WRONG_CA_CERT_ID="ID_OF_CA_CERT"

Update the Datakit plugin’s ca_certificates to reference the unrelated CA instead:

echo '
_format_version: "3.0"
plugins:
  - name: datakit
    route: example-route
    config:
      ca_certificates:
      - "${{ env "DECK_WRONG_CA_CERT_ID" }}"
      nodes:
      - name: AUTHOR
        type: call
        url: https://host.docker.internal:9443/author
        ssl_verify: true
      - name: EXIT
        type: exit
        inputs:
          body: AUTHOR.body
        status: 200
' | deck gateway apply -

Send the same request again:

curl -i "$KONNECT_PROXY_URL/anything" \
     --no-progress-meter --fail-with-body 
curl -i "http://localhost:8000/anything" \
     --no-progress-meter --fail-with-body 

You should get an HTTP 500 response, because the AUTHOR node’s TLS certificate no longer chains to a CA in ca_certificates. This confirms that Kong Gateway is enforcing the configured trust store rather than silently falling back to the global one.

You may need to wait a few seconds before Kong Gateway responds with a 500, as the responses are cached.

Cleanup

Stop and remove the internal service container, then delete the working directory:

docker stop internal-service && docker rm internal-service
rm -rf ~/datakit-custom-ca

If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.

curl -Ls https://get.konghq.com/quickstart | bash -s -- -d

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!