Configuring Mutual TLS

Uses: Kong Mesh

Usage of “builtin” CA

This is the fastest and simplest way to enable mTLS in Kong Mesh.

With a builtin CA backend type, Kong Mesh will dynamically generate its own CA root certificate and key that it uses to automatically provision (and rotate) certificates for every replica of every service.

We can specify more than one builtin backend with different names, and each one of them will be automatically provisioned with a unique pair of certificate + key (they are not shared).

To enable a builtin mTLS for the entire Mesh we can apply the following configuration:

Starting with Kong Mesh version 2.6.0, we no longer create a default TrafficPermission policy that’s essential for traffic to function after enabling mTLS. To prevent disruption of your traffic, it’s highly recommended to add a specific or default MeshTrafficPermission policy before enabling mTLS. This policy will allow communication between your applications.

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
      - name: ca-1
        type: builtin
        dpCert:
          rotation:
            expiration: 1d
        conf:
          caCert:
            RSAbits: 2048
            expiration: 10y

We will apply the configuration with kubectl apply -f [..].

type: Mesh
name: default
mtls:
  enabledBackend: ca-1
  backends:
    - name: ca-1
      type: builtin
      dpCert:
        rotation:
          expiration: 1d
      conf:
        caCert:
          RSAbits: 2048
          expiration: 10y

We will apply the configuration with kumactl apply -f [..] or via the HTTP API.

A few considerations:

The dpCert configuration determines how often Kong Mesh should automatically rotate the certificates assigned to every data plane proxy.
The caCert configuration determines a few properties that Kong Mesh will use when auto-generating the CA root certificate.

Storage of Secrets

When using a builtin backend Kong Mesh automatically generates a root CA certificate and key that are being stored as a Kong Mesh Secret resource with the following name:

{mesh name}.ca-builtin-cert-{backend name} for the certificate
{mesh name}.ca-builtin-key-{backend name} for the key

On Kubernetes, Kong Mesh secrets are being stored in the kong-mesh-system namespace, while on Universal they are being stored in the underlying store configured in kuma-cp.

We can retrieve the secrets via kumactl on both Universal and Kubernetes, or via kubectl on Kubernetes only:

The following command can be executed on any Kong Mesh backend:

kumactl get secrets [-m MESH]
# MESH      NAME                           AGE
# default   default.ca-builtin-cert-ca-1   1m
# default   default.ca-builtin-key-ca-1    1m

The following command can be executed only on Kubernetes:

kubectl get secrets \
    -n kong-mesh-system \
    --field-selector='type=system.kuma.io/secret'
# NAME                             TYPE                                  DATA   AGE
# default.ca-builtin-cert-ca-1     system.kuma.io/secret                 1      1m
# default.ca-builtin-key-ca-1      system.kuma.io/secret                 1      1m

Usage of “provided” CA

If you choose to provide your own CA root certificate and key, you can use the provided backend. With this option, you must also manage the certificate lifecycle yourself.

Unlike the builtin backend, with provided you first upload the certificate and key as Secret resource, and then reference the Secrets in the mTLS configuration.

Kong Mesh then provisions data plane proxy certificates for every replica of every service from the CA root certificate and key.

Sample configuration:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
      - name: ca-1
        type: provided
        dpCert:
          rotation:
            expiration: 1d
        conf:
          cert:
            secret: name-of-secret
          key:
            secret: name-of-secret

We will apply the configuration with kubectl apply -f [..].

type: Mesh
name: default
mtls:
  enabledBackend: ca-1
  backends:
    - name: ca-1
      type: provided
      dpCert:
        rotation:
          expiration: 1d
      conf:
        cert:
          secret: name-of-secret
        key:
          secret: name-of-secret

We will apply the configuration with kumactl apply -f [..] or via the HTTP API.

A few considerations:

The dpCert configuration determines how often Kong Mesh should automatically rotate the certificates assigned to every data plane proxy.
The Secrets must exist before referencing them in a provided backend.

Intermediate CA

You can also work with an Intermediate CA with a provided backend. Generate the certificate and place it first in certificate file, before the certificate from the root CA. The certificate from the root CA should start on a new line. Then create the secret to specify in the cert section of the config. The secret for the key should contain only the private key of the certificate from the intermediate CA.

You can chain certificates from multiple intermediate CAs the same way. Place the certificate from the closest CA at the top of the cert file, followed by certificates in order up the certificate chain, then generate the secret to hold the contents of the file.

Sample certificate file for a single intermediate CA:

CA requirements

When using an arbitrary certificate and key for a provided backend, we must make sure that we comply with the following requirements:

It MUST have basic constraint CA set to true (see X509-SVID: 4.1. Basic Constraints)
It MUST have key usage extension keyCertSign set (see X509-SVID: 4.3. Key Usage)
It MUST NOT have key usage extension ‘keyAgreement’ set (see X509-SVID: Appendix A. X.509 Field Reference)
It SHOULD NOT set key usage extension ‘digitalSignature’ and ‘keyEncipherment’ to be SPIFFE compliant (see X509-SVID: Appendix A. X.509 Field Reference)

Do not use the following example in production, instead generate valid and compliant certificates. This example is intended for usage in a development environment.

Below we can find an example to generate a sample CA certificate + key:

The following command will generate a CA root certificate and key that can be uploaded to Kong Mesh as a Secret and then used in a provided mTLS backend:

SAMPLE_CA_CONFIG="
[req]
distinguished_name=dn
[ dn ]
[ ext ]
basicConstraints=CA:TRUE,pathlen:0
keyUsage=keyCertSign
"

openssl req -config <(echo "$SAMPLE_CA_CONFIG") -new -newkey rsa:2048 -nodes \
  -subj "/CN=Hello" -x509 -extensions ext -keyout key.pem -out crt.pem

The command will generate a certificate at crt.pem and the key at key.pem. We can generate the Kong Mesh Secret resources by following the Secret resource.

Development Mode

In development mode we may want to provide the cert and key properties of the provided backend without necessarily having to create a Secret resource, but by using an inline value.

Using the inline modes in production presents a security risk since it makes the values of our CA root certificate and key more easily accessible from a malicious actor. We highly recommend using inline only in development mode.

Kong Mesh offers an alternative way to specify the CA root certificate and key:

Please note the inline properties that are being used instead of secret:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
      - name: ca-1
        type: provided
        conf:
          cert:
            inline: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHekNDQWdPZ0F3S... # cert in Base64
          key:
            inline: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHekNDQWdPZ0F3S... # key in Base64

Please note the inline properties that are being used instead of secret:

type: Mesh
name: default
mtls:
  enabledBackend: ca-1
  backends:
    - name: ca-1
      type: provided
      conf:
        cert:
          inline: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHekNDQWdPZ0F3S... # cert in Base64
        key:
          inline: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHekNDQWdPZ0F3S... # cert in Base64

Permissive mTLS

In version 1.4.1 and later, Kong Mesh provides PERMISSIVE mTLS mode to let you migrate existing workloads with zero downtime.

Permissive mTLS mode encrypts outbound connections the same way as strict mTLS mode, but inbound connections on the server-side accept both TLS and plaintext. This lets you migrate servers to an mTLS mesh before their clients. It also supports the case where the client and server already implement TLS.

PERMISSIVE mode is not secure. It’s intended as a temporary utility. Make sure to set to STRICT mode after migration is complete.

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
      - name: ca-1
        type: builtin
        mode: PERMISSIVE # supported values: STRICT, PERMISSIVE

type: Mesh
name: default
mtls:
  enabledBackend: ca-1
  backends:
    - name: ca-1
      type: builtin
      mode: PERMISSIVE # supported values: STRICT, PERMISSIVE

Certificate Rotation

Once a CA backend has been configured, Kong Mesh will utilize the CA root certificate and key to automatically provision a certificate for every data plane proxy that it connects to kuma-cp.

Unlike the CA certificate, the data plane proxy certificates are not permanently stored anywhere but they only reside in memory. These certificates are designed to be short-lived and rotated often by Kong Mesh.

By default, the expiration time of a data plane proxy certificate is 30 days. Kong Mesh rotates these certificates automatically after 4/5 of the certificate validity time (ie: for the default 30 days expiration, that would be every 24 days).

You can update the duration of the data plane proxy certificates by updating the dpCert property on every available mTLS backend.

You can inspect the certificate rotation statistics by executing the following command (supported on both Kubernetes and Universal):

We can use the Kong Mesh CLI:

kumactl inspect dataplanes
# MESH      NAME     TAGS          STATUS   LAST CONNECTED AGO   LAST UPDATED AGO   TOTAL UPDATES   TOTAL ERRORS   CERT REGENERATED AGO   CERT EXPIRATION       CERT REGENERATIONS
# default   web-01   service=web   Online   5s                   3s                 4               0              3s                     2020-05-11 16:01:34   2

Please note the CERT REGENERATED AGO, CERT EXPIRATION, CERT REGENERATIONS columns.

We can use the Kong Mesh HTTP API by retrieving the Dataplane Insight resource and inspecting the dataplaneInsight object.

...
dataplaneInsight": {
  ...
  "mTLS": {
    "certificateExpirationTime": "2020-05-14T20:15:23Z",
    "lastCertificateRegeneration": "2020-05-13T20:15:23.994549539Z",
    "certificateRegenerations": 1
  }
}
...

A new data plane proxy certificate is automatically generated when:

A data plane proxy is restarted.
The control plane is restarted.
The data plane proxy connects to a new control plane.