Zone proxy authentication

To obtain a configuration from the control plane, a zone proxy (zone ingress or zone egress) must authenticate itself. You can do this using:

• A service account token on Kubernetes
• A zone token on Universal

On Kubernetes, a zone proxy authenticates by leveraging the ServiceAccountToken that is mounted in every Pod.

On Universal, a zone proxy must be explicitly configured with a unique security token with the appropriate scope (egress or ingress).

The zone token used to identify zone proxies is a JWT token that contains:

• The zone in which zone proxy operates
• The expiration date of the token (10 years by default)
• The scope as a list of proxies for which the token is valid (egress and ingress by default)

The zone token is signed using a key that the control plane autogenerates on first startup. Tokens themselves are never stored in the control plane. Only the signing keys are stored, and these are used to verify token validity. The signing algorithm is RS256.

You can use the following command to check for the signing key:

kumactl get global-secrets

You should get the following result:

NAME                       AGE
zone-token-signing-key-1   7s

Generate a zone token

kumactl generate zone-token \
  --zone us-east \
  --scope egress \
  --valid-for 720h > /tmp/kuma-zone-proxy-token

curl -XPOST \
  -H "Content-Type: application/json" \
  --data '{"zone": "us-east", "validFor": "720h", "scope": ["egress", "ingress"]}' \
  http://localhost:5681/tokens/zone > /tmp/kuma-zone-proxy-token

The token should be stored in a file and then passed when you start kuma-dp:

kuma-dp run \
  --proxy-type=ingress # or egress \
  --dataplane-file=zone-proxy-definition.yaml
  --cp-address=https://127.0.0.1:5678 \
  --dataplane-token-file=/tmp/kuma-zone-proxy-token

You can also pass the token as a KUMA_DATAPLANE_RUNTIME_TOKEN environment variable.

Kong Mesh doesn’t keep a list of issued tokens. Whenever a single token is compromised, you can add it to the revocation list to invalidate it.

Every token has its own ID under jti key. You can extract the ID from the token using jwt.io or the jwt-cli tool.

To revoke tokens, specify a comma-separated list of revoked IDs in a global secret named zone-token-revocations.

REVOCATIONS=$(echo '0e120ec9-6b42-495d-9758-07b59fe86fb9' | base64) && echo "apiVersion: v1
kind: Secret
metadata:
  name: zone-token-revocations
  namespace: kong-mesh-system
data:
  value: $REVOCATIONS
type: system.kuma.io/global-secret" | kubectl apply -f -

echo "
type: GlobalSecret
name: zone-token-revocations
data: " | kumactl apply --var revocations=$(echo '0e120ec9-6b42-495d-9758-07b59fe86fb9' | base64) -f -

If the signing key is compromised, you must rotate it. You must also rotate all the tokens that were signed by it.

1. Generate a new signing key. The signing key is stored as a global secret named in the following format: zone-token-signing-key-<serial_number>.

   When generating a new signing key, assign it a serial number greater than the current key’s serial number.

   Kubernetes Universal

   Get the current key’s serial number:

   kubectl get secrets -n kong-mesh-system --field-selector='type=system.kuma.io/global-secret'
   NAME                       TYPE                           DATA   AGE
   zone-token-signing-key-1   system.kuma.io/global-secret   1      25m
   

   Copied!

   In this example, the highest serial number is 1.

   Generate a new signing key with a serial number of 2:

   TOKEN="$(kumactl generate signing-key)" && echo "
   apiVersion: v1
   data:
     value: $TOKEN
   kind: Secret
   metadata:
     name: zone-token-signing-key-2
     namespace: kong-mesh-system
   type: system.kuma.io/global-secret
   " | kubectl apply -f -
   

   Copied!

   Get the current key’s serial number:

   kumactl get global-secrets
   NAME                       AGE
   zone-token-signing-key-1   36m
   

   Copied!

   In this example, the highest serial number is 1.

   Generate a new signing key with a serial number of 2:

   echo "
   type: GlobalSecret
   name: zone-token-signing-key-2
   data: " | kumactl apply --var key=$(kumactl generate signing-key) -f -
   

   Copied!

2. Generate new tokens. New tokens are generated with the signing key with the highest serial number. At this point, tokens signed by either the new or old signing key are valid.

3. Remove the old signing key:

   Kubernetes Universal

   kubectl delete secret zone-token-signing-key-1 -n kong-mesh-system
   

   Copied!

   kumactl delete global-secret zone-token-signing-key-1
   

   Copied!

   All new connections to the control plane now require tokens signed with the new signing key.

In addition to the regular flow of generating signing keys, storing them in a secret, and using them to sign tokens on the control plane, Kong Mesh also offers offline signing of tokens.

In this flow, you can generate a pair of public and private keys and configure the control plane only with public keys for token verification. You can generate all the tokens without running the control plane.

The advantages of this mode are:

• It allows for easier and more reproducible deployments of the control plane, and it’s more in line with GitOps.
• It’s a potentially more secure setup, because the control plane doesn’t have access to the private keys.

Here’s how to use offline issuing:

1. Generate a pair of signing keys:

   kumactl generate signing-key --format=pem > /tmp/key-private.pem
   kumactl generate public-key --signing-key-path=/tmp/key-private.pem > /tmp/key-public.pem
   

   Copied!

   These commands generate standard RSA key of 2048 bits and outputs it in PEM-encoded format. You can use any external tool to generate a pair of keys. The result should look like this:

   cat /tmp/key-private.pem /tmp/key-public.pem 
   -----BEGIN RSA PRIVATE KEY-----
   MIIEpAIBAAKCAQEAsS61a79gC4mkr2Ltwi09ajakLyUR8YTkJWzZE805EtTkEn/r
   ...
   htKtzsYA7yGlt364IuDybrP+PlPMSK9cQAmWRRZIcBNsKOODkAgKFA==
   -----END RSA PRIVATE KEY-----
   -----BEGIN RSA PUBLIC KEY-----
   MIIBCgKCAQEAsS61a79gC4mkr2Ltwi09ajakLyUR8YTkJWzZE805EtTkEn/rL2u/
   ...
   se7sx2Pt/NPbWFFTMGVFm3A1ueTUoorW+wIDAQAB
   -----END RSA PUBLIC KEY----- 
   

2. Configure the control plane with the public key

   Configure a control plane with the following settings:

   dpServer:
     authn:
       zoneProxy:
         type: zoneToken
         zoneToken:
           enableIssuer: false # disable control plane token issuer that uses secrets
           validator:
             useSecrets: false # do not use signing key stored in secrets to validate the token
             publicKeys:
             - kid: "key-1"
               key: |
                 -----BEGIN RSA PUBLIC KEY-----
                 MIIBCgKCAQEAsS61a79gC4mkr2Ltwi09ajakLyUR8YTkJWzZE805EtTkEn/rL2u/
                 ...
                 se7sx2Pt/NPbWFFTMGVFm3A1ueTUoorW+wIDAQAB
                 -----END RSA PUBLIC KEY-----
   

   Copied!

3. Use the private key to issue tokens offline:

   kumactl generate zone-token \
     --zone us-east \
     --scope egress \
     --valid-for 720h \
     --signing-key-path /tmp/key-private.pem \
     --kid key-1
   

   Copied!

   The command is the same as with online signing, but with two additional arguments:

   • --kid: The ID of the key that should be used to validate the token. This should match kid specified in the control plane configuration.
   • --signing-key-path: The path to a PEM-encoded private key.

   You can also use any external system that can issue JWT tokens using RS256 signing method with the following claims:

   • Zone: The name of the zone.
   • Scope: The list of scopes (egress, ingress).

Migration

You can use both offline and online issuing by setting dpServer.authn.zoneProxy.zoneToken.enableIssuer to true. You can use both secrets and public key static config validators by setting dpServer.authn.zoneProxy.zoneToken.validator.useSecrets to true.

Management

Token revocation works the same when using both online and offline issuing.

Signing key rotation works similarly:

1. Generate another pair of signing keys.
2. Configure a control plane with old and new public keys.
3. Regenerate tokens for all existing zone proxies with the new private key.
4. Remove the old public key from the configuration.

When running in multi-zone mode, you can only generate zone tokens on the global control plane. The zone control plane only has a public key signing key to verify tokens.

You can turn off authentication by setting KUMA_DP_SERVER_AUTH_TYPE to none.

Do not disable authentication between the control plane and data plane proxies in production. Without authentication, any data plane proxy can impersonate any service, allowing unauthorized access to your mesh.