Home / Kong Operator / Konnect
Edit
Report

Store TLS certificate private keys in a Konnect Config Store

Uses: Kong Operator
Incompatible with
on-prem
Related Documentation
Operator Documentation
Tags
#konnect-crd #secrets-management #certificates
Related Resources
Create a Vault
Create a Certificate and CA Certificate
Config Store-backed Vaults
Cross namespace references
Status fields
Configure the Konnect Config Store vault
Minimum Version
Kong Operator - 2.3
TL;DR

Create a KonnectConfigStore, point a KongVault to it with backend: konnect using spec.configStoreRef, write the private key directly into the Config Store, and then set KongCertificate.spec.key to a {vault://PREFIX/KEY} reference instead of the key material.

Prerequisites

Kong Konnect

If you don’t have a Konnect account, you can get started quickly with our onboarding wizard.

  1. The following Konnect items are required to complete this tutorial:
    • Personal access token (PAT): Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.

  2. Set the personal access token as an environment variable:

    export KONNECT_TOKEN='YOUR KONNECT TOKEN'

Kong Operator running

  1. Add the Kong Helm charts:

    helm repo add kong https://charts.konghq.com
helm repo update

  2. Install Kong Operator using Helm:

    helm upgrade --install kong-operator kong/kong-operator -n kong-system \
  --create-namespace \
  --set image.tag=2.3 \
  --set env.ENABLE_CONTROLLER_KONNECT=true

    If you want cert-manager to issue and rotate the admission and conversion webhook certificates, install cert-manager to your cluster and enable cert-manager integration by passing the following argument while installing, in the next step:

    --set global.webhooks.options.certManager.enabled=true

    If you do not enable this, the chart will generate and inject self-signed certificates automatically. We recommend enabling cert-manager to manage the lifecycle of these certificates. Kong Operator needs a certificate authority to sign the certificate for mTLS communication between the control plane and the data plane. This is handled automatically by the Helm chart. If you need to provide a custom CA certificate, refer to the certificateAuthority section in the values.yaml of the Helm chart to learn how to create and reference your own CA certificate.

Create a KonnectAPIAuthConfiguration resource

 
kubectl create namespace kong --dry-run=client -o yaml | kubectl apply -f -
echo '
kind: KonnectAPIAuthConfiguration
apiVersion: konnect.konghq.com/v1alpha1
metadata:
  name: konnect-api-auth
  namespace: kong
spec:
  type: token
  token: "'$KONNECT_TOKEN'"
  serverURL: us.api.konghq.com
' | kubectl apply -f -

Create a KonnectGatewayControlPlane resource

 
echo '
kind: KonnectGatewayControlPlane
apiVersion: konnect.konghq.com/v1alpha2
metadata:
  name: gateway-control-plane
  namespace: kong
spec:
  createControlPlaneRequest:
    name: gateway-control-plane
  konnect:
    authRef:
      name: konnect-api-auth
' | kubectl apply -f -

A KongCertificate normally carries the private key as PEM material in spec.key, which means the key is stored in your Kubernetes manifests, in etcd, and in the Konnect certificate object. Anyone who can read the KongCertificate or the Konnect certificate can read the key.

Instead, you can store the key in a Konnect Config Store and reference it from spec.key with a Kong vault reference. The reference is passed through unchanged, so the key material never enters your cluster and never appears in the certificate object.

For more information, see Config Store-backed Vaults.

Create a KonnectConfigStore

Use the KonnectConfigStore resource to create the Config Store container in Konnect. It must reference the KonnectGatewayControlPlane that owns the Config Store:

echo '
kind: KonnectConfigStore
apiVersion: konnect.konghq.com/v1alpha1
metadata:
  name: cert-keys
  namespace: kong
spec:
  controlPlaneRef:
    type: namespacedRef
    namespacedRef:
      name: gateway-control-plane
  apiSpec:
    name: cert-keys
' | kubectl apply -f -

You can verify the KonnectConfigStore was reconciled successfully by checking its Programmed condition.

kubectl get -n kong konnectconfigstore cert-keys \
  -o=jsonpath='{.status.conditions[?(@.type=="Programmed")]}' | jq

The output should look similar to this:

{
  "observedGeneration": 1,
  "reason": "Programmed",
  "status": "True",
  "type": "Programmed"
}

Allow the KongVault to reference the Config Store

KongVault is cluster-scoped and KonnectConfigStore is namespaced, so the reference always crosses a namespace boundary and the namespace that holds the Config Store must allow it with a KongReferenceGrant. Because a KongVault has no namespace of its own, the grant must list namespace: "" in its from entry.

The same grant also covers the KongVault reference to the KonnectGatewayControlPlane, which crosses a namespace boundary for the same reason:

echo '
apiVersion: configuration.konghq.com/v1alpha1
kind: KongReferenceGrant
metadata:
  name: allow-kongvault-to-konnect-resources
  namespace: kong
spec:
  from:
    - group: configuration.konghq.com
      kind: KongVault
      namespace: ""
  to:
    - group: konnect.konghq.com
      kind: KonnectConfigStore
    - group: konnect.konghq.com
      kind: KonnectGatewayControlPlane' | kubectl apply -f -

Create a KongVault backed by the Config Store

Create a KongVault with backend: konnect and point spec.configStoreRef to the KonnectConfigStore. Kong Operator resolves the reference into the config_store_id of the Vault configuration:

echo '
kind: KongVault
apiVersion: configuration.konghq.com/v1alpha1
metadata:
  name: certvault
  namespace: kong
spec:
  backend: konnect
  prefix: certvault
  description: TLS certificate private keys
  configStoreRef:
    kind: KonnectConfigStore
    name: cert-keys
    namespace: kong
  controlPlaneRef:
    type: konnectNamespacedRef
    konnectNamespacedRef:
      name: gateway-control-plane
      namespace: kong
' | kubectl apply -f -

You can verify the KongVault was reconciled successfully by checking its Programmed condition.

kubectl get -n kong kongvault certvault \
  -o=jsonpath='{.status.conditions[?(@.type=="Programmed")]}' | jq

The output should look similar to this:

{
  "observedGeneration": 1,
  "reason": "Programmed",
  "status": "True",
  "type": "Programmed"
}

Confirm that the KongVault accepted the Config Store reference: 

kubectl get -o yaml -n kong kongvault certvault \
  | yq '.status.conditions[] | select(.type == "ConfigStoreRefValid") | .status'

You should see the value True.

For more information about spec.configStoreRef and spec.prefix, see Config Store-backed Vaults.

Generate a certificate

Generate a self-signed certificate and key to work with:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=example.localdomain.dev"

Write the private key into the Config Store

  1. Fetch the Konnect IDs of the control plane and of the Config Store that Kong Operator created. The Config Store ID is published on the KonnectConfigStore status, so you don’t have to look it up in the Konnect UI:

    export CONTROL_PLANE_ID=$(kubectl get -n kong konnectgatewaycontrolplanes.konnect.konghq.com gateway-control-plane -o jsonpath='{.status.id}')
echo $CONTROL_PLANE_ID
export CONFIG_STORE_ID=$(kubectl get -n kong konnectconfigstores.konnect.konghq.com cert-keys -o jsonpath='{.status.id}')
echo $CONFIG_STORE_ID

  2. Build the request body, using jq to embed the PEM file as a JSON string:

    jq -n --arg key example-tls-key --rawfile value tls.key '{key: $key, value: $value}' > secret.json

  3. Store the private key as a secret entry in the Config Store:

       curl -X POST "https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/config-stores/$CONFIG_STORE_ID/secrets" \
        --no-progress-meter --fail-with-body  \
        -H "Authorization: Bearer $KONNECT_TOKEN" \
        --json "$(cat secret.json)"

  4. Delete the local copies of the key once it’s stored:

    rm tls.key secret.json

Create a KongCertificate that references the key

Set spec.key to a vault reference of the form {vault://PREFIX/KEY}, where PREFIX is the spec.prefix of the KongVault and KEY is the Config Store entry you created. The public certificate isn’t sensitive, so it stays inline:

echo "
kind: KongCertificate
apiVersion: configuration.konghq.com/v1alpha1
metadata:
  name: cert-with-vault-key
  namespace: kong
spec:
  controlPlaneRef:
    type: konnectNamespacedRef
    konnectNamespacedRef:
      name: gateway-control-plane
  cert: |
$(sed 's/^/    /' tls.crt)
  key: '{vault://certvault/example-tls-key}'" | kubectl apply -f -

You can verify the KongCertificate was reconciled successfully by checking its Programmed condition.

kubectl get -n kong kongcertificate cert-with-vault-key \
  -o=jsonpath='{.status.conditions[?(@.type=="Programmed")]}' | jq

The output should look similar to this:

{
  "observedGeneration": 1,
  "reason": "Programmed",
  "status": "True",
  "type": "Programmed"
}

For more information about which certificate fields accept vault references, see Can I reference certificate material from a Vault?.

Validate

Confirm that the certificate stored in Konnect holds the vault reference rather than the key material:

curl -X GET "https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/core-entities/certificates" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN"

The value of the key field in the response is {vault://certvault/example-tls-key}. The reference is resolved after Kong Gateway connects to the control plane, so the certificate object itself never contains the key.

FAQs

Why isn’t the secret value stored declaratively?

The purpose of this guide is to avoid storing the secret in etcd, which would be the case if the key was passed through a Kubernetes resource. Instead, we pass it directly to Konnect using either the API or the UI.

Who can replace the private key once it’s in the Config Store?

Anyone who can write secrets into the Config Store can replace the key that your listener serves, so treat Config Store write access as equivalent to certificate issuance rights. For a full breakdown of where the key does and doesn’t exist, see the security boundary of a Config Store-backed Vault.

What happens if I delete the KonnectConfigStore?

Deleting the KonnectConfigStore deletes the Config Store in Konnect along with every secret stored in it, including keys that other Vaults or certificates still reference. Delete it only when you’re sure nothing depends on its contents. For more information, see lifecycle and deletion.

How do I rotate a key that’s stored in the Config Store?

Rotating a key is a Config Store operation: update the secret value in place, and the vault reference keeps resolving without any change to your Kubernetes resources.

Can I use vault references with spec.type: secretRef?

No. Vault references are only supported with the default spec.type: inline. spec.type: secretRef reads the certificate and key from a Kubernetes Secret, which stores the key in etcd.

Why isn’t my KongVault programmed in Konnect?

If the KongVault isn’t programmed in Konnect, check the ConfigStoreRefValid condition:

kubectl get kongvault certvault -o jsonpath-as-json="{.status.conditions[?(@.type=='ConfigStoreRefValid')]}"

For the meaning of each condition reason, see ConfigStoreRefValid on KongVault.

The Vault is programmed, but Kong Gateway fails the TLS handshake. What should I check?

If the Vault is programmed but Kong Gateway fails the TLS handshake, the reference itself is usually fine and the secret entry is the problem. Confirm that the key exists in the Config Store and that its name matches the vault reference:

curl -X GET "https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/config-stores/$CONFIG_STORE_ID/secrets" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN"

Next Steps

Map hostnames to the certificate with SNIs
Review what else can be stored as a secret
Something wrong?
Report an Issue | Edit this Page
View all Kong documentation

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!
Contribute

Still need help?

Ask in our Slack Community
Contact Support