Configure HashiCorp Vault as a vault backend with certificate authentication

Uses: Kong Gateway decK

Deployment Platform:

Prerequisites

Kong Gateway running

This tutorial requires Kong Gateway Enterprise. If you don’t have Kong Gateway set up yet, you can use the quickstart script with an enterprise license to get an instance of Kong Gateway running almost instantly.

Export your license to an environment variable:

 export KONG_LICENSE_DATA='LICENSE-CONTENTS-GO-HERE'

Run the quickstart script:

curl -Ls https://get.konghq.com/quickstart | bash -s -- -e KONG_LICENSE_DATA 

Once Kong Gateway is ready, you will see the following message:

 Kong Gateway Ready

decK

decK is a CLI tool for managing Kong Gateway declaratively with state files. To complete this tutorial you will first need to install decK.

Generate client certificates

To complete this tutorial, you need client certificates. If you don’t have client certificates, you can use the following script and steps to generate them:

Save this script as gen_certs.sh in your home directory:

#!/bin/bash

# Generate root CA private key
openssl genrsa -out rootCA.key 4096

# Create root CA certificate
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 -out rootCA.crt -subj "/C=US/ST=State/L=City/O=Organization/OU=Department/CN=Root CA"

# Generate server private key
openssl genrsa -out kong.example.com.key 2048

# Create server CSR
openssl req -new -key kong.example.com.key -out kong.example.com.csr -subj "/C=US/ST=State/L=City/O=Organization/OU=Department/CN=kong.example.com"

# Create server.ext file for SANs
cat > kong.example.com.ext <<EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = kong.example.com
EOF

# Sign server CSR with root CA
openssl x509 -req -in kong.example.com.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out kong.example.com.crt -days 365 -sha256 -extfile kong.example.com.ext

# Clean up
rm kong.example.com.csr
rm rootCA.srl

echo "Root CA and server key pair generated successfully."

Generate the certificates from the script:

mkdir -p ~/vault/certs && cd ~/vault/certs
bash ~/gen_certs.sh

Generate a certificate for HashiCorp Vault

To complete this tutorial, you need a certificate that is served by HashiCorp Vault. The subjectAltName must match the name requested by Kong Gateway. This is used in the config.hcl file.

To generate the certificate, run the following from your home directory in terminal:

openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout ./vault/certs/vault.key \
  -out ./vault/certs/vault.crt \
  -subj "/CN=localhost" \
  -addext "subjectAltName = DNS:host.docker.internal, IP:127.0.0.1"

Configure HashiCorp Vault

Before you can configure the Vault entity in Kong Gateway, you must configure HashiCorp Vault to authenticate clients based on certificates signed by the provided root CA certificate and store a secret.

Create configuration files

First, you need to create the primary configuration file config.hcl for HashiCorp Vault in the ./vault directory:

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "./vault/certs/vault.crt"
  tls_key_file  = "./vault/certs/vault.key"
}

storage "file" {
  path = "./vault/data"
}

ui = true

Then, create the HashiCorp policy file rw-secrets.hcl in the ./vault directory:

# Full access to everything — use with caution!
path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

Configure the Vault and store a secret

Now, you can configure HashiCorp Vault to use certificate-based authentication.

Start HashiCorp Vault:

vault server -config=./vault/config.hcl

In a new terminal, configure HashiCorp Vault to trust the TLS certificate:
```
export VAULT_CACERT=$HOME/vault/certs/vault.crt
```

Initialize the Vault:

vault operator init -key-shares=1 -key-threshold=1

This will output your unseal key and your inital root token. Export them as environment variables:

export HCV_UNSEAL_KEY='YOUR-UNSEAL-KEY'
export DECK_HCV_TOKEN='YOUR-INITIAL-ROOT-TOKEN'

Unseal your Vault:
```
vault operator unseal $HCV_UNSEAL_KEY
```
Login to your Vault:
```
vault login $DECK_HCV_TOKEN
```

Write the policy to access secrets:

vault policy write rw-secrets ./vault/rw-secrets.hcl

Enable cert-based authentication:
```
vault auth enable cert
```
Configure Vault to authenticate clients based on certificates signed by the provided root CA certificate:
```
vault write auth/cert/config certificate=./vault/certs/rootCA.crt
```

vault write auth/cert/certs/gw311 \
   display_name="gw311" \
   policies="rw-secrets" \
   certificate=@./vault/certs/rootCA.crt \
   allow_subdomains=false

Test the login using certificate authentication:

vault login -method=cert \
  -client-cert=./vault/certs/kong.example.com.crt \
  -client-key=./vault/certs/kong.example.com.key

Enable the K/V secrets engine:
```
vault secrets enable -path=kong kv
```

Create a secret:

vault kv put kong/headers/request header="x-kong:test"

Confirm you can retrieve the secret through Vault:
```
vault kv get kong/headers/request
```

Configure the Lua SSL trusted certificate

Because the certificates in this tutorial are self-signed, we must configure the lua_ssl_trusted_certificate parameter in kong.conf to use the certificate that is served from HashiCorp Vault, vault.crt.

In terminal, copy your vault.crt file to your Docker container:

 docker cp ./vault/certs/vault.crt kong-quickstart-gateway:./vault.crt

In your Docker container, make a copy of the default Kong configuration file:
```
 cp /etc/kong/kong.conf.default /etc/kong/kong.conf
```
Open kong.conf in your Docker container, find lua_ssl_trusted_certificate, uncomment it and replace it with the following:
```
 lua_ssl_trusted_certificate = ./vault.crt
```
Reload Kong Gateway in your Docker container to get the setting to take effect:
```
 kong reload -c /etc/kong/kong.conf
```

Set environment variables

Export the following environment variables:

export DECK_HCV_HOST=host.docker.internal
export DECK_HCV_CERT_KEY=$(awk 'NR > 1 {printf "\\n"} {printf "%s", $0} END {printf ""}' ./vault/certs/kong.example.com.key)
export DECK_HCV_CERT=$(awk 'NR > 1 {printf "\\n"} {printf "%s", $0} END {printf ""}' ./vault/certs/kong.example.com.crt)

In this tutorial, we’re using host.docker.internal as our host instead of the localhost variable that HashiCorp Vault is using. This is because if you used the quickstart script, Kong Gateway is running in a container and uses a different localhost.

Create a Vault entity for HashiCorp Vault

Using decK, create a Vault entity in the kong.yaml file with the required parameters for HashiCorp Vault:

echo '
_format_version: "3.0"
vaults:
  - name: hcv
    prefix: hashicorp-vault
    description: Storing secrets in HashiCorp Vault
    config:
      host: "${{ env "DECK_HCV_HOST" }}"
      token: "${{ env "DECK_HCV_TOKEN" }}"
      kv: v1
      mount: kong
      port: 8200
      protocol: https
      auth_method: cert
      cert_auth_cert_key: "${{ env "DECK_HCV_CERT_KEY" }}"
      cert_auth_cert: "${{ env "DECK_HCV_CERT" }}"
' | deck gateway apply -

Validate

To validate that the secret was stored correctly in HashiCorp Vault, you can call a secret from your vault using the kong vault get command within the Data Plane container.

 kong vault get {vault://hashicorp-vault/headers/request/header}

If the vault was configured correctly, this command should return the value of the secret. You can use {vault://hashicorp-vault/headers/request/header} to reference the secret in any referenceable field.

Cleanup

Clean up HashiCorp Vault

Stop the HashiCorp Vault dev server process by running the following:

pkill vault

Unset environment variables:

unset VAULT_ADDR

Destroy the Kong Gateway container

curl -Ls https://get.konghq.com/quickstart | bash -s -- -d

Clean up Konnect environment

If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.

FAQs

How do I rotate my secrets in HashiCorp Vault and how does Kong Gateway pick up the new secret values?

You can rotate your secret in HashiCorp Vault by creating a new secret version with the updated value. You’ll also want to configure the ttl settings in your Kong Gateway Vault entity so that Kong Gateway pulls the rotated secret periodically.

How does Kong Gateway retrieve secrets from HashiCorp Vault?

Kong Gateway retrieves secrets from HashiCorp Vault’s HTTP API through a two-step process: authentication and secret retrieval.

Step 1: Authentication

Depending on the authentication method defined in config.auth_method, Kong Gateway authenticates to HashiCorp Vault using one of the following methods:

If you’re using the token auth method, Kong Gateway uses the config.token as the client token.
If you’re using the kubernetes auth method, Kong Gateway uses the service account JWT token mounted in the pod (path defined in the config.kube_api_token_file) to call the login API for the Kubernetes auth path on the HashiCorp Vault server and retrieve a client token.
v3.4+ If you’re using the approle auth method, Kong Gateway uses the AppRole credentials to retrieve a client token. The AppRole role ID is configured by field config.approle_role_id, and the secret ID is configured by field config.approle_secret_id or config.approle_secret_id_file.
- If you set config.approle_response_wrapping to true, then the secret ID configured by config.approle_secret_id or config.approle_secret_id_file will be a response wrapping token, and Kong Gateway will call the unwrap API /v1/sys/wrapping/unwrap to unwrap the response wrapping token to fetch the real secret ID. Kong Gateway will use the AppRole role ID and secret ID to call the login API for the AppRole auth path on the HashiCorp Vault server and retrieve a client token.
v3.11+ If you’re using the cert auth method, Kong Gateway uses a client certificate and private key to retrieve a client token. The certificate must be previously configured in HashiCorp vault as a trusted certificate. Alternatively, the issuing CA certificate can be set as a trusted CA. The trusted certificate role name is configured by the field config.cert_auth_role_name. If one isn’t provided, HashiCorp vault attempts to authenticate against all configured trusted certificates or trusted CAs. The certificate is configured with config.cert_auth_cert and the key with cert_auth_cert_key.

By calling the login API, Kong Gateway will retrieve a client token and then use it in the next step as the value of X-Vault-Token header to retrieve a secret.

Step 2: Retrieving the secret

Kong Gateway uses the client token retrieved in the authentication step to call the Read Secret API and retrieve the secret value. The request varies depending on the secrets engine version you’re using. Kong Gateway will parse the response of the read secret API automatically and return the secret value.

I get a Client sent an HTTP request to an HTTPS server. error when I try to retrieve my secret, how do I fix this?

Configure your Vault entity to use HTTPS instead of HTTP. This can be done updating your config.hcl. You will need to set the address to include the https protocol and include the certificate/key in the tls_cert_file and tls_key_file parameters.

I’m getting an unable to retrieve secret from vault: 18: self-signed certificate error, how do I fix this?

Add the self-signed certificate in kong.conf to the parameter lua_ssl_trusted_certificate.

I’m getting an unable to retrieve secret from vault: certificate host mismatch error, how do I fix this?

The hostname specified in your Vault entity does not match the cert presented by the Vault server. Confirm the Kong Vault config.host matches the name of the certificate presented by the Vault server certificate.

I’m getting an invalid response code 400 received when performing certificate-based authentication: {"errors":["auth methods cannot create root tokens"]} error, how do I fix this?

The certificate is mapped to a policy that would allow creation of a root token, which Vault explicitly forbids. Check the policy associated with your certificate to ensure that it does not include CREATE, UPDATE, READ operations on the path auth/token/root

I’m getting an ailure performing certificate-based authentication: 21: unable to verify the first certificate error, how do I fix this?

This will usually occur for one of two reasons:

You have a certificate chain and only a portion of it was uploaded for Kong Gateway to use. Fix: Include the entire chain in KONG_LUA_SSL_TRUSTED_CERTIFICATE.
HashiCorp Vault was setup in dev mode. This does not allow you to provide your own CA and instead uses an ephemeral cert for SAN: “localhost, 127.0.0.1, 0.0.0.0.”

I’m getting an invalid response code 503 received when performing certificate-based authentication: {"errors":["Vault is sealed"]} error, how do I fix this?

To perform any operation on the Vault, it must be unsealed first. It was likely sealed intentionally or through a restart of the Vault process.

Next Steps

Review the Vaults entity