Kong Gateway: "could not find cached values" error with the environment variables vault backend

When referencing a secret from the environment variables vault backend with a configured prefix, Kong Gateway logs a could not find cached values error instead of resolving the value.

2024/03/07 16:47:52 [notice] 2307#0: *69496 [kong] vault.lua:708 error updating secret reference {vault://env/clientsecret}: could not find cached values, client: 172.28.0.1, server: kong, request: "GET /check HTTP/1.1", host: "localhost:48000"

There are different kinds of vault backends, one of which is environment variables.

When using this, we would reference the variable to be used, as an environment variable in the Kong configuration file and then the reference will be able to pick it up when executing the plugin.

Things to check:

1. The environment variable in the docker file should have these added: The variables that would be referenced with the vault and if you are using an environment variable prefix, then KONG_VAULT_ENV_PREFIX should be added. In my example below, my KONG_VAULT_ENV_PREFIX=MY_ and hence all the variables that I want this vault to reference will also have this prefix:

-e "MY_CLIENTID=<Clientid>" \

-e "KONG_VAULT_ENV_PREFIX=MY_" \

-e "MY_CLIENTSECRET=<clientsecret>" \

1. Note that these environment variables should be specified in upper case. If specified in lower case, then they are not valid. More on this can be found here.

2. Vault configuration: The name of the vault is: my-env-vault, type: env, config.prefix="MY_". The config.prefix advanced setting should match the env variable: KONG_VAULT_ENV_PREFIX

3. Check from inside Kong container if the reference is right. For this exec into the Kong container and execute the following command to view env variables and make sure the variables you created are showing up:

printenv | sort

Execute the below command to know if from Kong container, we are able to reference these variables through vault parameters:

kong vault get <Vault name/prefix>/environment_variable 
Eg: kong vault get my-env-vault/clientsecret

Note that the prefix (MY_) is not added here to the env variable name. If we add them (my_clientid) and check, we will get the error found here:

1. You could then reference as shown below in any of the supported referenceable fields in the plugin.

Here I am referencing these in the OIDC plugin clientid and clientsecret fields.

Syntax:

{vault://<vault name/prefix/<env variable name>}

I am getting redirected to the IDP since the client ID and secret is valid: