We’re trying to set up Kong Mesh with the Vault Policy for integration with Hashicorp Vault but we are receiving errors after a period of time. It’s as if the Hashicorp token is no longer valid and thus it’s unable to generate the certificate. We see this error in the Kong Mesh logs:
2023-08-08T17:21:29.041Z ERROR xds-server.dataplane-sync-watchdog OnTick() failed {"dataplaneKey": {"Mesh":"{nameOfMeshCluster}","Name":"cluster-mgi-58cf779d4c-tntjr.kong-mesh-system"}, "error": "failed to generate a snapshot: imports[0]{name=\"gateway-proxy\"}: secrets.Generator failed: failed to generate dataplane identity cert and CAs: could not generate certificates: could not get Dataplane cert pair: could not generate dataplane cert for mesh: \"nonprod\" backend: \"vault-1\" services: \"kuma.io/service={ServiceName} kuma.io/zone={ServiceName}\": permission denied - use token that allows to generate cert of pki-internal/issue/np-kong-mesh", "errorVerbose": "imports[0]{name=\"gateway-proxy\"}: secrets.Generator failed: failed to generate dataplane identity cert and CAs: could not generate certificates: could not get Dataplane cert pair: could not generate dataplane cert for mesh: \"{nameOfMeshCluster}\" backend: \"vault\" services: \"kuma.io/service={ServiceName} kuma.io/zone={ServiceName}\": permission denied - use token that allows to generate cert of pki-internal/issue/np-kong-mesh\nfailed to generate a snapshot\ngithub.com/kumahq/kuma/pkg/xds/server/v3.(*reconciler).Reconcile\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/xds/server/v3/reconcile.go:59\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).syncDataplane\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/xds/sync/dataplane_watchdog.go:130\ngithub.com/kumahq/kuma/pkg/xds/sync.(*DataplaneWatchdog).Sync\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/xds/sync/dataplane_watchdog.go:66\ngithub.com/kumahq/kuma/pkg/xds/sync.(*dataplaneWatchdogFactory).New.func2\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/xds/sync/dataplane_watchdog_factory.go:46\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).onTick\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/util/watchdog/watchdog.go:54\ngithub.com/kumahq/kuma/pkg/util/watchdog.(*SimpleWatchdog).Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230414081006-9a2812c6b3a4/pkg/util/watchdog/watchdog.go:27\nruntime.goexit\n\truntime/asm_amd64.s:1598"}Why is this happening and what is the solution? Why does this token seem to expire so quickly?