Kong Mesh: 'admission webhook "validator.kuma-admission.kuma.io" denied the request: access denied' error when the default AccessRole or AccessRoleBinding is missing or too restrictive

When attempting to create a new resource we are receiving the below error despite my user being part of the system:masters & system:authenticated groups.

Error from server: error when creating "mesh.yaml": admission webhook "validator.kuma-admission.kuma.io" denied the request: access denied: user "arn:aws:iam::12345:role/NonProd_ServiceMesh_DevUsers/system:masters,system:authenticated" cannot access the resource

I have also confirmed my user has access to creating the resource, in this case, a new mesh. What is causing this?

kubectl auth can-i create mesh
yes

This issue can occur when the default AccessRole and AccessRoleBinding have either been modified in an overly restrictive manner or do not exist.

If you do not have another user with a different AccessRole/AccessRoleBinding associated you will need to modify the webhook to unblock yourself.

Note: As always, it is important to test this process in lower environments before attempting in production.

1. IMPORTANT: Backup the existing webhook config before proceeding

   kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io kong-mesh-validating-webhook-configuration -oyaml > webhook.yaml

   Copied!

2. Delete the webhook

   k delete -f webhook.yaml

   Copied!

3. Re-create the AccessRole and/or AccessRoleBinding (depending on what is missing)

   AccessRole

   echo "
   apiVersion: v1
   items:
   - apiVersion: kuma.io/v1alpha1
     kind: AccessRole
     metadata:
       name: admin
     spec:
       rules:
       - access:
         - CREATE
         - UPDATE
         - DELETE
         - GENERATE_DATAPLANE_TOKEN
         - GENERATE_USER_TOKEN
         - GENERATE_ZONE_CP_TOKEN
         - GENERATE_ZONE_TOKEN
         - VIEW_CONFIG_DUMP
         - VIEW_STATS
         - VIEW_CLUSTERS
   kind: List
   metadata:
     resourceVersion: ''
   " | kubectl apply -f -

   Copied!

   AccessRoleBinding

   echo "
   apiVersion: v1
   items:
   - apiVersion: kuma.io/v1alpha1
     kind: AccessRoleBinding
     metadata:
       name: default
     spec:
       roles:
       - admin
       subjects:
       - name: mesh-system:authenticated
         type: Group
       - name: mesh-system:unauthenticated
         type: Group
       - name: system:authenticated
         type: Group
       - name: system:unauthenticated
         type: Group
   kind: List
   metadata:
     resourceVersion: ''
   " | kubectl apply -f -

   Copied!

4. Re-create the webhook

   kubectl apply -f webhook.yaml

   Copied!

5. Confirm you can now remove resources

   kubectl delete mesh gruber
   mesh.kuma.io "gruber" deleted

   Copied!