Authenticate Kong Event Gateway connections to Kafka using AWS IAM

For the purposes of tutorial, we assume you have an IAM user with permissions needed to access the cluster as described in the prerequisites and an access key (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Adjust this to fit your production deployment. If you’re running on EC2, for example, KEG handles calling IMDS to get credentials.

Run the quickstart script to provision a local data plane and configure your environment:

curl -Ls https://get.konghq.com/event-gateway | bash -s -- -k $KONNECT_TOKEN -e AWS_ACCESS_KEY_ID=... -e AWS_SECRET_ACCESS_KEY=... -N kafka_event_gateway

Copy the exported variable into your terminal:

export EVENT_GATEWAY_ID=your-gateway-id

This quickstart script is meant for demo purposes only, therefore it runs locally with most default parameters and a small number of exposed ports. If you want to run Kong Gateway as a part of a production-ready platform, set up your control plane and data planes through the Konnect UI, or using Terraform.

Unlike the local Docker Compose examples for mTLS and SASL/PLAIN, this guide connects to a real Amazon MSK cluster. Make sure the host running the Event Gateway data plane can reach your MSK cluster’s bootstrap brokers, for example through VPC peering, a VPN, or by running the data plane inside the same VPC.

Create a kafkactl.yaml config file with a context for the AWS IAM virtual cluster:

cat <<EOF > kafkactl.yaml
contexts:
  aws-iam-vc:
    brokers:
      - localhost:19092
EOF

Clients only ever authenticate anonymously to this virtual cluster. AWS IAM is used solely by Event Gateway itself, when it connects to the backend cluster on the client’s behalf.

Export the bootstrap broker addresses for your MSK cluster’s IAM listener:

export MSK_BOOTSTRAP_SERVERS="b-1.example.abc123.c2.kafka.us-east-1.amazonaws.com:9098,b-2.example.abc123.c2.kafka.us-east-1.amazonaws.com:9098,b-3.example.abc123.c2.kafka.us-east-1.amazonaws.com:9098"

Build the request body for a backend cluster using sasl_aws_iam authentication with the default AWS credentials provider chain:

jq -n --arg brokers "$MSK_BOOTSTRAP_SERVERS" \
  '{
    "name": "msk_backend_cluster",
    "bootstrap_servers": ($brokers | split(",")),
    "authentication": {
      "type": "sasl_aws_iam",
      "sasl_aws_iam": {"type": "default_provider_chain"}
    },
    "insecure_allow_anonymous_virtual_cluster_auth": true,
    "tls": {"enabled": true}
  }' > msk_backend_cluster.json

Then, create the backend cluster:

MSK_BACKEND_CLUSTER_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/backend-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json "$(cat msk_backend_cluster.json)" | jq -r ".id"
)

default_provider_chain tells Event Gateway to resolve AWS credentials the same way the AWS CLI and SDKs do.

Assume an IAM role instead

If you need Event Gateway to assume a role before connecting, for example to access an MSK cluster in another AWS account, use sasl_aws_iam.type: assume_role instead:

{
  "authentication": {
    "type": "sasl_aws_iam",
    "sasl_aws_iam": {
      "type": "assume_role",
      "assume_role": {
        "arn": "arn:aws:iam::123456789012:role/example-role",
        "session_name": "event-gateway-session"
      }
    }
  }
}

Event Gateway first resolves base credentials from the default provider chain, then uses them to assume the role at arn. The role’s trust policy must allow the base credentials to assume it. session_name is optional and becomes part of the assumed role session’s ARN.

Create a virtual cluster with anonymous authentication:

MSK_VC_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "aws_iam_vc",
       "destination": {
         "id": "'$MSK_BACKEND_CLUSTER_ID'"
       },
       "dns_label": "aws-iam-vc",
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "acl_mode": "passthrough"
     }' | jq -r ".id"
)

Run the following command to create a new listener:

MSK_LISTENER_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/listeners" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "aws_iam_listener",
       "addresses": [
         "0.0.0.0"
       ],
       "ports": [
         "19092-19105"
       ]
     }' | jq -r ".id"
)

Add a Forward to Virtual Cluster policy, which will forward requests based on a defined mapping to our virtual cluster:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/listeners/$MSK_LISTENER_ID/policies" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "type": "forward_to_virtual_cluster",
       "name": "forward_to_aws_iam_vc",
       "config": {
         "type": "port_mapping",
         "advertised_host": "localhost",
         "destination": {
           "id": "'$MSK_VC_ID'"
         }
       }
     }'

Ensure there’s at least one topic on your MSK cluster. Then list the topics through the aws-iam-vc virtual cluster:

kafkactl -C kafkactl.yaml --context aws-iam-vc list topics

If Event Gateway authenticated to Amazon MSK using AWS IAM and forwarded the metadata request, you will see the topic listed:

TOPIC                    PARTITIONS     REPLICATION FACTOR
test-topic               1              1