Home / Event Gateway
Edit
Report

Configure topic aliases with Kong Event Gateway

Uses: Event Gateway Konnect API
Incompatible with
on-prem
Related Documentation
Event Gateway Documentation
Tags
#event-gateway #kafka
Related Resources
Productize Kafka topics with Kong Event Gateway
Event Gateway Control Plane API
Minimum Version
Event Gateway - 1.2.0
TL;DR

Use topic aliases on a virtual cluster to map client-visible names to backend topic names:

  1. Create a backend cluster connected to your Kafka brokers.
  2. Create a virtual cluster with topic_aliases that map friendly names to backend topics.
  3. Create a listener to route traffic to the virtual cluster.
  4. Clients produce and consume using the alias names.

Prerequisites

Install kafkactl

Install kafkactl. You’ll need it to interact with Kafka clusters.

Start a local Kafka cluster

Start a Docker Compose cluster with multiple Kafka services.

First, we need to create a docker-compose.yaml file. This file will define the services we want to run in our local environment:

cat <<EOF > docker-compose.yaml
name: kafka_cluster

networks:
  kafka:
    name: kafka_event_gateway

services:
  kafka1:
    image: apache/kafka:4.3.0
    networks:
      - kafka
    container_name: kafka1
    ports:
      - "9094:9094"
    environment:
      KAFKA_NODE_ID: 0
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENERS: INTERNAL://kafka1:9092,CONTROLLER://kafka1:9093,EXTERNAL://0.0.0.0:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_QUORUM_VOTERS: 0@kafka1:9093,1@kafka2:9093,2@kafka3:9093
      KAFKA_CLUSTER_ID: 'abcdefghijklmnopqrstuv'
      KAFKA_LOG_DIRS: /tmp/kraft-combined-logs

  kafka2:
    image: apache/kafka:4.3.0
    networks:
      - kafka
    container_name: kafka2
    ports:
      - "9095:9095"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENERS: INTERNAL://kafka2:9092,CONTROLLER://kafka2:9093,EXTERNAL://0.0.0.0:9095
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:9092,EXTERNAL://localhost:9095
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_QUORUM_VOTERS: 0@kafka1:9093,1@kafka2:9093,2@kafka3:9093
      KAFKA_CLUSTER_ID: 'abcdefghijklmnopqrstuv'
      KAFKA_LOG_DIRS: /tmp/kraft-combined-logs

  kafka3:
    image: apache/kafka:4.3.0
    networks:
      - kafka
    container_name: kafka3
    ports:
      - "9096:9096"
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENERS: INTERNAL://kafka3:9092,CONTROLLER://kafka3:9093,EXTERNAL://0.0.0.0:9096
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:9092,EXTERNAL://localhost:9096
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_QUORUM_VOTERS: 0@kafka1:9093,1@kafka2:9093,2@kafka3:9093
      KAFKA_CLUSTER_ID: 'abcdefghijklmnopqrstuv'
      KAFKA_LOG_DIRS: /tmp/kraft-combined-logs

  schema-registry:
    image: confluentinc/cp-schema-registry:8.2.1
    networks:
      - kafka
    container_name: schema-registry
    depends_on:
      - kafka1
      - kafka2
      - kafka3
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092,kafka3:9092

EOF

Now, let’s start the local setup:

docker compose up -d

Configure a kafkactl context

Let’s define a context we can use to create Kafka topics:

cat <<EOF > kafkactl.yaml
contexts:
    direct:
      brokers:
        - localhost:9095
        - localhost:9096
        - localhost:9094
EOF

Kong Konnect

If you don’t have a Konnect account, you can get started quickly with our onboarding wizard.

  1. The following Konnect items are required to complete this tutorial:
    • Personal access token (PAT): Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.

  2. Set the personal access token as an environment variable:

    export KONNECT_TOKEN='YOUR KONNECT TOKEN'

Kong Event Gateway running

Run the quickstart script to automatically provision a demo Kong Gateway control plane and data plane, and configure your environment:

curl -Ls https://get.konghq.com/event-gateway | bash -s -- -k $KONNECT_TOKEN -N kafka_event_gateway

This sets up an Kong Gateway control plane named event-gateway-quickstart, provisions a local data plane, and prints out the following environment variable export:

export EVENT_GATEWAY_ID=your-gateway-id

Copy and paste the command with your Event Gateway ID into your terminal to configure your session.

This quickstart script is meant for demo purposes only, therefore it runs locally with most default parameters and a small number of exposed ports. If you want to run Kong Gateway as a part of a production-ready platform, set up your control plane and data planes through the Konnect UI, or using Terraform.

Overview

Topic aliases let you expose backend Kafka topics under different, client-friendly names. This is useful when backend topics follow internal naming conventions (like team-alpha-orders-v2) that you don’t want to expose to clients.

 
flowchart LR
    A[Kafka client] -->|produces/consumes to 'orders'| B[Event Gateway]
    B -->|resolves to 'team-alpha-orders-v2'| C[Kafka cluster]

Supported operations

Aliases are a read-only abstraction over physical topics:

  • Allowed: produce, fetch, list offsets, consumer group operations, and metadata (ListTopics). Both the alias and the original backend topic name appear in metadata responses.
  • Rejected with InvalidTopicException: CreateTopics, DeleteTopics, CreatePartitions, DeleteRecords, AlterPartitionReassignments, and ElectLeaders. Modifying a physical topic through an alias would be surprising because other aliases or clients may also depend on it.

ACLs are evaluated on the name the client uses, before alias resolution. An ACL on the backend topic does not automatically grant access to its aliases, and vice versa. Each alias name must be granted access explicitly. With acl_mode: enforce_on_gateway, a new alias with no matching ACL is blocked by default.

Create Kafka topics

Create the backend topics that we’ll expose through aliases:

kafkactl -C kafkactl.yaml --context direct create topic \
team-alpha-orders-v2 analytics-raw-clicks

Create a backend cluster

Use the following command to create a backend cluster that connects to the Kafka servers you set up:

BACKEND_CLUSTER_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/backend-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "backend_cluster",
       "bootstrap_servers": [
         "kafka1:9092",
         "kafka2:9092",
         "kafka3:9092"
       ],
       "authentication": {
         "type": "anonymous"
       },
       "tls": {
         "enabled": false
       },
       "insecure_allow_anonymous_virtual_cluster_auth": true
     }' | jq -r ".id"
)

Create a virtual cluster with topic aliases

Create a virtual cluster that maps friendly alias names to the backend topics:

VC_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "aliased_cluster",
       "dns_label": "aliased",
       "destination": {
         "id": "'$BACKEND_CLUSTER_ID'"
       },
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "acl_mode": "passthrough",
       "topic_aliases": [
         {
           "alias": "orders",
           "topic": "team-alpha-orders-v2"
         },
         {
           "alias": "clicks",
           "topic": "analytics-raw-clicks"
         }
       ]
     }' | jq -r ".id"
)

Clients connecting to this virtual cluster will see orders and clicks as topic names. The original backend names (team-alpha-orders-v2, analytics-raw-clicks) also remain accessible.

Create a listener

Create a listener with a port forwarding policy to route traffic to the virtual cluster:

LISTENER_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/listeners" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "alias_listener",
       "addresses": [
         "0.0.0.0"
       ],
       "ports": [
         "19092-19095"
       ]
     }' | jq -r ".id"
)

Create the port mapping policy:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/listeners/$LISTENER_ID/policies" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "type": "forward_to_virtual_cluster",
       "name": "forward_to_aliased_cluster",
       "config": {
         "type": "port_mapping",
         "advertised_host": "localhost",
         "destination": {
           "id": "'$VC_ID'"
         }
       }
     }'

Add a virtual cluster context to kafkactl

Extend kafkactl.yaml with a vc context that points at the listener you just created:

cat <<EOF > kafkactl.yaml
contexts:
  direct:
    brokers:
      - localhost:9095
      - localhost:9096
      - localhost:9094
  vc:
    brokers:
      - localhost:19092
EOF

Validate

List topics through the virtual cluster 

kafkactl -C kafkactl.yaml --context vc list topics

You should see both the aliases and the original backend topic names:

TOPIC                    PARTITIONS     REPLICATION FACTOR
analytics-raw-clicks     1              1
clicks                   1              1
orders                   1              1
team-alpha-orders-v2     1              1

Produce via alias, consume from backend

Produce a message using the alias name orders:

kafkactl -C kafkactl.yaml --context vc produce orders --value='{"id": 123, "item": "widget"}'

Consume from the backend topic team-alpha-orders-v2 directly to verify the message arrived:

kafkactl -C kafkactl.yaml --context direct consume team-alpha-orders-v2 --from-beginning --exit

You should see:

{"id": 123, "item": "widget"}

The message produced to the alias orders is stored in the backend topic team-alpha-orders-v2.

Something wrong?
Report an Issue | Edit this Page
View all Kong documentation

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!
Contribute

Still need help?

Ask in our Slack Community
Contact Support