Virtual clusters

What is a virtual cluster?

Virtual clusters are the primary way clients interact with the Event Gateway proxy. They allow you to isolate clients from each other when connecting to the same backend cluster, and provide each client with modified view while still appearing as a standard Kafka cluster.

The virtual cluster workflow operates as follows:

  1. The Kafka client produces a request.
  2. A listener forwards it to the correct virtual cluster.
  3. The virtual cluster applies policies and proxies the modified request to the backend cluster.
  4. The backend cluster, representing a Kafka cluster, receives the request and sends a response.
 
flowchart LR
    
    A[Kafka 
    client]
    B[Listener 
    &#40TCP socket&#41
    + listener policies]
    C@{ shape: processes, label: "Virtual clusters
    + consume, produce, 
    and cluster policies"}
    D[Backend 
    cluster]
    E[Kafka 
    cluster]

    A --> B
    subgraph id1 [Event Gateway]
    B --> C 
    C --> D
    end
    D --> E

    style C stroke:#86e2cc

    style id1 rx:7,ry:7
  

Note: Each virtual cluster can only expose one backend cluster, but you can have multiple virtual clusters connected to one backend. In other words, a single virtual cluster can’t aggregate data from multiple backend clusters.

Why use a virtual cluster?

Virtual clusters let you apply governance and security features to event streams. This way, a single Kafka cluster can be sliced into multiple access points, each with its own security policy.

Use case

Description

Policy enforcement Define policies on virtual clusters to govern client behavior. Policies include transformations, filtering, enforcing encryption and decryption standards, access control, and more.
Authentication and mediation Use authentication mediation to control access between clients and backend clusters. Event Gateway authenticates clients (for example, with OAuth tokens) and re-authenticates separately when forwarding requests to the backend.
Topic and cluster virtualization Use topic and cluster virtualization to simplify change management and security. Virtual clusters can expose only a subset of topics on the backend cluster.
Namespacing and topic rewriting Virtual clusters support namespaces, which rewrite and enforce consistent prefixes for topic and consumer group names. This allows you to expose clean, simple names to clients while maintaining organization on the backend.
Infrastructure planning Through logical isolation, virtual clusters help organizations reduce Kafka infrastructure costs, as they eliminate the need to maintain multiple physical Kafka clusters for environment or team separation.

Managing multiple environments or products

You will need to increase the number of virtual clusters if you want to create multiple environments or products on top of the same physical cluster.

Here are some common patterns:

  • Environment isolation: You can create multiple isolated dev namespaces on top of the same physical Kafka cluster. For example, if you have a dev-1 and a dev-2 virtual cluster, a topic named orders in each one maps transparently to separate backend topics: dev-1-orders and dev-2-orders. This provides isolation and automatic name resolution per environment, so teams can share the same physical cluster without interfering with each other.

    When clients create new topics through a virtual cluster using Kafka’s CreateTopics request, the namespace prefix is automatically applied, ensuring that clients always stay within their designated namespace.

  • External partner isolation: You can expose the same backend topic to different external partners with data filtering. For instance, a single orders topic can be exposed through separate virtual clusters (customer-a, customer-b, customer-c), with each customer seeing only their own orders.

  • Reverse mapping: One backend topic (orders) can appear as multiple separate topics (dev-orders, test-orders, prod-orders) across different virtual clusters, each pre-filtered for specific users.

Authentication

Authentication on the virtual cluster is used to authenticate clients to the proxy. The virtual cluster supports multiple authentication methods and can mediate authentication between clients and backend clusters. See Backend cluster authentication to learn more.

Virtual clusters support the following auth methods:

Auth method (authentication.type)

Description

Credential mediation types (authentication.mediation)

Anonymous (anonymous) Doesn’t require clients to provide any authentication when connecting to the proxy.

See Get started with Event Gateway.
None
SASL/PLAIN (sasl_plain) Requires clients to provide a username and password.

Accepts a hardcoded list of usernames and passwords, either as strings or environment variables.
passthrough, terminate
SASL/OAUTHBEARER (oauth_bearer) Requires clients to provide an OAuth token and a JWKS endpoint to verify token signatures, optionally with claim mapping and validation rules.

See Set up Event Gateway with Kong Identity OAuth.
passthrough, terminate, validate_forward
SASL/SCRAM-SHA-256 (sasl_scram) Requires clients to provide a username and password using SCRAM-SHA-256 hashing. Set authentication.algorithm to sha256. passthrough
SASL/SCRAM-SHA-512 (sasl_scram) Requires clients to provide a username and password using SCRAM-SHA-512 hashing. Set authentication.algorithm to sha512. passthrough
mTLS (client_certificate) v1.1+ Requires clients to present a trusted TLS certificate during the TLS handshake. The client certificate is verified against a TLS trust bundle. The certificate’s principal (for example, the Common Name) can be used to enforce access control with ACL policies.

See Configure mTLS client authentication for a full walkthrough, including how to configure principal extraction.

mTLS is also supported for the gateway-to-backend connection; configure it in the backend cluster’s TLS settings.
None

Credential mediation

With virtual clusters, you can control how client credentials are handled between the proxy and backend cluster, and reuse existing credentials and principals defined on the backend cluster.

Use the virtual cluster authentication.mediation setting to configure a mediation mode. Choose the mode based on your security requirements and backend cluster configuration:

Mode

How it works

When to use

passthrough The client’s credentials pass through the proxy to the backend without validation. The backend cluster natively supports the same auth mechanism the client uses (for example, SASL/OAUTHBEARER or SASL/SCRAM). Required for SCRAM authentication.
terminate The proxy validates the client’s credential and terminates the auth session. It then opens a new, separate auth session with the backend using the credentials configured on the backend cluster entity.

When you use terminate mediation, the gateway-to-backend connection acts as a single service account, using the credentials configured on the backend cluster. The backend Kafka cluster sees only that service account, not individual client principals. Per-client authorization is enforced at the gateway layer using ACL policies on the virtual cluster.
The backend cluster only supports username/password mechanisms (SASL/PLAIN or SASL/SCRAM) and clients authenticate with a different mechanism, such as OAuth. We recommend using this pattern in production, as it’s the most secure option.
validate_forward The proxy validates the client’s OAuth token with the OAuth provider, then forwards the token to the backend as-is. The backend cluster natively supports SASL/OAUTHBEARER. Use this instead of passthrough to fail fast on invalid tokens before they reach the backend.

ACL mode

The acl_mode field controls whether ACL policies are evaluated. When set to enforce_on_gateway (the default), clients start with no access to any resources on the virtual cluster. Operators grant access by adding ACL policies with principal-specific conditions. When set to passthrough, ACL policies are not evaluated and all authenticated clients have full access to whatever the backend cluster exposes.

See ACL policy for details on defining access rules.

Enrich connections with caller metadata

After a client authenticates, you can configure the virtual cluster to look up the client in a Kong Identity directory and attach its metadata to the connection context. This metadata is available at context.auth.principal.metadata and can be used to drive policies automatically. For example, setting a header based on which team a client belongs to, or skipping records that a client isn’t entitled to see.

To enable this, add a fetch_kong_identity_principal block to the authentication config. For SASL/PLAIN connections, specify a fetch_by.key matching the identity key (for example, sasl_username). For OAuth Bearer connections, the lookup uses the JWT’s iss and sub claims automatically.

See the following how-to guides for end-to-end examples:

Namespaces

With namespaces, you can preserve any naming systems that you have in place, and ensure they remain consistent. Namespaces let you:

  • Rewrite and enforce topics, consumer groups, and transaction IDs with a consistent prefix
  • Expose topics and consumer groups through the virtual cluster

This allows you to expose clean, simple names to clients while maintaining organization on the backend.

For example, a virtual cluster exposes a topic named orders to the client. Behind the scenes, this maps to team-a-orders on the actual Kafka cluster. The client doesn’t need to know about or manage the team-a- prefix. This enables transparent multitenancy, where multiple teams can share the same Kafka cluster without needing to manually prefix every topic and consumer group name in their applications.

When clients send messages that contain Consumer Group names, the gateway passes the names to the backend unchanged, unless a namespace prefix is configured. Kafka handles consumer group coordination and offset tracking on the backend as normal. Consumer Group management commands such as create and delete are intercepted by the gateway and evaluated against ACL policies using the group resource type.

The following examples provide some common use cases for namespaces and show how to set them up.

Apply prefixes automatically

The most common use case for namespaces is to automatically prefix read and create operations when interacting with topics and consumer groups. This helps avoid overlapping from multiple tenants.

You can do this by setting a prefix on the virtual cluster:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example-virtual-cluster",
       "destination": {
         "name": "example-backend-cluster"
       },
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "dns_label": "virtual-cluster-1",
       "acl_mode": "passthrough",
       "namespace": {
         "mode": "hide_prefix",
         "prefix": "my-prefix"
       }
     }'

In this example, the prefix my-prefix will be used for all consumer group and topics that connect to this virtual cluster.

Access additional topics

Along with topics owned by a specific team, you can pull in a select group of additional topics. This is useful when you want to:

  • Consume topics owned by other teams
  • Gradually migrate to a namespace while still using old topics temporarily

Here’s an example configuration using a glob pattern:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example-virtual-cluster",
       "destination": {
         "name": "example-backend-cluster"
       },
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "dns_label": "virtual-cluster-1",
       "acl_mode": "passthrough",
       "namespace": {
         "mode": "hide_prefix",
         "prefix": "my-prefix",
         "additional": {
           "topics": [
             {
               "type": "glob",
               "glob": "my-topic-*",
               "conflict": "warn"
             }
           ]
         }
       }
     }'

These topics are accessed using their full unmodified names.

This example uses a glob expression to capture topics using name patterns. You can also pass an exact list of topics as an array:

"topics": [
  {
    "type": "exact_list",
    "list": [
      {
        "backend": "allowed_topic",
        "backend": "another_allowed_topic"
      }
    ]
  }
]

Access additional consumer groups

You can access existing consumer groups to avoid migrating offsets:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example-virtual-cluster",
       "destination": {
         "name": "example-backend-cluster"
       },
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "dns_label": "virtual-cluster-1",
       "acl_mode": "passthrough",
       "namespace": {
         "mode": "hide_prefix",
         "prefix": "my-prefix",
         "additional": {
           "consumer_groups": [
             {
               "type": "glob",
               "glob": "my-app-*",
               "conflict": "warn"
             }
           ]
         }
       }
     }'

End users of this virtual cluster can use their existing, unnamespaced consumer groups.

This example uses a glob expression to capture consumer groups using name patterns. You can also pass an exact list of consumer groups as an array:

"consumer_groups": [
  {
    "type": "exact_list",
    "list": [
      {
        "value": "foo",
        "value": "bar"
      }
    ]
  }
]

Topic aliases v1.2+

Topic aliases let you expose a backend Kafka topic under a different, client-facing name. Clients connecting to the virtual cluster see the alias, while Event Gateway transparently routes requests to the original backend topic.

This is useful when you want to:

  • Expose internally-named topics (like team-alpha-orders-v2) under client-friendly names (like orders)
  • Migrate clients to renamed backend topics without updating client configuration
  • Provide multiple client-facing names for the same backend topic

Unlike namespaces, which apply a prefix to every topic and consumer group, topic aliases are explicit one-to-one mappings between an alias and a backend topic. Both the alias and the original backend topic name remain accessible through the virtual cluster.

Map a friendly name to a backend topic

Define topic aliases on the virtual cluster using the topic_aliases field:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example-virtual-cluster",
       "destination": {
         "name": "example-backend-cluster"
       },
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "dns_label": "virtual-cluster-1",
       "acl_mode": "passthrough",
       "topic_aliases": [
         {
           "alias": "orders",
           "topic": "team-alpha-orders-v2"
         },
         {
           "alias": "clicks",
           "topic": "analytics-raw-clicks"
         }
       ]
     }'

In this example, clients can produce and consume to orders and clicks. Event Gateway resolves these to the backend topics team-alpha-orders-v2 and analytics-raw-clicks.

Supported operations

Aliases are a read-only abstraction over physical topics. Read and write operations are forwarded to the backend topic, but topic-modifying operations are rejected.

Operation

Behavior

Produce, fetch, list offsets, consumer group operations Allowed. Requests reference the alias name and are transparently resolved to the backend topic.
Metadata (ListTopics) Allowed. Both the alias and the original backend topic name appear in the response.
CreateTopics, DeleteTopics, CreatePartitions, DeleteRecords, AlterPartitionReassignments, ElectLeaders Rejected with InvalidTopicException when the request references an alias. This avoids a client unintentionally modifying a physical topic that other aliases or clients depend on.

ACLs are evaluated on the name the client uses, before alias resolution. An ACL on the backend topic does not automatically grant access to its aliases, and vice versa. Operators must configure ACLs for each alias name explicitly. With acl_mode: enforce_on_gateway (deny-by-default), a new alias with no matching ACL is blocked.

Interaction with namespaces

The topic field in topic_aliases references namespace-visible names, not raw backend topic names. In the request path, aliases resolve after policies but before the namespace transformer:

client → policies → alias resolution → namespace → backend

If the virtual cluster has a namespace, the alias must target a name that the namespace already exposes. For example, with namespace.mode: hide_prefix and prefix: analytics_, topic: foo resolves to the backend topic analytics_foo.

A backend topic that the namespace doesn’t expose (not matching the prefix and not listed in additional_topics) can’t be aliased directly. Expose it through the namespace first:

namespace:
  mode: hide_prefix
  prefix: analytics_
  additional_topics:
    - type: exact_list
      list:
        - team-alpha-orders-v2
topic_aliases:
  - alias: orders
    topic: team-alpha-orders-v2 # valid because additional_topics exposes it

Policy context

Policies attached to a virtual cluster see the alias name, not the backend topic name. In a CEL match expression, topic.name evaluates to the alias name. Policies that match on a backend topic name don’t automatically apply to its aliases. Match the alias name explicitly:

condition: topic.name == "orders"

Consumer groups

Avoid consuming from both an alias and its backend topic with the same consumer group. Event Gateway treats the alias and the backend topic as distinct topic names that share the same underlying partitions. Mixing them in one consumer group leaves offset commits and partition assignment undefined. Pick one name per consumer group and stick with it.

For a full walkthrough, see Configure topic aliases.

Virtual cluster policies

Virtual clusters can be modified by policies, which let you do things like modify headers, encrypt and decrypt records, validate record schemas, and much more.

To learn more, see:

Set up a virtual cluster

Before setting up a virtual cluster, make sure you have a backend cluster configured. A virtual cluster must connect to an existing backend cluster.

Schema

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!