Backend clusters

Uses: Event Gateway kongctl Konnect API Terraform

What is a backend cluster?

A backend cluster is an abstraction of a real Kafka cluster. It stores the connection and configuration details required for Kong Event Gateway to proxy traffic to Kafka.

Multiple Kafka clusters can be proxied through a single Kong Event Gateway. The Event Gateway control plane manages information such as:

Authentication credentials for connecting to Kafka clusters
TLS verification preferences
Metadata refresh intervals for fetching cluster information

 
flowchart LR
    
    A[Kafka 
    client]
    B[Listener 
    &#40TCP socket&#41
    + listener policies]
    C@{ shape: processes, label: "Virtual clusters
    + consume, produce, 
    and cluster policies"}
    D[Backend 
    cluster]
    E[Kafka 
    cluster]

    A --> B
    subgraph id1 [Event Gateway]
    B --> C 
    C --> D
    end
    D --> E

    style D stroke:#86e2cc

    style id1 rx:7,ry:7

Authentication on the backend cluster defines the credentials the Event Gateway uses when connecting to Kafka to fetch cluster metadata, such as brokers, topic and partition information. The supported types reflect what your Kafka cluster accepts.

Note: These credentials aren’t used to represent individual client actions. For passthrough and validate_forward mediation, each client authenticates directly to Kafka using their own credentials for their connection.

Backend clusters support the following auth methods:

Auth method (`authentication.type`)	Description	Credential mediation types (`authentication.mediation`)
Anonymous	Doesn’t require clients to provide any authentication when connecting to the proxy.	None
SASL/PLAIN	Requires clients to provide a username and password. Accepts a hardcoded list of usernames and passwords, either as strings or environment variables. See Authenticate Event Gateway connections to Kafka using SASL/PLAIN.	`passthrough`, `terminate`
SASL/SCRAM-SHA-256	Requires clients to provide a username and password using SCRAM-SHA-256 hashing.	`passthrough`
SASL/SCRAM-SHA-512	Requires clients to provide a username and password using SCRAM-SHA-512 hashing.	`passthrough`

Depending on what your Kafka cluster supports, you’ll need to configure authentication on the associated virtual cluster:

If your Kafka cluster only accepts SASL/PLAIN or SASL/SCRAM credentials, use passthrough or terminate mediation on the virtual cluster.
If your backend Kafka cluster supports SASL/OAUTHBEARER natively, use passthrough or validate_forward mediation on the virtual cluster.

Set up a backend cluster

To create a backend cluster, call the Event Gateway API’s /backend-clusters endpoint.

curl -X POST https://{region}.api.konghq.com/v1/event-gateways/{eventGatewayId}/backend-clusters \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "example-backend-cluster",
      "bootstrap_servers": [
        "host:9092"
      ],
      "authentication": {
        "type": "anonymous"
      },
      "insecure_allow_anonymous_virtual_cluster_auth": true,
      "tls": {
        "insecure_skip_verify": false
      }
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
virtualClusterId: The id of the Virtual Cluster.
eventGatewayId: The id of the Event Gateway.
eventGatewayListenerId: The id of the Event Gateway Listener.

See the Konnect Event Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect-beta = {
      source  = "kong/konnect-beta"
    }
  }
}

provider "konnect-beta" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_event_gateway_backend_cluster" "my_backend_cluster" {
  provider = konnect-beta
  bootstrap_servers = ["host:9092"]
  authentication = {
    type = "anonymous"
  }
  insecure_allow_anonymous_virtual_cluster_auth = true
  tls = {
    insecure_skip_verify = false
  }



  gateway_id = konnect_event_gateway.my_event_gateway.id
}

Copied!

The following creates a new backend cluster called example-backend-cluster.

Add this snippet to an event_gateways resource in your declarative configuration file, and then manage it with kongctl:

backend_cluster.yaml

Copied!

event_gateways:
  - ref: eventGatewayName
    name: eventGatewayName
    backend_clusters:
    - ref: example-backend-cluster
      name: example-backend-cluster
      bootstrap_servers:
      - host:9092
      authentication:
        type: anonymous
      insecure_allow_anonymous_virtual_cluster_auth: true
      tls:
        insecure_skip_verify: false

Make sure to replace the following placeholders with your own values:

eventGatewayName: The name of your Event Gateway.

The following creates a new backend cluster called example-backend-cluster with basic configuration:

In Konnect, navigate to Event Gateway in the sidebar.
Click an Event Gateway.
Click the Backend Clusters tab.
Click New backend cluster.
In the Name field, enter example-backend-cluster.
In the Bootstrap servers field, enter host:9092.
From the Authentication dropdown menu, select anonymous
Select the Allow anonymous authentication from virtual clusters checkbox.
Click Save.

Backend clusters

What is a backend cluster?

Authentication

Set up a backend cluster

Schema

Help us make these docs great!

Still need help?