Transparent proxying with Kong Mesh

Uses: Kong Mesh
Related Documentation

A transparent proxy is a type of server that can intercept network traffic to and from a service without changes to the client application code. In the case of Kong Mesh, it is used to capture traffic and redirect it to a data plane to apply policies.

Kong Mesh uses iptables and offers additional, experimental support for eBPF.

Here’s a high level visualization of how transparent proxying works:

 
 sequenceDiagram
 autonumber
     participant Browser as client
(Mobile app) participant Kernel as Kernel participant ServiceMeshIn as kuma sidecar(15006) participant Node as example.com:5000
(Front-end app) participant ServiceMeshOut as kuma sidecar(15001) Browser->>+Kernel: GET / HTTP1.1
Host: example.com:5000 rect rgb(233,233,233) Note over Kernel,ServiceMeshOut: EXAMPLE.COM Note over Node: (Optional)
Apply inbound policies Note over ServiceMeshOut: (Optional)
Apply inbound policies Kernel->>+ServiceMeshIn: Capture inbound TCP traffic
and redirect to the sidecar
(listener port 15006) ServiceMeshIn->>+Node: Redirect to the
original destination
(example.com:5000) Node->>+Kernel: Send the
front-end response Kernel->>+ServiceMeshOut: Capture outbound TCP traffic
and Redirect to the sidecar
(listener port 15001) end ServiceMeshOut->>+Browser: Response to client

If you choose to not use transparent proxying, or if you’re running on a platform where transparent proxying isn’t available, there are some additional considerations:

  • You need to specify the inbound and outbound ports that you want to capture traffic on.
  • .mesh addresses are unavailable.
  • You may need to update your application code to use the new capture ports.
  • VirtualOutbound is unsupported.

Without manipulating iptables to redirect traffic, you will need to explicitly tell kuma-dp where to listen to capture it. This can require changes to your application code.

The example below specifies that kuma-dp will listen on the address 10.119.249.39:15000. This in turn creates an Envoy listener for the port. When consuming a service over this address, it will cause traffic to redirect to 127.0.0.1:5000 where our app is running:

  type: Dataplane
  mesh: default
  name: demo-app
  networking: 
    address: 10.119.249.39 
    inbound: 
      - port: 15000
        servicePort: 5000
        serviceAddress: 127.0.0.1
        tags: 
          kuma.io/service: app
          kuma.io/protocol: http

Kubernetes

On Kubernetes, kuma-dp leverages transparent proxying automatically via iptables installed with the kuma-init container or the CNI. All incoming and outgoing traffic is automatically intercepted by kuma-dp without having to change the application code.

Kong Mesh integrates with the service naming provided by Kubernetes DNS, and also provides its own Kong Mesh DNS for multi-zone service naming.

Universal

On Universal, kuma-dp leverages the data plane proxy specification for receiving incoming requests on a pre-defined port.

To enable transparent proxying, the zone control plane must exist on a separate server. If transparent proxying is on the same machine, you can’t run the zone control plane with PostgreSQL.

There are several advantages when using transparent proxying in Universal mode:

  • The Dataplane resource is simpler because you can omit the outbound section.
  • Universal service naming with the .mesh DNS domain instead of explicit outbounds like https://localhost:10001.
  • Better service manageability (for example: security and tracing).

If you run firewalld to manage firewalls and wrap iptables, add the --store-firewalld flag to kumactl install transparent-proxy. This persists the relevant rules across host restarts. The changes are stored in /etc/firewalld/direct.xml. There is no uninstall command for this feature.

Configuring transparent proxying on Universal

To configure transparent proxying in Universal mode, you must first:

  • Install kuma-dp, envoy, and coredns on the node that runs your service mesh workload.
  • Set coredns in the path so that kuma-dp can access it. You can also set the location with the --dns-coredns-path flag in the kuma-dp command.

Configuring the service host

Kong Mesh comes with the kumactl executable, which can help you prepare the host for transparent proxying.

Due to the wide variety of Linux setup options, these steps may vary and may need to be adjusted for the specifics of the particular deployment.

The host that will run the kuma-dp process in transparent proxying mode needs to be prepared with the following steps, executed as root:

  1. Create a new dedicated user on the machine.

    useradd -u 5678 -U kuma-dp
  2. Redirect all the relevant inbound, outbound, and DNS traffic to the Kong Mesh data plane proxy:

    kumactl install transparent-proxy \
      --kuma-dp-user kuma-dp \
      --redirect-dns \
      --exclude-inbound-ports 22

    If you’re running any other services on that machine, adjust the comma separated lists of --exclude-inbound-ports and --exclude-outbound-ports accordingly.

    This command will change the host’s iptables rules.

    The changes won’t persist over restarts. You must either add this command to your start scripts or use firewalld.

Configuring the Dataplane resource

In transparent proxying mode, the Dataplane resource should omit the networking.outbound section and use networking.transparentProxying section instead:

type: Dataplane
mesh: default
name: {{ name }}
networking:
  address: {{ address }}
  inbound:
  - port: {{ port }}
    tags:
      kuma.io/service: demo-client
  transparentProxying:
    redirectPortInbound: 15006
    redirectPortOutbound: 15001

The ports used above are the default ones that kumactl install transparent-proxy will set. These can be changed using the relevant flags to that command.

Invoking the Kong Mesh data plane

It’s’ important that the kuma-dp process runs with the same system user that was passed to kumactl install transparent-proxy --kuma-dp-user. The service itself should run with any other user than kuma-dp. Otherwise, it won’t be able to leverage transparent proxying.

When using systemd, you can invoke the data plane with a User=kuma-dp entry in the [Service] section of the service file.

When starting kuma-dp with a script or some other automation, you can use runuser:

runuser -u kuma-dp -- \
  /usr/bin/kuma-dp run \
    --cp-address=https://$CONTROL_PLANE_HOST:5678 \
    --dataplane-token-file=$TOKEN_FILEPATH \
    --dataplane-file=$DATAPLANE_CONFIG_FILE \
    --dataplane-var name=dp-demo \
    --dataplane-var address=$VM_IP \
    --dataplane-var port=$SERVICE_PORT  \
    --binary-path /usr/local/bin/envoy

Once this is configured, you’ll be able to reach the Service on the same IP and port as before installing transparent proxy, but the traffic will go through Envoy. You can also connect to Services using Kong Mesh DNS.

Upgrades

The core iptables rules that Kong Mesh’s transparent proxy applies rarely change, but new features occasionally require updates.

Uninstall the transparent proxy

Before upgrading to the next version of Kong Mesh, we recommend uninstalling the transparent proxy before replacing the kumactl binary:

kumactl uninstall transparent-proxy

Clean up existing iptables rules manually

v2.9+ If you’re upgrading from Kong Mesh version 2.9 or later, and you have not manually disabled comments by setting comments.disabled to true in the transparent proxy configuration, this step is unnecessary.

Starting with Kong Mesh 2.9, the transparent proxy tags all iptables rules with comments so Kong Mesh can track rule ownership. kumactl uses these comments to automatically clean up rules and custom chains created by previous transparent proxy versions. The cleanup runs at the start of the installation, so no manual cleanup is needed.

To manually remove existing iptables rules, either restart the host (if the rules weren’t persisted with system start-up scripts or firewalld) or run the following commands.

These commands remove all iptables rules and all custom chains in the specified tables, including those created by Kong Mesh and any other applications or services.

iptables --table nat --flush         # Flush all rules in the NAT table (IPv4)
ip6tables --table nat --flush        # Flush all rules in the NAT table (IPv6)
iptables --table nat --delete-chain  # Delete all custom chains in the NAT table (IPv4)
ip6tables --table nat --delete-chain # Delete all custom chains in the NAT table (IPv6)

# The raw table contains rules for DNS traffic redirection
iptables --table raw --flush         # Flush all rules in the raw table (IPv4)
ip6tables --table raw --flush        # Flush all rules in the raw table (IPv6)

# The mangle table contains rules to drop invalid packets
iptables --table mangle --flush      # Flush all rules in the mangle table (IPv4)
ip6tables --table mangle --flush     # Flush all rules in the mangle table (IPv6)

Install the new transparent proxy

Reinstall the transparent proxy:

kumactl install transparent-proxy --kuma-dp-user kuma-dp --redirect-dns --verbose

The command installs the latest version of the transparent proxy with the specified configuration. Adjust the flags as needed for your environment.

Configuration

The following sections provide transparent proxy configuration examples.

Intercepted traffic

Reachable Services

By default, every data plane proxy in the mesh follows every other data plane proxy. This may lead to performance problems in larger deployments of the mesh. We recommend defining a list of Services that your Service connects to.

Reachable backends v2.9+

The reachable backends feature provides similar functionality to reachable services, but it applies to the MeshService, MeshExternalService, and MeshMultiZoneService resources.

This feature works only when a MeshService is enabled.

By default, every data plane proxy in the mesh tracks every other data plane proxy. Configuring reachableBackends can improve performance and reduce resource use.

Unlike reachable services, the model for providing data in reachable backends is more structured.

Model

Reachable backends are configured using the following parameters:

  • refs: A list of all resources your application should track and communicate with.
    • kind: The type of resource. Possible values include:
    • name: The name of the resource.
    • namespace: The namespace where the resource is located. When this is defined, the name is required. This parameter is only used on Kubernetes.
    • labels: A list of labels to match the resources. Either labels or name can be defined.
    • port: (Optional) The port of the Service you want to communicate with. This works with MeshService and MeshMultiZoneService

Examples

Here are some reachable backend configuration examples.

The following configures a demo-app that communicates only with redis on port 6379:

The following configures a demo-app that doesn’t need to communicate with any Service:

The following configures a demo-app that communicates with all MeshServices in the kong-mesh-demo namespace:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: kong-mesh-demo
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        kuma.io/reachable-backends: |
          refs:
          - kind: MeshService
            labels:
              k8s.kuma.io/namespace: kong-mesh-demo
    spec:
      containers:
        ...

Transparent Proxy with eBPF (experimental)

In Kong Mesh 2.0 or later, you can set up transparent proxy to use eBPF instead of iptables.

To use transparent proxying with eBPF, your environment must use Kernel version 5.7 or later and have cgroup2 available.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!