Transparent proxying with Kong Mesh

A transparent proxy is a type of server that can intercept network traffic to and from a service without changes to the client application code. In the case of Kong Mesh, it is used to capture traffic and redirect it to a data plane to apply policies.

Kong Mesh uses iptables and offers additional, experimental support for eBPF.

Here’s a high level visualization of how transparent proxying works:

 
 sequenceDiagram
 autonumber
     participant Browser as client
(Mobile app)
     participant Kernel as Kernel
     participant ServiceMeshIn as kuma sidecar(15006)
     participant Node as example.com:5000
(Front-end app)
     participant ServiceMeshOut as kuma sidecar(15001)
     Browser->>+Kernel: GET / HTTP1.1
Host: example.com:5000
 
     rect rgb(233,233,233)
     Note over Kernel,ServiceMeshOut: EXAMPLE.COM
     Note over Node: (Optional)
 Apply inbound policies
     Note over ServiceMeshOut: (Optional)
 Apply inbound policies
     Kernel->>+ServiceMeshIn: Capture inbound TCP traffic
and redirect to the sidecar
 (listener port 15006)
     ServiceMeshIn->>+Node: Redirect to the
original destination 
(example.com:5000)
         Node->>+Kernel: Send the 
front-end response
     Kernel->>+ServiceMeshOut: Capture outbound TCP traffic
and Redirect to the sidecar
 (listener port 15001)
     end
     ServiceMeshOut->>+Browser: Response to client
  

If you choose to not use transparent proxying, or if you’re running on a platform where transparent proxying isn’t available, there are some additional considerations:

• You need to specify the inbound and outbound ports that you want to capture traffic on.
• .mesh addresses are unavailable.
• You may need to update your application code to use the new capture ports.
• VirtualOutbound is unsupported.

Without manipulating iptables to redirect traffic, you will need to explicitly tell kuma-dp where to listen to capture it. This can require changes to your application code.

The example below specifies that kuma-dp will listen on the address 10.119.249.39:15000. This in turn creates an Envoy listener for the port. When consuming a service over this address, it will cause traffic to redirect to 127.0.0.1:5000 where our app is running:

  type: Dataplane
  mesh: default
  name: demo-app
  networking: 
    address: 10.119.249.39 
    inbound: 
      - port: 15000
        servicePort: 5000
        serviceAddress: 127.0.0.1
        tags: 
          kuma.io/service: app
          kuma.io/protocol: http

On Kubernetes, kuma-dp leverages transparent proxying automatically via iptables installed with the kuma-init container or the CNI. All incoming and outgoing traffic is automatically intercepted by kuma-dp without having to change the application code.

Kong Mesh integrates with the service naming provided by Kubernetes DNS, and also provides its own Kong Mesh DNS for multi-zone service naming.

On Universal, kuma-dp leverages the data plane proxy specification for receiving incoming requests on a pre-defined port.

To enable transparent proxying, the zone control plane must exist on a separate server. If transparent proxying is on the same machine, you can’t run the zone control plane with PostgreSQL.

There are several advantages when using transparent proxying in Universal mode:

• The Dataplane resource is simpler because you can omit the outbound section.
• Universal service naming with the .mesh DNS domain instead of explicit outbounds like https://localhost:10001.
• Better service manageability (for example: security and tracing).

If you run firewalld to manage firewalls and wrap iptables, add the --store-firewalld flag to kumactl install transparent-proxy. This persists the relevant rules across host restarts. The changes are stored in /etc/firewalld/direct.xml. There is no uninstall command for this feature.

Configuring transparent proxying on Universal

To configure transparent proxying in Universal mode, you must first:

• Install kuma-dp, envoy, and coredns on the node that runs your service mesh workload.
• Set coredns in the path so that kuma-dp can access it. You can also set the location with the --dns-coredns-path flag in the kuma-dp command.

Configuring the service host

Kong Mesh comes with the kumactl executable, which can help you prepare the host for transparent proxying.

Due to the wide variety of Linux setup options, these steps may vary and may need to be adjusted for the specifics of the particular deployment.

The host that will run the kuma-dp process in transparent proxying mode needs to be prepared with the following steps, executed as root:

1. Create a new dedicated user on the machine.

   useradd -u 5678 -U kuma-dp
   

   Copied!

2. Redirect all the relevant inbound, outbound, and DNS traffic to the Kong Mesh data plane proxy:

   kumactl install transparent-proxy \
     --kuma-dp-user kuma-dp \
     --redirect-dns \
     --exclude-inbound-ports 22
   

   Copied!

   If you’re running any other services on that machine, adjust the comma separated lists of --exclude-inbound-ports and --exclude-outbound-ports accordingly.

   This command will change the host’s iptables rules.

   The changes won’t persist over restarts. You must either add this command to your start scripts or use firewalld.

Configuring the Dataplane resource

In transparent proxying mode, the Dataplane resource should omit the networking.outbound section and use networking.transparentProxying section instead:

type: Dataplane
mesh: default
name: {{ name }}
networking:
  address: {{ address }}
  inbound:
  - port: {{ port }}
    tags:
      kuma.io/service: demo-client
  transparentProxying:
    redirectPortInbound: 15006
    redirectPortOutbound: 15001

The ports used above are the default ones that kumactl install transparent-proxy will set. These can be changed using the relevant flags to that command.

Invoking the Kong Mesh data plane

It’s’ important that the kuma-dp process runs with the same system user that was passed to kumactl install transparent-proxy --kuma-dp-user. The service itself should run with any other user than kuma-dp. Otherwise, it won’t be able to leverage transparent proxying.

When using systemd, you can invoke the data plane with a User=kuma-dp entry in the [Service] section of the service file.

When starting kuma-dp with a script or some other automation, you can use runuser:

runuser -u kuma-dp -- \
  /usr/bin/kuma-dp run \
    --cp-address=https://$CONTROL_PLANE_HOST:5678 \
    --dataplane-token-file=$TOKEN_FILEPATH \
    --dataplane-file=$DATAPLANE_CONFIG_FILE \
    --dataplane-var name=dp-demo \
    --dataplane-var address=$VM_IP \
    --dataplane-var port=$SERVICE_PORT  \
    --binary-path /usr/local/bin/envoy

Once this is configured, you’ll be able to reach the Service on the same IP and port as before installing transparent proxy, but the traffic will go through Envoy. You can also connect to Services using Kong Mesh DNS.

Upgrades

Before upgrading to the next version of Kong Mesh, we recommend uninstalling the transparent proxy before replacing the kumactl binary:

kumactl uninstall transparent-proxy

The following sections provide transparent proxy configuration examples.

Intercepted traffic

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
  namespace: kuma-example
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        # all incoming connections on ports 1234 won't be intercepted by Envoy
        traffic.kuma.io/exclude-inbound-ports: "1234"
        # all outgoing connections on ports 5678, 8900 won't be intercepted by Envoy
        traffic.kuma.io/exclude-outbound-ports: "5678,8900"
    spec:
      containers:
        ...

KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_INBOUND_PORTS=1234
KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_OUTBOUND_PORTS=5678,8900

Reachable Services

By default, every data plane proxy in the mesh follows every other data plane proxy. This may lead to performance problems in larger deployments of the mesh. We recommend defining a list of Services that your Service connects to.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
  namespace: kuma-example
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        # a comma separated list of kuma.io/service values
        kuma.io/transparent-proxying-reachable-services: "redis_kong-mesh-demo_svc_6379,elastic_kong-mesh-demo_svc_9200"
    spec:
      containers:
        ...

type: Dataplane
mesh: default
name: {{ name }}
networking:
  address: {{ address }}
  inbound:
  - port: {{ port }}
    tags:
      kuma.io/service: demo-client
  transparentProxying:
    redirectPortInbound: 15006
    redirectPortOutbound: 15001
    reachableServices:
      - redis_kong-mesh-demo_svc_6379
      - elastic_kong-mesh-demo_svc_9200 

Reachable backends v2.9+

The reachable backends feature provides similar functionality to reachable services, but it applies to the MeshService, MeshExternalService, and MeshMultiZoneService resources.

This feature works only when a MeshService is enabled.

By default, every data plane proxy in the mesh tracks every other data plane proxy. Configuring reachableBackends can improve performance and reduce resource use.

Unlike reachable services, the model for providing data in reachable backends is more structured.

Model

Reachable backends are configured using the following parameters:

• refs: A list of all resources your application should track and communicate with.
  • kind: The type of resource. Possible values include:
    • MeshService
    • MeshExternalService
    • MeshMultiZoneService
  • name: The name of the resource.
  • namespace: The namespace where the resource is located. When this is defined, the name is required. This parameter is only used on Kubernetes.
  • labels: A list of labels to match the resources. Either labels or name can be defined.
  • port: (Optional) The port of the Service you want to communicate with. This works with MeshService and MeshMultiZoneService

apiVersion: apps/v1
kind: Deployment
metadata:
...
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        kuma.io/reachable-backends: |
          refs:
          - kind: MeshService
            name: redis
            namespace: kong-mesh-demo
            port: 8080
          - kind: MeshMulitZoneService
            labels:
              kuma.io/display-name: test-server
          - kind: MeshExternalService
            name: mes-http
            namespace: kuma-system

type: Dataplane
mesh: default
name: {{ name }}
networking:
  ...
  transparentProxying:
    redirectPortInbound: 15006
    redirectPortOutbound: 15001
    reachableBackends:
      refs:
      - kind: MeshService
        name: redis
      - kind: MeshMulitZoneService
        labels:
          kuma.io/display-name: test-server
      - kind: MeshExternalService
        name: mes-http

Examples

Here are some reachable backend configuration examples.

The following configures a demo-app that communicates only with redis on port 6379:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: kong-mesh-demo
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        kuma.io/reachable-backends: |
          refs:
          - kind: MeshService
            name: redis
            namespace: kong-mesh-demo
            port: 6379
    spec:
      containers:
        ...

type: Dataplane
mesh: default
name: {{ name }}
networking:
  address: {{ address }}
  inbound:
  - port: {{ port }}
    tags:
      kuma.io/service: demo-app
  transparentProxying:
    redirectPortInbound: 15006
    redirectPortOutbound: 15001
    reachableBackends:
      refs:
      - kind: MeshService
        name: redis
        port: 6379

The following configures a demo-app that doesn’t need to communicate with any Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: kong-mesh-demo
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        kuma.io/reachable-backends: ""
    spec:
      containers:
        ...

type: Dataplane
mesh: default
name: {{ name }}
networking:
  address: {{ address }}
  inbound:
  - port: {{ port }}
    tags:
      kuma.io/service: demo-app
  transparentProxying:
    redirectPortInbound: 15006
    redirectPortOutbound: 15001
    reachableBackends: {}

The following configures a demo-app that communicates with all MeshServices in the kong-mesh-demo namespace:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: kong-mesh-demo
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        kuma.io/reachable-backends: |
          refs:
          - kind: MeshService
            labels:
              k8s.kuma.io/namespace: kong-mesh-demo
    spec:
      containers:
        ...

Transparent Proxy with eBPF (experimental)

In Kong Mesh 2.0 or later, you can set up transparent proxy to use eBPF instead of iptables.

To use transparent proxying with eBPF, your environment must use Kernel version 5.7 or later and have cgroup2 available.

kumactl install control-plane \
  --set "kuma.experimental.ebpf.enabled=true" | kubectl apply -f-

kumactl install transparent-proxy \
  --experimental-transparent-proxy-engine \
  --ebpf-enabled \
  --ebpf-instance-ip $IP_ADDRESS \
  --ebpf-programs-source-path $PATH