Route requests to backup Targets during failures

In this tutorial, we’re going to have two primary Targets and one failover Target. In this configuration, traffic will be routed to only the primary Targets if they are both healthy. If they are both unhealthy, traffic will be routed to the failover Target.

We’ll use three backends to represent our three Targets using Docker’s ultra-small hashicorp/http-echo image:

docker run -d --rm --name primary1 -p 9001:5678 hashicorp/http-echo -text "PRIMARY-1"
docker run -d --rm --name primary2 -p 9002:5678 hashicorp/http-echo -text "PRIMARY-2"
docker run -d --rm --name failover -p 9003:5678 hashicorp/http-echo -text "FAILOVER"

Configure a Gateway Service and Route to point to the Upstream:

echo '
_format_version: "3.0"
upstreams:
  - name: example-upstream
services:
  - name: example-service
    host: example-upstream
    protocol: http
routes:
  - name: example-route
    paths:
    - "/anything"
    service:
      name: example-service
' | deck gateway apply -

Now, you can configure Upstream with the two primary Targets failover: false and one failover Target failover: true.

1. Configure the first primary Target:

    curl -X POST "http://localhost:8001/upstreams/example-upstream/targets/" \
        -H "Accept: application/json" \
        --json '{
          "target": "host.docker.internal:9001",
          "weight": 100,
          "failover": false
        }'
   

   Copied!

2. Configure the second primary Target:

    curl -X POST "http://localhost:8001/upstreams/example-upstream/targets/" \
        -H "Accept: application/json" \
        --json '{
          "target": "host.docker.internal:9002",
          "weight": 100,
          "failover": false
        }'
   

   Copied!

3. Configure the failover Target:

    curl -X POST "http://localhost:8001/upstreams/example-upstream/targets/" \
        -H "Accept: application/json" \
        --json '{
          "target": "host.docker.internal:9003",
          "weight": 50,
          "failover": true
        }'
   

   Copied!

Run the following to verify that only the primary Targets handle traffic because they are both healthy:

for i in {1..10}; do curl -sS http://localhost:8000/anything; echo; done

You’ll get an output like the following, where you can see the Targets cycling between PRIMARY-1 and PRIMARY-2:

PRIMARY-2

PRIMARY-1

PRIMARY-1

PRIMARY-2

PRIMARY-2

PRIMARY-2

To validate that the failover Target works, let’s mark the primary Targets as unhealthy by shutting down the hosts. Run the following:

docker stop primary1 primary2

You can now validate that since the primary Targets are both unhealthy, only the failover Target routes traffic:

for i in {1..6}; do curl -s http://localhost:8000/anything; echo; done

This time, the response should show the FAILOVER Target instead of PRIMARY.