Open in app

Sign in

Write

Sign in

Load Balance a Consul Cluster using Docker-Compose, NGINX, and Consul Template

Rick Kemery
8 min readOct 27, 2020

In this article, I will go over how to deploy a consul cluster with docker compose along with nginx using consul template to automatically inject consul backend server addresses into the nginx.conf and load balance the consul docker containers.

Prerequisites

The following article uses an Ubuntu 20.04.1 LTS virtual machine with docker-ce and docker-compose installed.

Install docker on Ubuntu: https://docs.docker.com/engine/install/ubuntu/

Install docker-compose following the Linux instructions: https://docs.docker.com/compose/install/

Add your current user to the docker group: sudo usermod -aG docker $USER

DNS A Record

I created a DNS A record to point back to the virtual machine IP so I would be able to navigate to my consul cluster when started.

TLS Certificates

This article also calls for TLS certificates. I used my own Certificate Authority to create a wildcard certificate also adding *.consul_datacenter_identifier.consul as a FQDN to the SAN field of the cert.

For example, if you set your consul datacenter id to “test” you would add *.test.consul in the SAN.

│   └── tls
│       ├── ca.crt
│       ├── consul.crt
│       └── consul.key

Build NGINX Container

The first step is to build a docker container with NGINX and consul-template. I am using the latest version of consul and consul-template but you could also opt to automate the retrieval of the latest version on each docker build.

Build the docker image with: docker build -t <name>-nginx .

Dockerfile

We need consul-template and consul-agent installed on nginx.

FROM nginx:latest# Tell apt-get we're never going to be able to give manual feedback
RUN export DEBIAN_FRONTEND=noninteractive# Download latest listing of available packages
RUN apt-get -y update# Upgrade already installed packages
RUN apt-get -y upgrade# Install wget
RUN apt install -y wget# Install unzip
RUN apt install -y unzip# Download consul-template
RUN wget https://releases.hashicorp.com/consul-template/0.25.1/consul-template_0.25.1_linux_amd64.zip -P /tmp# Install consul-template
RUN unzip /tmp/consul-template_0.25.1_linux_amd64.zip -d /tmp && mv /tmp/consul-template /usr/local/bin/consul-template# Download consul-agent
RUN wget https://releases.hashicorp.com/consul/1.8.5/consul_1.8.5_linux_amd64.zip -P /tmp# Install consul-agent
RUN unzip /tmp/consul_1.8.5_linux_amd64.zip -d /tmp && mv /tmp/consul /usr/local/bin/consul# Start nginx
CMD service nginx start ; while true ; do sleep 100; done;

Initial Setup

To-do: Make a directory that will house the docker-compose and other files. Refer to the directory tree structure below. I created each directory under /opt and then docker volume mounted in the docker compose file.

Docker-Compose Overview

Image: I am using the latest image of consul.

Ports: For each consul node, I opted to expose the consul http/https/dns ports to the host. http is 8500, https is 8501, and dns is 8600 on the container.

Networks: I created a docker network definition and assigned each container a docker container ipv4 addr.

Command: Each consul-server calls the agent and starts the server based on the appropriate server config these are consul-config-1, 2 and 3 respectively. The first consul-server container has “bootstrap_expect”: 3 to indicate the number of consul members and to init a leader on start.

Volumes: Each docker-compose entity for consul-server has its own config and data folder under /opt as well as the tls certificate folder is mounted on each container.

— /opt/consul-server/config/config-#:/opt/consul-server/config
— /opt/consul-server/data-#:/opt/consul-server/data
— /opt/consul-server/tls:/opt/consul-server/tls

For nginx, I specified the image name I created from the custom nginx I built earlier. The command section will start nginx first, remove the initial nginx.conf file, start consul agent and join the cluster, then use consul-template to input the values from the consul-template config file into the nginx.conf, write the nginx.conf and reload nginx configuration.

command: 
      - /bin/bash
      - -c
      - |
        /etc/init.d/nginx start
        /bin/rm /etc/nginx/nginx.conf
        /usr/local/bin/consul agent -data-dir=/opt/consul-agent/nginx/data -config-dir=/opt/consul-agent/nginx/config -config-file=/opt/consul-agent/nginx/config/consul-config.json &
        /usr/local/bin/consul-template -config=/opt/consul-agent/nginx/template/consul-template-config.hcl
    networks:
      consul:
        ipv4_address: 192.168.172.14
    volumes:
      - /opt/consul-server/tls:/etc/nginx/tls
      - /opt/nginx/nginx.ctmpl:/etc/nginx/nginx.ctmpl
      - /opt/consul-agent/nginx/data:/opt/consul-agent/nginx/data
      - /opt/consul-agent/nginx/config:/opt/consul-agent/nginx/config
      - /opt/consul-agent/nginx/template:/opt/consul-agent/nginx/template

Consul Template Config: consul-template-config.hcl

When nginx is started with consul agent, it will read this file the directs it where to find the consul API endpoint at :8500, retry stanza, and where the source and destination template files are located along with the command to run after the file is templated.

consul {
  address = "localhost:8500"retry {
    enabled  = true
    attempts = 12
    backoff  = "250ms"
  }
}
template {
  source      = "/etc/nginx/nginx.ctmpl"
  destination = "/etc/nginx/nginx.conf"
  perms       = 0600
  command     = "service nginx reload"
}

Consul Template File: nginx.ctmpl

The below file is what I used to template the backend servers for nginx. This will range over the service checks in the consul registry and only select ones with the name “nginx.”

events {}stream {
    upstream stream_backend {
    {{range services}}
      {{ if .Name | contains "nginx" }}
      {{- else -}}
        {{ range service .Name }} 
          server {{ .Address }}:8500;
        {{- end -}}
       {{- end -}}
    {{- end -}}
    }server {
        listen                443 ssl;
        proxy_pass            stream_backend;ssl_certificate       /etc/nginx/tls/consul.crt;
        ssl_certificate_key   /etc/nginx/tls/consul.key;
        ssl_protocols         SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers           HIGH:!aNULL:!MD5;
        ssl_session_cache     shared:SSL:20m;
        ssl_session_timeout   4h;
        ssl_handshake_timeout 30s;
     }
}

consul-config-1.json

Only one of the consul config json files needs “bootstrap_expect”: 3.

{
  "datacenter": "test",
  "node_name": "consul-server-1",
  "bootstrap_expect": 3,
  "bind_addr": "0.0.0.0",
  "client_addr": "0.0.0.0",
  "log_level": "DEBUG",
  "server": true,
  "ui": true,
  "enable_script_checks": true,
  "addresses": {
    "https": "0.0.0.0"
  },
  "ports": {
    "https": 8501
  },
  "check_update_interval": "1m",
  "ca_file": "/opt/consul-server/tls/ca.crt",
  "cert_file": "/opt/consul-server/tls/consul.crt",
  "key_file": "/opt/consul-server/tls/consul.key",
  "http_config": {
    "response_headers": {
      "Access-Control-Allow-Origin": "*"
    }
  },
  "leave_on_terminate" : true,
  "retry_interval" : "5s",
  "retry_join" : [
        "consul-server-1.consul_consul:8301",
        "consul-server-2.consul_consul:8301",
        "consul-server-3.consul_consul:8301",
 "nginx.consul_consul:8301"
  ],
  "server_name" : "consul-server-1.test",
  "skip_leave_on_interrupt" : true,
  "node_meta": {
      "instance_type": "Docker container"
  }
}

NGINX consul agent config: consul-config.json

The following file is used as the consul agent config for the nginx docker container. The important bit here is “enable_script_checks” set to true to enable the service checks.

{
  "datacenter": "test",
  "node_name": "nginx",
  "bind_addr": "0.0.0.0",
  "client_addr": "0.0.0.0",
  "log_level": "DEBUG",
  "check_update_interval": "1m",
  "enable_script_checks": true,
  "ca_file": "/etc/nginx/tls/ca.crt",
  "cert_file": "/etc/nginx/tls/consul.crt",
  "key_file": "/etc/nginx/tls/consul.key",
  "leave_on_terminate" : true,
  "retry_interval" : "5s",
  "retry_join" : [
        "consul-server-1.consul_consul:8301",
        "consul-server-2.consul_consul:8301",
        "consul-server-3.consul_consul:8301",
 "nginx.consul_consul:8301"
  ],
  "server_name" : "nginx.test",
  "skip_leave_on_interrupt" : true,
  "node_meta": {
      "instance_type": "Docker container"
  }
}

Service Check files: consul-server-1, 2, and 3.json

Below is the service check definition for consul-server-1. I also created nginx-2, and nginx-3. These live within the nginx container. This is what consul template uses to associate a service with a backend server.

{
  "service": {
    "Name": "nginx-1",
    "Port": 8500,
    "check": {
      "args": ["curl", "consul-server-1.consul_consul:8500"],
      "interval": "3s"
    }
  }
}

docker-compose.yml

version: '3'services:
  consul-server-1:
    image: consul:latest
    container_name: consul-server-1
    restart: always
    ports:
      - "9990:8500"
      - "9991:8501"
      - "9993:8600"
    networks:
      consul:
        ipv4_address: 192.168.172.11
    command: "agent -data-dir=/opt/consul-server/data -config-dir=/opt/consul-server/config -config-file=/opt/consul-server/config/consul-config-1.json"
    volumes: 
      - /opt/consul-server/config/config-1:/opt/consul-server/config
      - /opt/consul-server/data-1:/opt/consul-server/data
      - /opt/consul-server/tls:/opt/consul-server/tlsconsul-server-2:
    image: consul:latest
    container_name: consul-server-2
    restart: always
    ports:
      - "9994:8500"
      - "9995:8501"
      - "9996:8600"
    networks:
      consul:
        ipv4_address: 192.168.172.12
    command: "agent -data-dir=/opt/consul-server/data -config-dir=/opt/consul-server/config -config-file=/opt/consul-server/config/consul-config-2.json"
    volumes:
      - /opt/consul-server/config/config-2:/opt/consul-server/config
      - /opt/consul-server/data-2:/opt/consul-server/data
      - /opt/consul-server/tls:/opt/consul-server/tlsconsul-server-3:
    image: consul:latest
    container_name: consul-server-3
    restart: always
    networks:
      consul:
        ipv4_address: 192.168.172.13
    ports:
      - "9997:8500"
      - "9998:8501"
      - "9999:8600"
    command: "agent -data-dir=/opt/consul-server/data -config-dir=/opt/consul-server/config -config-file=/opt/consul-server/config/consul-config-3.json"
    volumes:
      - /opt/consul-server/config/config-3:/opt/consul-server/config
      - /opt/consul-server/data-3:/opt/consul-server/data
      - /opt/consul-server/tls:/opt/consul-server/tlsnginx:
    image: consul-nginx:latest
    container_name: nginx
    restart: always
    ports:
      - 9443:443
    command: 
      - /bin/bash
      - -c
      - |
        /etc/init.d/nginx start
        /bin/rm /etc/nginx/nginx.conf
        /usr/local/bin/consul agent -data-dir=/opt/consul-agent/nginx/data -config-dir=/opt/consul-agent/nginx/config -config-file=/opt/consul-agent/nginx/config/consul-config.json &
        /usr/local/bin/consul-template -config=/opt/consul-agent/nginx/template/consul-template-config.hcl
    networks:
      consul:
        ipv4_address: 192.168.172.14
    volumes:
      - /opt/consul-server/tls:/etc/nginx/tls
      - /opt/nginx/nginx.ctmpl:/etc/nginx/nginx.ctmpl
      - /opt/consul-agent/nginx/data:/opt/consul-agent/nginx/data
      - /opt/consul-agent/nginx/config:/opt/consul-agent/nginx/config
      - /opt/consul-agent/nginx/template:/opt/consul-agent/nginx/templatenetworks:
  consul:
    ipam:
      driver: default
      config:
        - subnet: 192.168.172.0/24

Directory Structure

Directory structure under virtual machine host /opt folder. I had to chmod -R 777 the consul-server folders as a way to get them writable by the docker container.

├── consul-agent
│   └── nginx
│       ├── config
│       │   ├── consul-config.json
│       │   ├── consul-server-1.json
│       │   ├── consul-server-2.json
│       │   └── consul-server-3.json
│       ├── data
│       │   ├── checkpoint-signature
│       │   ├── node-id
│       │   └── serf
│       │       └── local.snapshot
│       └── template
│           └── consul-template-config.hcl
├── consul-server
│   ├── config
│   │   ├── config-1
│   │   │   └── consul-config-1.json
│   │   ├── config-2
│   │   │   └── consul-config-2.json
│   │   └── config-3
│   │       └── consul-config-3.json
│   ├── data-1
│   │   ├── checkpoint-signature
│   │   ├── node-id
│   │   ├── raft
│   │   │   ├── peers.info
│   │   │   ├── raft.db
│   │   │   └── snapshots
│   │   │       └── 184-16385-1603784745444
│   │   │           ├── meta.json
│   │   │           └── state.bin
│   │   └── serf
│   │       ├── local.snapshot
│   │       └── remote.snapshot
│   ├── data-2
│   │   ├── checkpoint-signature
│   │   ├── node-id
│   │   ├── raft
│   │   │   ├── peers.info
│   │   │   ├── raft.db
│   │   │   └── snapshots
│   │   │       └── 184-16386-1603784746009
│   │   │           ├── meta.json
│   │   │           └── state.bin
│   │   └── serf
│   │       ├── local.snapshot
│   │       └── remote.snapshot
│   ├── data-3
│   │   ├── checkpoint-signature
│   │   ├── node-id
│   │   ├── raft
│   │   │   ├── peers.info
│   │   │   ├── raft.db
│   │   │   └── snapshots
│   │   │       └── 184-16392-1603784787600
│   │   │           ├── meta.json
│   │   │           └── state.bin
│   │   └── serf
│   │       ├── local.snapshot
│   │       └── remote.snapshot
│   └── tls
│       ├── ca.crt
│       ├── consul.crt
│       └── consul.key
├── containerd [error opening dir]
└── nginx
    ├── nginx.conf
    ├── nginx.ctmpl
    └── scripts
        └── init.sh

Operation

When starting docker-compose up -d to start the cluster the following happens:

  • A docker network consul_consul will be created with the IPAM information in the compose file.
  • Consul-server-1, consul-server-2, and consul-server-3 containers start the latest consul docker image, mount their tls/config/data directories from the host and being to join the consul cluster. A leader will be elected and the UI will be accessible from the outside at the <host_port> we specified to map to port :8501 on each docker container.
  • Our custom nginx container will start, read its volume mounts, grab its tls certs from the volume mount, consul agent will start, join the current consul cluster, start service checks for nginx-1, nginx-2, and nginx-3 respectively, consul template will run and reach out to the consul template config/template file, it will search for available services with the name nginx, template the nginx.conf file, write it to the config directory, and reload nginx.
  • After these containers have started, your consul cluster will be load balanced by the nginx container using a consul template backend.
  • You can access the consul cluster frontend through the DNS A record you created at port 9443 as the nginx docker container points 9443 to 443.
consul nodes
consul services
consul service detail
nginx-1 service detail

Thank You! 😃

Hashicorp
Consul
Docker
Docker Compose
Nginx

--

--

Rick Kemery

Written by Rick Kemery

8 Followers

I am a devoted learner and writer of technical articles with a devops focus.

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams