Zaher Ghaibeh
PHP Backend developer
I've experience in a few PHP Frameworks, such as Laravel, Lumen and Slim (The last two are used for building Microservices/API services).
Gitlab App-Review with Docker Swarm, Traefik and Letsencrypt
Published at Saturday, June 1, 2019 , Categorized under: gitlab, development, docker, docker-swarm, developers, testing

Prerequisite

A few things to mention before starting:

  1. I'll assume that you have read my last article.
  2. I'll assume that you know how to work with Docker Swarm and Gitlab CI/CD.
  3. I'll assume that you have taken the time to read David Négrier article Continuous delivery with Gitlab, Docker, and Traefik on a dedicated server.
  4. I'll assume that you are already familiar with PHP/Linux and Laravel for sure.
  5. I'll assume that you are familiar with Traefik or at least the concepts of a reverse proxy.

Finally, please make sure you are fully aware of the security risks of every step we are doing. After all of this let's start.

Introduction

As David has explained in his article, we need to make sure that Traefik is the first service which will route the HTTP requests to the corresponding container. This won't change in the case of using docker swarm. Also we will need to make sure that Traefik will work like a load-balancer between the swarm nodes.

What I'll describe here will work with one node swarm cluster and up, and I'll try to explain what to do when you expand your swarm cluster to more than one node.

When your docker swarm cluster consists of one node, you will not need to have any other components other than your server. The same way David has only a dedicated server for his work. But when you have a docker swarm cluster with 3, 5, 7 nodes you will need to have a load-balancer setting in-front of your cluster which will communicate with your manager's node.

Like the following image:

The reason that you need to have a load-balancer setting in-front of your docker swarm cluster is that you don't need to setup your cluster to fail. We all know that docker swarm managers will redirect the communication between them, so whenever you send a request to one of the managers it will load-balance your request and get back with a response, so let's assume that you have pointed out your development domain to manager 1 IP, but at some point manager 1 is not responding what you will do? for sure none of the other managers will response even if they have elected another manager as the lead because your domain does not point to the new lead manager, to solve such a problem you will need to have a load-balancer that will load the requests between the three managers as I draw in the above image, but make sure not to include any of the workers.

In the image above we have setup a docker swarm cluster of 5, 3 managers and 2 workers with a load-balancer setting up in front of the leading managers. Now, whenever you add a new manager, you can add them to the load-balancer.

Setting up Traefik on Docker Swarm

To be honest, Traefik documentation has everything you need, just read it, so setting up Traefik on the cluster was not hard. Basically you can do it using the following steps:

1- Create a network interface which everyone can use docker network create webgateway -d overlay --scope swarm 2- Use the following docker-compose.yml file to launch your traefik stack, I am using version 1.7.12 as this release fixes a CVE, and is an important release for that reason, but feel free to use the latest image instead:

version: "3.7"
services:
  traefik:
    image: traefik:1.7.12
    environment:
      - "DO_AUTH_TOKEN=<PUT_YOUR_DO_AUTH_TOKEN_HERE>"
    command:
      - "--api"
      - "--api.statistics.recentErrors=100"
      - "--entrypoints=Name:http Address::80 Redirect.EntryPoint:https"
      - "--entrypoints=Name:https Address::443 TLS"
      - "--defaultentrypoints=http,https"
      - "--logLevel=DEBUG"
      - "--acme"
      - "--acme.acmelogging"
      - "--acme.storage=/tmp/acme.json"
      - "--acme.entryPoint=https"
      - "--acme.onHostRule=true"
      - "--acme.onDemand=false"
      - "--acme.email=youremail@provider.com"
      - "--acme.dnschallenge"
      # For list of all supported providers check 
      # https://docs.traefik.io/configuration/acme/#provider     
      - "--acme.dnsChallenge.provider=digitalocean"
      - "--acme.domains=*.yourdomain.com"
      - "--docker"
      - "--docker.swarmMode=true"
      - "--docker.domain=yourdomain.com"
      - "--docker.watch"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - webgateway
      - traefik
    ports:
      - target: 80
        published: 80
        mode: host
      - target: 443
        published: 443
        mode: host
      - target: 8080
        published: 8080
        mode: host
    deploy:
      mode: global
      placement:
        constraints:
          - node.role == manager
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure

networks:
  webgateway:
    driver: overlay
    external: true
  traefik:
    driver: overlay

Since am using DigitalOcean DNS for my domain, I had to specify the provider for my DNS challenge which Traefik will setup for me automatically to make sure that any domain I get will have Letsencrypt SSL read for it, more info can be found here and here.

3- Replace the DO_AUTH_TOKEN environment based on what you have generated.

once you are done from creating the required files you should have something like:

$ ls -al
total 40
drwxr-xr-x  7 zaher  staff   224 Jun  1 16:41 .
drwxr-xr-x  4 zaher  staff   128 May 29 19:38 ..
-rw-r--r--  1 zaher  staff    78 Jun  1 17:34 .env
-rw-r--r--  1 zaher  staff  1579 Jun  1 17:34 docker-compose.yml

4- If everything run as the plan, you can create the stack using the command

$ docker stack deploy traefik -c docker-compose.yml 

Setting up Gitlab SSH

This process is not fun; You can find more information about using SSH with Gitlab CI in the Gitlab documentation. I'll try to make it as simple as I can, but remember you need to read the docs to understand the security risks about working with SSH within Gitlab CI.

1- Assuming that you are not using the root user. Create SSH key from within one of the managers' servers.

$ ssh-keygen -t rsa

2- Add the public key to your authorized_keys so you can access the server from outside.

$ cat id_rsa.pub | tee -a ~/.ssh/authorized_keys

If you have a key that you use within your team to access the servers, you can skip the steps one and two.

3- Copy the content of your private key, the one you use to access the server or the one you just created:

$ cat ~/.ssh/id_rsa

In Gitlab 10, in your project page, go to: Settings > CI/CD > Secret variables, Create a new variable named SSH_PRIVATE_KEY, whose value is the content of the key you just copied.

4- Get the public key or the fingerprint for your server, but be careful about this step, if you have a load-balancer in front of your cluster then you should use the IP address of the server you will always access, if not you can use your domain name.

$ ssh-keyscan <IP/DomainName>

In Gitlab's CI/CD secret variables settings, create a new variable named SSH_KNOWN_HOSTS, and paste the values you got from the previous command. This value will be used later to identify your server within the known_hosts file.

Last note, if you are using a different user other than root, make sure that you have added it to docker group using the command

$ sudo usermod -aG docker <username>

Setting up your Gitlab CI/CD

Am not going to go in too many details here, I'll assume you already knew and use Gitlab CI for testing, but I'll go into what I did to make my life simpler.

1- Make sure that the stages within your CI definition have the dokerize step before the deploy step.

stages:
  - build
  - test
  - dockerize
  - deploy

This to make sure that your code will be dockerized before it is deployed to the test servers.

2- You will need to create a Dockerization task within your Gitlab CI tasks:

Dockerize:
  stage: dockerize
  image: docker:stable
  services:
    - docker:dind
  cache:
    policy: pull
    paths:
      - public/
      - .env
  only:
    - branches       
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG || true
    - docker build --cache-from $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

This task will dockerize your code after passing the test and pushed it to gitlab docker registry. You notice that we are using a lot of the predefined Gitlab variables and they are:

Variable Explanation
$CI_REGISTRY_USER The username to use to push containers to the GitLab Container Registry
$CI_REGISTRY_PASSWORD The password to use to push containers to the GitLab Container Registry
$CI_REGISTRY If the Container Registry is enabled it returns the address of GitLab’s Container Registry
$CI_COMMIT_REF_SLUG $CI_COMMIT_REF_NAME lowercased, shortened to 63 bytes, and with everything except 0-9 and a-z replaced with -. No leading / trailing -. Use in URLs, host names and domain names.
$CI_REGISTRY_IMAGE If the Container Registry is enabled for the project it returns the address of the registry tied to the specific project
$CI_COMMIT_SHORT_SHA The first eight characters of CI_COMMIT_SHA

3- You will need to create a Dockerfile which will be used by your Dockerization task

FROM zaherg/php-and-nginx:7.3

ENV APP_ENV production

COPY . /web/html

WORKDIR /web/html

RUN composer install  --prefer-dist --no-ansi --no-interaction --no-scripts --no-progress --no-suggest --optimize-autoloader --no-dev \
    && sed -i 's/root \/web\/html;/root \/web\/html\/public;/' /etc/nginx/nginx.conf \
    && chgrp -R nobody /web/html/storage /web/html/bootstrap/cache \
    && chmod -R ug+rwx /web/html/storage /web/html/bootstrap/cache \
    && chown -R nginx /web/html/storage /web/html/bootstrap/cache

Now all we need to do is to work on our deploy tasks, where we connect to our Server, pull the image and run our application, most of what am doing here is similar to what David has explained, so am not going to go into details.

4- Create our deploy script with our custom docker-compose.yml file, the important of the later file is that we can use it to build the whole infrastructure including and not limited to any microservice or internal service we use, at the end, we are not going to expose it.

I like to put my deployment scripts within a folder called deploy this way I can be specific and separate them from the rest of the code.

The content of the deploy script is:

#!/usr/bin/env bash

export CI_REGISTRY=$1
export CI_COMMIT_SHORT_SHA=$2
export DEPLOY_USER=$3
export DEPLOY_KEY=$4
export APPLICATION_URL=$5

docker login registry.gitlab.com -p ${DEPLOY_KEY} -u ${DEPLOY_USER}

docker pull ${CI_REGISTRY}:${CI_COMMIT_SHORT_SHA}

docker stack deploy ${CI_COMMIT_SHORT_SHA} -c /home/developer/deploy/${CI_COMMIT_SHORT_SHA}/docker-compose.yml

And the content of the docker-compose.yml file is:

version: "3.7"

services:
  app:
    image: "${CI_REGISTRY}:${CI_COMMIT_SHORT_SHA}"    
    networks:
      - webgateway
    logging:
      driver: "json-file"
      options:
        max-size: "200k"
        max-file: "10"
    deploy:
      labels:
        - "traefik.backend=${CI_COMMIT_SHORT_SHA:-master}"
        - "traefik.frontend.rule=Host:${CI_COMMIT_SHORT_SHA:-www}.app.${APPLICATION_URL}"
        - "traefik.docker.network=webgateway"
        - "traefik.enable=true"
        - "traefik.port=80"
        - "traefik.backend.loadbalancer.method=drr"
        - "traefik.default.protocol=http"
      mode: replicated
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure

  database:
    image: mysql:5.7
    hostname: database
    environment:
      - "MYSQL_USER=homestead"
      - "MYSQL_DATABASE=homestead"
      - "MYSQL_RANDOM_ROOT_PASSWORD=yes"
      - "MYSQL_PASSWORD=secret"  
    networks:
      - webgateway
    deploy:
      labels:
        - "traefik.backend=${CI_COMMIT_SHORT_SHA:-master}"
        - "traefik.frontend.rule=Host:${CI_COMMIT_SHORT_SHA:-master}.db.${APPLICATION_URL}"
        - "traefik.docker.network=webgateway"
        - "traefik.enable=true"
        - "traefik.port=3306"
        - "traefik.backend.loadbalancer.method=drr"
        - "traefik.default.protocol=http"      
      mode: replicated
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure              

  redis:
    image: redis:alpine
    hostname: cache
    networks:
      - webgateway
    deploy:
      labels:
        - "traefik.backend=${CI_COMMIT_SHORT_SHA:-master}"
        - "traefik.frontend.rule=Host:${CI_COMMIT_SHORT_SHA:-master}.cache.${APPLICATION_URL}"
        - "traefik.docker.network=webgateway"      
        - "traefik.enable=true"
        - "traefik.port=6379"
        - "traefik.backend.loadbalancer.method=drr"
        - "traefik.default.protocol=http"
      mode: replicated
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure        

networks:
  webgateway:
    external: true
    name: webgateway

Using the above docker-compose.yml file we will create a separate stack within our docker swarm that consists of our application, MySQL, and Redis all of them are separated from all the services within the same swarm.

5- Our last step is to configure our deploy tasks like the following:

deploy_review:
  image: kroniak/ssh-client:3.9
  stage: deploy
  before_script:
    # add the server as a known host
    - mkdir ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
    # add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
    - eval $(ssh-agent -s)
    - ssh-add <(echo "$SSH_PRIVATE_KEY")
    - ssh $SSH_SERVER "mkdir -p /home/developer/deploy/$CI_COMMIT_SHORT_SHA"
  script:
    - scp ./deploy/docker-compose.yml $SSH_SERVER:/home/developer/deploy/$CI_COMMIT_SHORT_SHA/docker-compose.yml
    - ssh $SSH_SERVER 'bash -s' < ./deploy/deploy $CI_REGISTRY $CI_COMMIT_SHORT_SHA $DEPLOY_USER $DEPLOY_KEY $APPLICATION_URL
  environment:
    name: review/$CI_COMMIT_REF_NAME
    url: http://$CI_COMMIT_SHORT_SHA.app.$APPLICATION_URL
    on_stop: stop_review
  only:
    - branches
  except:
    - master

stop_review:
  image: kroniak/ssh-client:3.9  
  stage: deploy
  variables:
    GIT_STRATEGY: none
  before_script:
    # add the server as a known host
    - mkdir ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
    # add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
    - eval $(ssh-agent -s)
    - ssh-add <(echo "$SSH_PRIVATE_KEY")   
  script: 
    - ssh $SSH_SERVER "docker stack rm $CI_COMMIT_SHORT_SHA"
    - ssh $SSH_SERVER "docker rmi -f $CI_REGISTRY:$CI_COMMIT_SHORT_SHA"
    - ssh $SSH_SERVER "rm -fr /home/developer/deploy/$CI_COMMIT_SHORT_SHA"
  when: manual
  environment:
    name: review/$CI_COMMIT_REF_NAME
    action: stop
  except:
    - master    

We SSH into our server, we do the magic and let the server know that this is us, please check David post, then we copy our docker-compose.yml file to the server and execute the deploy script.

Now every time you push your code you will get a working environment with a secure URL based on the short SHA1 of the commit you have pushed. Once your QA team test it, they can just run the stop deploy action which will terminate the stack and delete the image from the swarm.

I know it was a long post, but I hope it will be useful for you.

Share It via: