Looking for new opportunities
I am always looking for a new opportunities to expand my knowledge and help others while doing so, there for if you are looking for someone to join your team or you have something you want some help with feel free to reach out via email zaher[@]zah.me.
In the last post we talked bout preparing your PIs to be part of a cluster, configure it and make sure it is up-to-date. In this article I'll be talking about installing the most essential programs:
The main reason for using Gluster is to have a sharable persistent storage between your PIs that Docker Swarm can use, since Docker Swarm (at least the CE version) does not support a persistent storage out of the box (if you know otherwise feel free to share the information).
The easiest way to install Docker is to use the installation script provided by Docker team, which you can obtain from this link.
To install Docker you run the following command as root on each PI:
curl -fsSL https://get.docker.com | sh -
Once the script finish installing Docker, you need to add your user to the
docker group, and this can be done using the following command on all your PIs:
sudo usermod -aG docker pi
Remember to replace
pi with your username if you choose to use a different user than
pi. You will need to logout/login again for your user to get the correct permissions.
That's it, now you have docker installed on your pi, and you can verify it by running the following command:
If everything was correct you will get some info like the following:
Client: Debug Mode: false Plugins: app: Docker Application (Docker Inc., v0.8.0) buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker) mutagen: Synchronize files with Docker Desktop (Docker Inc., testing) Server: Containers: 4 Running: 4 Paused: 0 Stopped: 0 Images: 8 Server Version: 19.03.11 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 5.4.44-v8+ Operating System: Debian GNU/Linux 10 (buster) OSType: linux Architecture: aarch64 CPUs: 4 Total Memory: 3.534GiB Name: main ID: 4J77:EY2N:NLM4:2JX3:S7XQ:4F42:EXRU:EVI2:YZD2:CYHC:PZGD:ROFM Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: provider=generic Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
Things might be a bit different at your end from what I presented, but this is a good sign that everything is working as plan.
Docker in PI will produce a few warnings, you can ignore them as they are harmless.
Now, you have docker installed on your PIs, and it is the time to initiate the Swarm cluster and join the PIs together, but before we do that we need to change a few things.
All your PIs will have the
raspberry as the hostname, but we need to make sure we differentiate each one by giving it a unique name, so we need to edit the
/etc/hostname file and change the name to something we choose.
sudo nano /etc/hostname
You can call the first one
main, the second one
worker1, the third one
worker2 .. etc. Since I only have two PIs I've called them
Then you need to edit
/etc/hosts file to reflect the changes and to identify the PIs you have in your cluster, so the file should have something like the following:
/etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters 192.168.68.120 main 192.168.68.125 worker
As I mentioned, this is what the file should look like in all your PIs.
Now we need to go to the main one, and run the following command:
docker swarm init
You will then presented with something like:
Swarm initialized: current node (3yj9paq9hn1dp6rpjjhvxjs06) is now a manager. To add a worker to this swarm, run the following command: docker swarm join --token SWMTKN-1-4posux641n7yiqbniwf3sr6rjpqbuth0bnogebajajhqf4bpub-805veloekj7fkx6cd0h8x1tjw 192.168.68.120:2377 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
AS the message noted we need to run the following command, to get the token for adding a manager:
docker swarm join-token manager
It will return something like:
To add a manager to this swarm, run the following command: docker swarm join --token SWMTKN-1-4posux641n7yiqbniwf3sr6rjpqbuth0bnogebajajhqf4bpub-5hft39oha9rdtvrhvbwum2zkv 192.168.68.120:2377
Now SSH into the second device and run the following command:
docker swarm join --token SWMTKN-1-4posux641n7yiqbniwf3sr6rjpqbuth0bnogebajajhqf4bpub-5hft39oha9rdtvrhvbwum2zkv 192.168.68.120:2377
docker node ls on any of the PIs should return something like:
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION uty5oo1shl9dl7k119ov1xcnw * main Ready Active Leader 19.03.11 6a5v2sofsow9qsefybip8901z worker Ready Active Reachable 19.03.11
Please note that I am not following the recommendations from docker about building a swarm cluster as I am running two managers, but the minimum number of managers for redundancy is 3. More info can be found here.
The talk about having a persistent shared volume in Docker swarm is not going to be small, but since many have already talked about the topic, I am not going to go into details, I'll just explain how to install Gluster as it is the only solution I have found working with my PI Cluster.
By installing Gluster we will achieve the following:
Image credited to theworkaround.com
To install Gluster we need to run the following commands on each PI:
sudo apt update -y sudo apt install -y glusterfs-server
These commands will install
5.5-3 for arm64, sadly this is the latest version which was published to the repository, at the time of writing the latest version is
7 but I am not going to install it from the source. If you like to do so, you can check Gluster documentations.
All commands should be run on each PI unless I mentioned otherwise.
First of all you need to enable Gluster by running the command:
sudo systemctl enable glusterd
Then we need to start it by running
sudo systemctl start glusterd
To make sure everything is working as it should be, we run the following commands:
Verify glusterd is enabled sudo systemctl is-enabled glusterd Check the system service status of the glusterd sudo systemctl status glusterd
The result for the last command shoule be something like:
● glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/lib/systemd/system/glusterd.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-06-10 10:44:32 +03; 1h 15min ago Docs: man:glusterd(8) Process: 467 ExecStart=/usr/sbin/glusterd -p /run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCES Main PID: 502 (glusterd) Tasks: 43 (limit: 4033) CGroup: /system.slice/glusterd.service ├─502 /usr/sbin/glusterd -p /run/glusterd.pid --log-level INFO ├─689 /usr/sbin/glusterfsd -s master --volfile-id staging-gfs.master.gluster-brick -p /var/run/gluster/vols/staging-gfs/mast └─771 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var Jun 10 10:44:18 main systemd: Starting GlusterFS, a clustered file-system server... Jun 10 10:44:32 main systemd: Started GlusterFS, a clustered file-system server.
For simplicity, I'll probe all nodes so run the following command:
sudo gluster peer probe main; sudo gluster peer probe worker;
Running the following command on our
main PI should return something like:
sudo gluster pool list should return UUID Hostname State 715ab501-177d-4337-8b9b-e1405ff133c3 worker Connected 0afa69d0-4484-4a6b-b500-d79f8c6eab2d localhost Connected to check the peer status run sudo gluster peer status should return Number of Peers: 1 Hostname: worker Uuid: 715ab501-177d-4337-8b9b-e1405ff133c3 State: Peer in Cluster (Connected)
We are now ready to create the gluster storage directory and volume by running the following commands:
sudo mkdir -p /gluster/brick create a gluster volume across all nodes sudo gluster volume create staging-gfs replica 2 main:/gluster/brick worker:/gluster/brick force Start the volume sudo gluster volume start staging-gfs Check the volume status sudo gluster volume info You will get something like Volume Name: staging-gfs Type: Replicate Volume ID: 7004cb0a-0040-4b62-b4d4-79489c37ef68 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: master:/gluster/brick Brick2: worker:/gluster/brick Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet
If you have more than two nodes, the number of replica should reflect to the number or your nodes so if for example you have five nodes the volume command should be something like
sudo gluster volume create staging-gfs replica 5 main:/gluster/brick worker1:/gluster/brick worker2:/gluster/brick worker3:/gluster/brick worker4:/gluster/brick force
Now we need to specify a place where we should mount the gluster volume to, I prefer not to use
/mnt directly instead creating a directory inside
shared, as from my experience whenever you restart your node gluster won't be able to mount it and your cluster will fail to work.
So we run the following commands:
sudo mkdir -p /mnt/data sudo echo 'localhost:/staging-gfs /mnt/data glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab sudo mount.glusterfs localhost:/staging-gfs /mnt/data sudo chown -R pi:pi /mnt/data
Let's validate that we have
staging-gfs is listed under our mounted partitions:
run the following command sudo df -h should get something like Filesystem Size Used Avail Use% Mounted on /dev/root 59G 5.6G 51G 11% / devtmpfs 1.7G 0 1.7G 0% /dev tmpfs 1.8G 0 1.8G 0% /dev/shm tmpfs 1.8G 480K 1.8G 1% /run tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup /dev/mmcblk0p1 253M 54M 199M 22% /boot localhost:/staging-gfs 59G 6.2G 51G 11% /mnt/data tmpfs 362M 0 362M 0% /run/user/1000
As we can see
localhost:/staging-gfs is mounted on
Just a reminder, all the commands should be run on all your nodes not only the manager.
To validate that everything is working as it should, we create a file in our main node, and it should be listed on our worker directory.
Click here for bigger animated image.