homegrowing a 'datacenter' (to scale)

Sat, Feb 6, 2021 6-minute read

I’m currently in the process of building my own datacenter/cluster/name it what you want!

I decided to write down what happened so far and what’s going to happen next, as this is not only a good way of keeping this blog alive, but also helps me re-thinking what I want to achieve / where I’d like to go with this project.

the usecase(s)

In “the cluster”, I’m going to run a couple of services that are already running on my PC, my NAS, a couple of Raspberry Pis as well as an old Intel two-Xeon processor server that requires a massive amount of power in comparison to a modern/lightweight approach I’m trying to build now.

The services include SoftEther VPN, webserver, home automation, photo deduplication and backup, a git server, nuget/maven/npm repository, a ZoneMinder server and a couple of others as well.

Because my daughter Rebecca now can basically access every part of the house on her own, I wanted to make my equipment as child safe as possible, so I got a 19" rack to put all my equipment in there and only have a bunch of cables running to my desk.

considerations

I want to build something that…

  • has an actual use-case
  • is scalable
  • is “fully automated”
  • is possible to troubleshoot
  • is not considered “boring”
  • should not be garbage in a couple of years
  • is considered “low cost” but great value

LC,LE,HP a.k.a. “low cost, low energy, high performance” unfortunately is quite hard to achieve in the real world. You can basically choose two of these three pillars to focus on, the third one is going to be much harder to get by.

hardware

The Raspberry Pi 4 is currently (imo) as good as it gets when aiming for low cost, low energy and ‘high performance’.

I’ve decided to get two of the 8GB Pis that are going to do the ‘heavy load’ work, in addition to that I’ve already had four “Raspberry Pi 3” single board computers somewhere spread over the house.

When looking for solutions on how to mount a number of Raspberry Pis into a 19" rack, I came across this blogpost by ulab which provides a lot of detail on how to 3d-print a case for either 14 or 18 Raspberry Pis (the version with 14 Pis can also hold the same number of SSD-drives, which is why I went for this option).

As I’ve got no 3D-printer myself, a friend was kind enough to do the work for me, not that it takes a while to print all the necessary parts, also be careful not to break stuff when doing the final assembly!

I’m quite satisfied with the results so far, maybe I should cleanup the rack before I take pictures next time 😄

software

There are lot of resources out there on how to build a “Raspberry Pi Cluster” powered by Kuberentes and Docker. I tried it. I didn’t like it. At all. Not that it’s not great if you’re aiming for uptime and scalability, this is what Kubernetes is designed for. If you are looking for Kubernetes-based solution, see michael-robbins/rpi-k8s-ansible - I’ve taken this for a test and it wasn’t all that hard to wire things up correctly. As mentioned in my considerations, I’m trying to build something I can fully understand and troubleshoot myself,- which is where I’m (currently) not going to go down the rabbit hole of trying to get along with Kubernetes. (something to consider for the future though)

Currently I’ve identified following software stack as sufficient for me:

  • Anisble - will be used to wire up the whole environment, I want to be able to set-up everything in this cluster by just running a couple of playbooks, each of the following libs/tools/pieces will be strictly defined in a custom playbook/task (thus adding new nodes should be a no-brainer)
  • Jenkins - Jenkins will be used to setup and orchestrate jobs
  • PowerShell - a lot of my already existing scripts are written in PowerShell
  • OpenJDK 11 - some of my older tools are written in Java, it also is required by Jenkins
  • sonatype Nexus - a amazing repository service that runs just fine on rpi
  • ElasticSearch / Kibana - used for cluster health/metrics as well as home automation data
  • Flickr Photo Uploadr - is responsible for backup and de-duplication of my rather large photo library

setting things up

Make sure to get the latest Raspberry Pi OS Lite or something similar, forget about the UI, yagni! After flashing the image to the SD cards (I tend to use balena etcher), make sure to create a file called ssh on the boot partition of the card that shows up in your file manager. (that file will ensure a ssh server is being started for you, so that you never actually need to connect a display in order to get going.)

Log in to your router and figure out the IP address of the Raspberry Pis using the DHCP table. Do yourself a favor and do this for one after another, or you’ll have the same “fun” I did when I attached 5 Pis at once and had to figure out which Pi had which address.

As soon as you log in to the pi (username pi, password raspberry), change hostname and password using raspi-config. Do NOT use the same password for multiple nodes! Use a password manager, such as keepass to store your credentials.

Once this is done, I tend to push my id_rsa.pub to ~/.ssh/authorized_keys in order to use public key authentication.

hosts="192.168.0.130 192.168.0.131 192.168.0.132 192.168.0.133 192.168.0.134"
for h in $hosts; do
  ssh [email protected]$h "echo \"`cat ~/.ssh/id_rsa.pub`\" >> .ssh/authorized_keys"
done

As mentioned above, I try to use Ansible to describe the whole environment as detailed as possible, you can find an early version of the base setup on my GitHub repo for this project.

In theory it should be no more than a

ansible-playbook -i cluster.yml site.yml

to have everything set up correctly.

what’s next

I’m still very unfamiliar with Ansible, so I’m basically playing around with it, creating scripts to auto-update the whole cluster, deploy packages and configs based on labels and such. Some of the applications I want to run, such as Jenkins require more work than others (i.e. setting up the Jenkins Master on one node and auto-join an agent on each of the other nodes and label them according to labels in the ansible config).

Stay tuned for more parts of this “cluster homegrowing series”, happy for all the comments you got! 😄

~ stay tuned, happy hacking!