2017. március 19., vasárnap

Infrastructure as Code - Basic cluster setup with Ansible

2:47 Posted by Juhász Sándor No comments

As you may already know, I have built the Pi cluster to be a sandbox in which I can tinker with various technology stacks, to experiment, to tweak configs, to try out new ideas quickly and easily. My goal is to play with the currently fashionable stacks like Docker, Hadoop, ELK, you name it.

Everyone who has tried setting up a server stack knows that it is not an easy job: it may take days or weeks and hundreds of little steps just to get a stack running. Installation is an essential part of the fun especially for hobby projects, but it also makes you more and more cautious as your system's complexity grows, ultimately limiting the speed of innovation. What if you mess up a configuration and you need to restore the system to an earlier state? What would happen if you would need to start over from scratch? Will you still remember all the steps you need to execute to install the stack in 6 months? If you just add customization to a stack without any documentation, you will fear reinstalling it in just a few weeks. Even if you write a very detailed setup documentation, repeating each step by hand, in a 4-node cluster is tedious and error prone to say at least. Over time, you end up with snowflake-servers: unique and irreproducible machines with configs you do not dare to touch.

To overcome the obstacle of configuration management, I decided to use "Infrastructure as Code" principles to set up the cluster. The basic idea is to formalize the state of the infrastructure in a markup language (ruby, yaml, etc.) and use a provisioning / configuration management software to realize the setup on cloud or on-premise hardware. This code is usually declarative, describing properties of the desired system and the configuration management software will execute operating system commands to build a system. A configuration like this is usually written to be idempotent: it can be run multiple times without the fear of corrupting the system. The source code is written using software engineering best practices: it is stored in version control system, it is reviewed and tested much like the source code of any other regular software product. With this approach the system configuration is well documented and it can be executed by a machine which leads to quicker setup times and less errors. Provisioning systems becomes very inexpensive. In early 2010, with cloud computing, "Infrastructure as Code" became popular among early adopters and it appeared in the ThoughtWorks technology radar in 2011 as a philosophy mature enough to be adopted in projects. Since then the provisioning tools has matured even further and have became more enterprise grade. The most commonly used configuration management software as of today are: Puppet, Chef and Ansible.

I have chosen Ansible from the above list as it seemed to have a very low bar of entry. It is an agentless solution, meaning you install the Ansible application on your "Control Machine" and no additional software is needed to be installed on the nodes which are being provisioned, just standard SSH access. The control machine has access to the provisioning scripts called "playbooks" and it uses SSH to execute the playbooks on remote nodes. The playbooks are written in yaml, and can be extended with new modules in any programming language. The built-in modules have been more than adequate to serve my provisioning needs so far, and I was able to quickly script my playbooks in yaml using the Emacs yaml-mode plugin. I found the following resources to be very good introductions to the topic:


To illustrate the power of the technology, with the help of my custom Ansible playbooks I can provision the 4-node cluster in less than 2 hours from scratch. It takes about 30 minutes to flash the memory cards, perform basic network setup and install Ansible on the controller node, then the scripts take control and install basic tooling, NFS, collectd/Graphite-based telemetry, docker and noip/SSH-based remote access in 70 minutes without user interaction. The Raspberry Pi's slow SD card I/O is the main bottleneck in the process. On real world systems with modern SSD-s the provisioning would take much less time.

2017. március 14., kedd

Building a Raspberry Pi cluster - First Steps

3:58 Posted by Juhász Sándor No comments

Back in February 12th while browsing the net on Docker containers I found articles, YouTube videos from enthusiasts building their own Raspberry Pi clusters and experimenting with servers-side technologies. I decided to build my own cluster as a hobby project. This post is the summary of the past 1-month from the inception to the working blinking stack of Pi boards.

To build a Raspberry Pi cluster, you will need Pi boards, micros SDHC memory cards, a power supply, some networking hardware and a stack-able case. You can buy the whole package at https://www.picocluster.com or you can buy everything separately. To spare some money and to experience the thrill of building the whole tower from scratch, I ordered the parts from various web shops. It took exactly four weeks to select, order and get everything delivered, and my cluster was ready by March 11th.

It sounds easy, but selecting the right hardware may be a bit tricky. Fortunately there is a wealth of information on which hardware works and which does not. The source I used the most were Jeff Geerling's blog posts, YouTube videos and Raspberry Pi Dramble website.

As it turns out from Jeff's benchmarks, selecting the memory cards is maybe the trickiest decision. Don't be fooled by the numbers on the cards, the specifications list read/write speeds for most common use cases, namely recording HD video or taking photos with cameras. There is one or two orders of magnitude difference between the read/write speeds of various cards, especially for small-block I/O: the exact thing a Pi running MySQL does most of the time. Just to illustrate the difference, consider that the best performing card reads 4k blocks at 8 MB/s and writes at 3 MB/s due to the limitations of the platforms, even though the cards are all high-speed SDHC cards.

If you browse pictures online, most Pi clusters are powered through a USB hub. Selecting the right one is not trivial though. The 5.1V Pi-s consume 2.5A at its peak load, so I needed a power supply capable of delivering 51W of power. Many builds use the Anker 60W 6-Port wall charger, so I selected this one with 1 ft Anker USB cables. Many cheap USB cables are designed for data transfer and may not deliver the full 2.5 A to your device.

Even though the Pi 3 comes with on-board WiFi, I chose to connect the nodes with 100 MB/s wired Ethernet using a minimalist TP-Link switch. Regarding the rest of the hardware, I liked the design of the GeauxRobot stack so I ordered that from EBay. You can buy similar cheaper cases from China or if you are really purist, you can always use Mouser M3 20 mm spacers.

Assembling the hardware was very easy, less than 2 hours. All components are very high quality and everything fit together perfectly.

I flashed stock Raspbian Jessie with Pixel for node 1, and Jessie Lite for the rest of the nodes. I was glad to see how well-integrated the default Raspbian distribution and the whole ecosystem is around it . The https://www.raspberrypi.org homepage is very clean and well written, I have download and flashed the images in minutes. The system is ready to be used in seconds after booting it up, with Chrome, LibreOffice, Java 8, Python 2 and 3 and various other software development tools. Even Wolfram Mathematica is included for personal use! If you wish to read ideas on tinkering projects, check out the official MagPi magazine. https://www.raspberrypi.org/magpi/

Booting up the nodes the first day they arrived.

The nodes joined my LAN with DHCP right away. After setting up host names and fixed IPs for the nodes, the cluster was ready to be used. SSH and VNC are disabled by default but they can be easily enabled from the settings menu. But please change your pi password as malicious SSH connections are pretty frequent in the wild - I'm speaking from experience, but this is the topic of another post.

SSH, VNC up and running.
The total build cost was a bit more than 400 USD, see the full shopping list below.

 Shopping list