I have recently been experimenting with Kubernetes, starting with a managed cluster.
Quickly enough, I’ve wanted to try setting up my own cluster though.
Not for production use, I’m not crazy. But I learn by practicing things, and I figured if I wanted to learn more about k8, I needed to get my hands dirty.
So at the end of december, I ordered the parts I needed, and as they slowly arrived during the holidays, I’ve been building my own Kubernetes cluster on Raspberry Pis.
There is a lot of people describing how they built their own clusters, and we probably don’t need an additional one.
The reason I am writing this article is because I’ve been trying to automate the provisioning of my cluster, to be able to dump it and recreate it in under an hour.
Just show me the code! Sure. There you go: github.com/dmathieu/kubepi
The parts
I didn’t do a lot of research before ordering my cluster parts. So I’ve been following the recommendation from the kubernetes.io blog post.
My amazon order had the following (I am giving you my exact order. So that’s on amazon.fr):
- 5x Raspberry-Pi 3
- 5x 16Gb SD Memory Card
- 1x Ethernet cables (pack of 5)
- 1x Ethernet Switch
- 1x USB electrical charger
- 1x Micro USB cables (pack of 4) - I had a spare cable to use as the 5th one.
- 3x Stackable Raspberry Pi Case
The network
While this is not a production-grade cluster, I’ve wanted to build it the most production-like possible.
So only the nodes that need explicitely to be available from outside have WiFi access. All other ones are on their own ethernet subnetwork.
Node-2 to Node-5 are standard kubernetes nodes. They don’t have wifi access, and get their connection from iptable rules on kubemaster. They are therefore unreachable unless you have a physical connection on the Ethernet router.
Kubemaster (the cluster master) and Node-1 are a bit special. They are both connected to the wifi and therefore accessible from the outside.
- Kubemaster needs wifi access so I can operate the cluster without needing to connect to the Ethernet. It also need to be able to serve as an internet proxy for all the other nodes.
- Node-1 serves as a Load Balancer. It hosts an nginx pod forwarded to ports 80 and 443. Any service that needs to be exposed publicly will be available through it.
On top of that, my machine has dnsmasq configured to point *.**.kube to the Load Balancer.
So I can setup any hostname on the services I configure within my cluster, and call them with nice hostnames from my machine.
With this setup, I feel like I am reproducing what a production cluster inside a VPC would look like.
The provisioning
As I mentioned earlier, my goal here is to be able to reprovision the cluster easily.
I want to be able to play with this cluster, and break it without having to worry too much about setting it up again.
This is where kubepi comes in.
It is very simple in itself. There are 3 shell scripts that can do all of the provisioning.
While still too manual in my opinion, it makes the provisioning much easier than if I had to do it by hand.
image.sh
This script downloads raspbian and configures an SD card, which needs to be connected directly on my laptop.
It also enables SSH access on the SD card, and modified the boot sequence to enable cgroups.
master.sh <node ip>
This script configures the master node. It is the most manual part, and needs to be the first too.
I will connect the raspberry pi to my ISP router through Ethernet (I don’t have a choice, as it’s a brand new SD card, it can’t connect to the Wifi yet), and I will find it’s IP address with nmap -sP 192.0.0.255/24.
I will then run the script, which will do the following:
- Connect the node to the wifi
- Install a DHCP server and configure the Ethernet static IP address to be
10.0.0.1 - Install docker and kubernetes (but not configure them)
At this point, I will need to unplug the board, connect it to the final Ethernet switch it will be on and reconnect it.
This is because even though we have setup a static IP address for Ethernet, my ISP router overrides that, being the network DHCP server.
If we configured kubernetes right away, it wouldn’t know the right IP addresses and wouldn’t configure itself properly.
After restarting the node, I run the script again. It will skip all the already executed steps and configure Kubernetes, using kubeadm.
At the end of the configuration, I fetch all the master configuration certificates and configure kubectl locally.
At this point, I have a running cluster with only a master node.
node.sh <name> <node ip> [–ingress]
This script configures a standard node. The name argument will be the node name. Each name needs to be unique.
I am using DC Comics Super Heroes. So far, I have wonderwoman, raven, superman and batman.
It will install docker and kubernetes. Then, I will send the master certificate configuration so it can perform the join dance.
At the end of the configuration, the node is connected to the cluster and can start launching pods.
For one of the nodes (or several if you want multiple LoadBalancers, but that wouldn’t be very useful), I pass the --ingress option to the script.
The node then becomes the cluster’s load balancer.
In addition to the normal node configuration, it also gets the following:
- It connects to the wifi
- It installs ingress-nginx - I am using a custom manifest here, because there is no official ones that support ARM.
- It forwards the 80 and 444 ports on the machine to the Nginx Pod.
What’s next
As I mentioned, my goal for this is to learn more about Kubernetes.
So now, I want to start better understanding how the kubelet process runs. With this relatively easily replacable cluster, I can start doing so (though at this point, booting a patched version would be quite tricky).
There’s a bunch of other cool things I have for this cluster that I may talk about here in the future. Stay tuned!