Hi,

today I spent most of my day setting up a local cluster for kubernetes. I thought this would be nice and simple but It wasn't, let me tell you why.

The reason why I wanted to have my own custom cluster is mostly because I don't like popular solutions based on virtual machines. I have always done all my development in a cluster with Kind and that has been awesome, Kind is light weight and easy to setup. However, Kind uses docker containers and It works only locally. For my application, I need to test It on various cluster topologies, especially in multiple nodes running different kernels. I have this requirement because my application behaves differently based on the number of running kernels in the cluster since I need to load eBPF programs in the kernel.

Well, I tried to use Minikube but both my laptop and my Desktop would completely freeze. Minikube is incredibly resource hungry and a lot of the drivers do not work at all. Additionally, I have so many problems on running particular topologies with multiple nodes and different images. I think Minikube is slow and heavy, and buggy. The alternatives do not satisfy me for similar reasons: I knew that I could setup a simple and light cluster on my own and have any kind of configuration I wanted, like 2 nodes on my desktop, one on my laptop and another one on my other laptop for example. So I started setting up everything by myself.

My vision is to have any number of Linux images running via qemu and KVM, all connected to the same network. The images would then run the various kubernetes daemons and I would control everything from a tmuxed terminal. All of this is managed centrally using a bunch of scripts. Since I am bootstrapping everything, I have full control on any part of my cluster, making It simple to hack. However, I have to say that setting up this did not went exactly smoothly.

To make the machines communicate with the internet and themselves I needed to setup some tap interface and connect them via a bridge. You need to do this correctly in order to make everything work, my final script to bootstrap the interface is the following:

#!/bin/bash
set -e

echo "Creating bridge br0..."

sudo ip link set br0 down 2>/dev/null || true
sudo ip link delete br0 type bridge 2>/dev/null || true

sudo ip link add name br0 type bridge
sudo ip addr add 192.168.50.1/24 dev br0
sudo ip link set br0 up

echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
sudo iptables -t nat -A POSTROUTING -s 192.168.50.0/24 ! -o br0 -j MASQUERADE

for i in 0 1; do
  tap="tap$i"

  echo "Setting up $tap..."

  sudo ip link set "$tap" down 2>/dev/null || true
  sudo ip tuntap del "$tap" mode tap 2>/dev/null || true

  sudo ip tuntap add dev "$tap" mode tap
  sudo ip link set "$tap" up
  sudo ip link set "$tap" master br0
done

echo "Bridge and tap devices ready."

I also made a script to clean everything and can be found on the project's repo. Then I needed to setup a static IP in each machine and … It didn't quite work. The problem was that I also needed to set a different MAC address for each virtual machine or the TCP/IP stack could not route the packets… that took a LOONG time to figure out because nothing hinted that this was the problem. I basically tried random things for 3 hours until this worked, wonderful.

Thankfully I endured that and now I am here to tell a beautiful story instead of being ashamed of my own incompetence. Anyway, now I have a bunch of VMs all connected to the same network that can ping each other and the internet. Next time I will make so that they will automatically install k3s (a kubernetes distribution) and we should be good to go.

Another interesting thing I learned is cloud-init which helps setting up the userspace of the VMs from an external config file, which looks like this:

#cloud-config
users:
    - name: fedora
      groups: user,wheel
      hashed_passwd: ...
      lock_passwd: false
      sudo: ALL=(ALL) NOPASSWD:ALL
write_files:
  # Enable password authentication in ssh
  - path: /etc/ssh/sshd_config.d/20-enable-passwords.conf
    content: |
      PasswordAuthentication yes
runcmd:
  - systemctl restart NetworkManager
  - systemctl restart sshd
  - nmcli connection up static-ens3

I find this very useful, thank you Fedora for letting me discover this.

The rest of my day was spent signing an enormous amount of documents for my internship and doing physics exercises at the blackboards in my university until closing hour, had a lasagna and now back to study physics.

– Giovanni

Travel: Ephemeris, Index