After setting up lots and lots of virtual machines and container-based workloads on my NUC cluster, I have made many changes to the underlying infrastructure since my last post.

Harvester

Using the community-maintained Proxmox docker machine driver turned out to be pretty finnicky and I would frequently end up throwing away nodes that never got started properly. Because of this frustration and the fact that most of my workloads are container-based rather than VM-based, I decided to try out Harvester, Rancher’s new virtualization platform based on KubeVirt. The Harvester nodes themselves run k8s on metal, and then VMs are provisioned in KubeVirt within that k8s cluster.

Storage

Whereas Proxmox utilized Ceph for multi-node shared storage, Harvester defaults to Rancher’s own Longhorn, a distributed block-storage solution that automatically takes care of placing replicas across my nodes.

Longhorn is much simpler to manage, and comes with a nice dashboard showing scheduled storage vs used storage, as well as a listing of all volumes in the cluster and their status:

Additionally, it is very easy to configure Longhorn to backup volumes to S3-compatible storage. I back up my important PVs on a recurring schedule to Backblaze B2.

Templates

Proxmox templates had basic support for cloud-init, but were totally incomplete compared to the advanced cloud-init configuration that Harvester allows.

For example, I can create a cloud-init template for setting up Docker in an Ubuntu VM, and it looks like this:

#cloud-config

groups:
  - docker:
      - ubuntu

apt:
  sources:
    docker.list:
      source: deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

package_update: true

packages:
  - docker-ce
  - docker-ce-cli

This cloud-init yaml can be added to a “Cloud Config Template” in Harvester, and then just selected at VM creation to end up with a complete installation of Docker in the VM without any extra steps.

It is also possible to add SSH keys directly to Harvester, and then choose those during VM creation to have them added to the default user. This also works via cloud-init but is totally transparent – if all you care about is setting SSH keys, you don’t have to worry about learning the cloud-init syntax whatsoever, as Harvester handles setting up the relevant YAML to inject the keys.

Cluster Management

Rancher has built-in support for “cluster management”, as described in the initial NUC cluster article. However, in that article, I had to use a community-maintained docker machine driver for Proxmox in order to provision k8s nodes automatically, and it was a bit flakey. Now that I have switched to Harvester, there’s first-class support for provisioning k8s clusters in Harvester VMs right from my bootstrap node.

The cluster management page makes setting up k8s clusters very simple, and has a nice UI for setting up pools of workers and control nodes:

Downsides

Harvester is a brand-new product and is definitely a bit rough around the edges. I would not recommend it (yet!) to anyone without a good amount of Kubernetes experience. I have had to debug Harvester internals quite a few times but have not yet hit an issue I was not able to solve on my own or via referencing an issue in their GitHub issue tracker.

Simplifying the Bootstrap Cluster

The bootstrap cluster as setup with kind and caddy was alright, but did not feel very stable and was more complicated than I wanted it to be.

In order to reduce complexity here, I abandoned the original VM in favor of a plain Ubuntu 20.04 VM running RKE2 (Rancher’s Kubernetes distribution, which can be set up with a one-liner) and just setting up the SSL certificate using cert-manager inside of the cluster.

The relevant k8s state for that certificate, along with a LetsEncrypt issuer, can be found in my nuc-cluster repo, under clusters/bootstrap.

Simplified Workload Cluster Deployments with helmfile

The k8s state for my main workload cluster (comprised of VMs running in Harvester, managed by Rancher’s cluster management), also lives in the nuc-cluster repo, under clusters/workload.

helmfile allows me to declaratively state (in YAML files, of course) Helm releases that should be applied to my cluster.

For example, the helmfile.yaml for my deployment of Miniflux (my feed reader of choice) looks like this:

repositories:
- name: k8s-at-home
  url: https://k8s-at-home.com/charts

releases:
- name: miniflux
  namespace: miniflux
  chart: k8s-at-home/miniflux
  version: 4.5.0
  installed: true
  values:
    - values.yaml
  secrets:
    - secrets.yaml

This file tells the helmfile tool which repository to grab the Helm chart from, as well as its name and version, and a path to values files to set the right options in the chart when it is being deployed.

Closing Thoughts

I’m happy with Harvester right now and the direction it is headed. I would like to check out TrueNAS Scale once it’s stable enough to try, but for now I couldn’t even get it to boot in a VM…

Let me know if you have any thoughts or questions by commenting below.