Hello everyone.
Man, it sure has been a while since we last spoke, huh… I barely wrote anything this whole year, despite having a mountain of time on my hands. I do apologize for that, gentle reader.
On the homelab front, a lot has changed: last we spoke, i had just built my new home server with the Threadripper CPU. It’s a wonderful machine, it has had very few issues in the intervening year, but some more… structural issues have cropped up.
I will just relate to you how the lab changed over time: It is much more advanced now, with much better hardware and networking, but it was a long journey of silly decisions and peace-meal upgrades.
It all began a few months after i wrote that update: the root SSD in my server died and i was forced to start over from scratch 💀
I swore to never let this happen again, and so i bought a little Ugreen NAS, put some drives into it, and flashed it with Proxmox Backup Server. It has been running great so far, and i actually back up all my client machines to it using the official client.
It is a slightly unorthodox approach, but nothing some custom Python scripts and systemd units can’t fix.
Having safeguarded my data, I looked at power usage and uptime: In brief, i felt kinda bad leaving the Threadripper machine running 24/7 since it idled at 200+ watts, so i turned it off at night or when it was not needed.
In order to solve this issue, the one Proxmox server became a 3-node Proxmox cluster: I added 2 mini-pcs (both based on the same Ryzen 3000 laptop CPU), and made a Proxmox cluster with High Availability. The mini-pcs host all the VMs that need serious uptime, while the big boy is reserved for anything that needs hardware passthrough like Jellyfin transcoding, local AI, and experimenting with stuff.
The next step in the journey was getting a dedicated NAS: until that point i shoved all my storage into the big Threadripper machine and used a TrueNAS VM to serve NFS shares to other services. This worked fine for a single machine, but now that some services were on the mini-pcs, the NFS volumes they needed would die as soon as the big machine turned off for the night. I briefly considered re-using an old machine, or building a NAS with CPUs and mobos i had lying around, but i quickly decided against that and went with another NAS box.
You may be starting to sense a pattern here…
I actually hate building computers: some people call it Lego for nerds, but I personally find it a very fiddly and stressful process. I will do it when i absolutely have to (buying a pre-built gaming computer, let alone a Threadripper workstation, would have been outrageously expensive), but i try to avoid it otherwise
And thus, i waited for Prime day and i set off in search of a little box: i would have been fine with a little dingus N100 NAS (serving NFS is not particularly difficult) but the only good offers i could find were more Ryzen 3000 machines, so i went with one of those. Once it arrived, i felt extremely guilty about using an 8-core part to just run a NAS, so i turned it into Proxmox node #4! Yay!
Yes, i am acutely aware of the fact that a 4-node cluster provides absolutely no benefit over a 3-node cluster, but alas that is just how the chips have fallen)
Once that was done, i immediately set upon ripping ghastly TrueNAS from my life: My dislike of TrueNAS has not changed since i talked about it in a previous post, so i replaced it with an Ansible playbook that does the same stuff but on bare Debian 13; the features from TrueNAS that i actually need are very small, so it’s basically some plumbing to use ZFS DKMS modules, an NFS server, and some alerting and monitoring 🙂
Next up, networking: i had been using a dumb switch i found, but the number of devices was getting out of hand and i wished to learn more about network configuration/management. I went for a pair of MikroTik 2.5G switches: I already owned a Mikrotik router/AP unit, and i quite liked Mikrotik for their relatively open philosophy (it’s based on Linux, they do not gate any features behind more expensive products etc.). The 2.5G thing was more of a compromise, it is faster than 1GbE, yet i had heard that 10GbE installations usually prefer using fiber optics: I was already a networking noob, so I did not want to add too much complexity at once.
Armed with more enterprise-y networking hardware, i could finally implement VLANs, proper firewalls and proper routing between the lab and the rest of the network: it was quite challenging to wrap my head around, but with some help from some wise discord users i figured it out.
To my delight, i discovered that not only does RouterOS have an API you can use to interact with it, this API is also leveraged by the Ansible module for it: I got to centralize all the configuration for all Mikrotik devices, and in a way that is highly reproducible and idempotent to boot!
I am quite happy doing configuration this way, even though my workflow is not fully declarative: a tool like Terraform is probably better suited to this, since the network configuration changes rarely, i should treat it more like an appliance, but it will do for now. I actually tried to port my existing configuration to Terraform/OpenTofu, but i ran into some issues with importing the state and i decided this is a job best done when/if i’m willing to start from a blank slate.
Where to go from here
Overall i am fairly happy with the way the lab turned out: it feels much more robust now, and the networking in particular is quite an improvement.
As for short-term goals, i will have to admit the memory and storage situation is kind of a mess: each node has anywhere between 16 and 128 GB of memory, and anywhere between 256 GB and 2 TB of storage. I would like to standardize on a minimum spec of 32GB of RAM and 1TB of storage across the cluster, but given the current shortage this may prove challenging…
In the longer term, i am toying with the idea of building a 16-18U rack: i know very little about this stuff, but i think it would be an improvement in terms of how much floor space the lab takes: if i could consolidate my gaming PC and all 4 nodes into a single rack, that would also more easily allow me to put them behind a UPS. That said, this involves taking apart and re-building PCs, so motivation has been an ongoing issue lol.
I plan to make a new article with some more thoughts that i have on the software side of things in the near future, so i will hopefully see you soon.