Another blog from another guy called Matt
Occasional blog posts from a random systems engineer

Homelab history

· Read in about 10 min · (2063 Words)

My Infrastructure History: From Under the Bed to the Cloud (and Back)

I’ve been running my own infrastructure for a long time. Looking back at the notes I’ve kept, the path from a noisy hobbyist setup to a modern orchestration platform wasn’t a straight line. It involved a lot of failed experiments, loud fans, and learning lessons the hard way.

The Absolute Beginning

It all started with a Packard Bell Club. It had a massive 64MB of RAM. By today’s standards, that’s nothing, but back then, it was my first exposure to running a “server.”

I tried to be fancy early on. I got my hands on an IBM server and attempted to do 32-bit virtualization. It failed. I tried my hand at Linux for the first time. That failed too. But those failures were necessary. They taught me that I had to crawl before I could run a data center.

The ESX Era (The Windows Years)

After those initial stumbles, I eventually got serious with VMware ESX, but I was still firmly planted in the Microsoft ecosystem.

  • The Stack: During this period, my entire environment was Windows-based. I was running Windows Server 2003 Active Directory for identity, Windows ISA Server for the firewall, and even my build automation relied on Windows PXE builds.
  • The Constraints: I was running this on RAM-constrained machines like the HP ProLiant DL380 G3s and DL360 G3. I remember “shovelling” Windows Server 2003 instances onto hosts. I was giving VMs as little as 128MB or 256MB of RAM, just to squeeze them onto a box that had a whopping 8GB of RAM.
  • The Physical Reality: The rack didn’t live in a climate-controlled data center; it lived under my bed.
    • The Noise: It was loud.
    • The Heat: During hot summers, the fans would go absolutely berserk trying to keep the systems cool. I would wake up in the middle of the night to the sound of jets taking off, realizing the room was baking.
  • The Upgrade: It wasn’t until I moved to a new bedroom and could build a dedicated cupboard for the gear that things got slightly more manageable. Around this time, storage moved to a Whitebox PC running FreeNAS 8.9, serving up iSCSI and Samba shares for my media and backups.

The Expansion Phase (2012–2016) — The Big Switch

By 2012, the bedroom cupboard wasn’t enough anymore, and I was ready for a massive architectural change. I moved the operation out of the house entirely to an external building where I set up a couple of racks, but more importantly, I made the Big Switch to Linux.

  • The Pivot: I dropped the Windows Server software stack entirely. No more Windows AD, no more ISA Server, no more Windows PXE. I moved everything to Linux.
  • Hardware Upgrade: I shifted focus to IBM x3455 servers (eventually accumulating about two to four of them), while keeping the older DL380 G3s running as the backbone.
  • Networking & Access: In 2012, I was managing complex network segmentation using pfSense (running on BSD/Linux). I set up multiple captive portals with “fancy” custom logins just to authenticate and jump between different networks. It was a complicated setup, but it taught me a lot about network security.
  • Experimentation: This era was defined by playing with multi-server labs. I spent a lot of time working on PXE booting experiments (now Linux-based) and setting up early KVM virtualization. I even tried using low-power laptops as compute nodes to test the concept of “elastic compute.”
  • The Setup: I was getting serious enough that I ran an Ethernet cable out the window just to get a temporary uplink to the outside world. This was also when I started dabbling with early oVirt exploration to see if I could manage the resources more effectively.

The Power-Saving Experiment (Pre-Datacenter) Before I made the jump to the datacenter, electricity costs became a real concern. I needed to shut down the power-hungry rack but keep the core VMs running.

  • The Setup: I took the original VMs, extracted their root filesystems, and placed them on the fileserver (the 4x3TB unit).
  • The Execution: I set up a PXE boot environment. About 3 ThinkPads would boot over the network, mount their root volumes via NFS, and run the core infrastructure directly from the fileserver. It was a fragile, low-power setup, but it kept the lights on without spinning up the massive rack.

The Datacenter Experiment (2017–2018)

For a brief period, I graduated from the external building to a real facility. I rented rack space and deployed a formal cluster.

  • Hardware: I moved to HP DL160 G6 servers and introduced a Nexsan SATABoy SAN with roughly 14TB of storage.
  • Platform: I used oVirt to manage the virtualization.
  • Access & Networking: The complex pfSense captive portals were replaced by a cleaner OpenVPN setup for remote access to the datacenter environment.
  • Orchestration Begins: This is also where I first deployed Rancher to manage container workloads alongside the virtual machines.
  • The Outcome: While it was a great learning experience in enterprise networking and SAN management, the era was short-lived. By 2018, I repatriated the hardware, moving back to a consolidation model at home.

The “Big Squeeze” (Post-Datacenter)

After bringing everything back from the datacenter in early 2018, I couldn’t immediately jump straight to a new enterprise server. I entered a “squeezed” phase where I had to make do with what I had on hand.

  • The Hardware: I relied on a Synology unit for storage, an HP desktop with 32GB of RAM, and an IBM laptop.
  • The Reality: I was running VM hosts on consumer-grade laptops and desktops. It was a tight fit for resources, but it served as the bridge between the professional datacenter gear and the eventual home consolidation.

The Cloud “Edge” & Thin Swarm

While the datacenter experiment was happening, I was also building out a presence in the cloud. It’s important to note that for my public cloud/VPS nodes, I didn’t jump straight into Kubernetes.

  • The Strategy: I used a “thin” Docker Swarm setup spread across multiple providers (Hetzner, OVH).
  • Networking: I used a mesh VPN (Tinc) and tools like HAProxy to create a highly available web and database cluster.
  • The Goal: This provided redundancy for my public-facing services without the overhead of a full orchestration layer on every small VPS.

Local Container Orchestration (Rancher -> Nomad)

The heavy orchestration journey began in the datacenter and followed me home.

The Rancher / Kubernetes Era I brought Rancher and the local RKE (Rancher Kubernetes Engine) cluster home with me. It continued to run on the R720xd for several years.

  • The Stack: I used canal (Calico + Flannel) for networking, an nginx-ingress-controller, and a private Docker registry.
  • The Workload: This ran everything: GitLab, Nexus, Jenkins, Nextcloud, and my monitoring stacks.
  • The Issue: While it worked, managing a full Kubernetes cluster on local hardware felt like using a sledgehammer to crack a nut. The resource overhead for the control plane was significant.

The Migration to Nomad (2025–2026) Eventually, I made the architectural decision to migrate away from K8s to the HashiCorp stack (Nomad and Consul).

  • Why? Nomad provided me with a unified service plane that was much lighter weight. It could schedule Docker containers, legacy Java apps, and even QEMU VMs from a single interface.
  • The Result: I kept the robustness of orchestration but drastically reduced the complexity and resource footprint on my physical hosts.

Consolidation & Modern Hardware (The R720xd Era)

After the interim “squeeze,” I settled on a consolidation strategy. I replaced the sprawling rack with a single, powerful enterprise host: a Dell R720xd.

To keep the overhead low, I skipped complex management platforms and went back to basics, running the host on plain libvirt and qemu. This gave me direct control and let the orchestration layer handle the heavy lifting. With 384GB of RAM in this box, I finally stopped starving the VMs.

The Storage Evolution The storage backend went through several distinct iterations to get to where it is now:

  1. Synology iSCSI: Initially, I relied on a Synology DS1813+ connected strictly via iSCSI to handle all the VM storage.
  2. LVM Caching: To combat latency over the network, I added a local SSD layer using LVM cache. This sat in front of the iSCSI connection, speeding up reads and writes significantly.
  3. The SSD Array: I eventually moved away from networked storage for the hot data and loaded the server with 8x 500GB SSDs for raw performance.
  4. The ZFS Hybrid: Finally, I settled on my current setup. I moved to ZFS utilizing high-performance 10K RPM drives for capacity, paired with dedicated SSDs for ZFS Read and Write caching (L2ARC and SLOG). This gives me the IOPS of an SSD array with the capacity of spinning disks.

The Control Plane Interestingly, with Nomad, the heavy lifting is done by the R720xd (the client), but the actual control plane (Nomad servers) runs on smaller, distinct nodes. This separation keeps the scheduler lightweight and resilient.

How Other Things Changed

Identity I started with Windows NT Domains on that early hardware, then moved to Windows Server 2003 AD. After the Big Switch in 2012, I eventually deployed FreeIPA in 2016, and it remains my core identity provider to this day. I didn’t replace it; I evolved it. Later, I introduced Keycloak as a modern Identity Provider layer. Keycloak doesn’t replace FreeIPA—it federates from it. Recently, I migrated the Keycloak instance from a standalone VM to run directly in the Nomad orchestration layer for better resilience.

Hosting & Web Services My approach to hosting applications has evolved through the standard open-source stack. I started with IIS, moved to Apache, and then used control panels like Kloxo to manage domains. Eventually, I moved away from monolithic web servers entirely to Containers. This shift started on the VPS edge with Nginx, and later evolved to use HAProxy as the primary ingress controller.

Code & Infrastructure as Code My approach to managing code and configuration has completely flipped.

  • The ESX Era: Everything was completely hand-rolled. No version control, just scripts run manually.
  • Source Control: I eventually moved from manual files to SVN, then made the jump to Git.
  • Configuration Management: I adopted Puppet to handle configuration drift.
  • Modern IaC: Today, I use Terraform to provision the underlying infrastructure and Ansible for configuration management. It’s a fully automated, declarative pipeline.

CI/CD & Pipelines I’ve been using CI/CD tools since around 2012.

  • The Jenkins Era: Jenkins was the workhorse for over a decade. Over time, I split it into two instances—one public, one private—to handle different access requirements.
  • The Shift: Today, Jenkins is considered old and deprecated in my stack. I have moved towards GitLab, which now handles the CI/CD pipelines natively alongside the source code, simplifying the workflow significantly.

Monitoring Monitoring has gone through a standard open-source evolution, moving away from heavy proprietary tools to lightweight exporters.

  • The Path: I started with SolarWinds, moved to Nagios, then adopted Icinga.
  • The Current State: I am currently building a custom solution leveraging node exporter, stripping away the complexity of the previous suites to get the metrics I actually need.

DNS DNS has had a winding path. It started in Windows AD, moved to pfSense, and then settled on FreeIPA. But it didn’t stop there. The architecture has evolved to split the load: Consul now handles dynamic service discovery for internal and VPS traffic, while BIND manages the public-facing VPS edge. FreeIPA remains the authoritative master, but it’s no longer doing the heavy lifting alone.

Networking I moved from unmanaged switches to VLANs, and finally to a Zero Trust model. After the OpenVPN era (which lasted from the datacenter days through the consolidation), I have recently modernized access to include Netbird for mesh networking and Boundary for Zero Trust authorization.

Where I Am Now

My infrastructure today is a hybrid. I have a powerful R720xd at home running the heavy orchestration (Nomad/Consul), while the “Edge” runs the thin Docker Swarm for external HA.

I have moved on just a little from the Packard Bell with 64MB of RAM under the bed to a powerful, automated platform. Looking back, I realize I’ve just been rebuilding the same logical architecture over and over—Identity, DNS, Firewall, App Hosting—I just keep swapping out the engine for something more abstract. The focus has shifted from managing noisy hardware to managing services, finally allowing the infrastructure to do its job without waking me up at night.