Occasional blog posts from a random systems engineer
home
about
posts
🧾 1. INFRASTRUCTURE EVOLUTION TIMELINE (HIGH LEVEL)
Era
Approx timeframe
Key characteristics
Core systems
Early server era
(pre-~2007 → unknown)
First exposure to servers, NT domain
Compaq Windows NT server
Early homelab / ESX era
~2007–2013
Physical servers, early virtualization (RAM constrained)
DL380 G3 (x3), DL360 G3, ESX/vCenter, Whitebox NAS
Expansion / distributed experimentation
~2012–2016
Multi-server lab, PXE booting experiments, Jenkins (sheila)
x3455 servers, DL160 G6 (later), fileserver
VPS High-Availability Era
~2018
Multi-provider VPS cluster, Mesh VPN, HA MariaDB
Tinc, OpenVPN, Docker Swarm, HAProxy
Identity foundation era
~2016
First stable Linux identity system
FreeIPA (marge/homer)
Datacentre / colo era
~2017–2018
Formal cluster + SAN deployment
DL160 G6 cluster, Nexsan SAN
Consolidation era
~2018–2019
Transition (ThinkCentre) $\to$ R620/R720xd consolidation
ThinkCentre, R620, R720xd, Synology, Keycloak
Platform orchestration era
~2019–2024
Container orchestration, Rancher, early automation
GitLab (Rancher), Jenkins systems
Modern orchestration era
~2024–2026
Nomad-based unified service plane
Nomad, Consul, Vault, Keycloak migration
🧾 1.1 THE RANCHER/K8S ERA (RETIRED)
During this era, container orchestration was centralized through Rancher, managing Kubernetes clusters across the core nodes. This provided a GUI-driven approach to managing workloads but was eventually superseded by the more lightweight and unified Hashicorp stack (Nomad/Consul).
🛠️ Orchestration Stack
Platform : Rancher (RKE-managed Kubernetes)Versions :
naxos: v1.11.6-rancher1rhodes: v1.20.15-rancher1
CNI : canal (Calico + Flannel VXLAN)Ingress : nginx-ingress-controllerRegistry : Private registry at fare-docker-reg.dock.studios:5000Identity : Keycloak-integrated OIDC/RBAC
📦 Workload Inventory by Node
creteCore services and collaboration tools:
nextcloudnexusonlyofficeperforceseafileterrareg
naxosDevelopment and monitoring tools:
cachetfreeipaselfservicegitlabmattermostpromailgatesentry
rhodesCI/CD and data services:
giteahsbcint-jenkinsmatomoodotoracle-neo4jphabricatorsonarqubewiki
🧾 1.5 LEGACY SERVICE STACKS (PRE-2012)
Service Category
Era/Tool
Notes
Email hMailserver $\to$ Zimbra $\to$ G-SuiteLocal hMailserver (~2011) $\to$ Zimbra $\to$ Cloud (G-Suite).
Web & App Hosting Kloxo / Apache / IISManaged domains (loveyourmusic.co.uk, mhsolutions.co.uk, etc.). Hosted eyeOS, phpMyAdmin, and various custom apps via IIS/Apache.
Database Backend MySQL
Hosted application data (lymDB, subscribers, mailserver, etc.).
File Storage FreeNAS 8.9 (fileserver-01)Early storage via FreeNAS; provided iSCSI targets and Samba shares (media$, downloads$, applications$, etc.).
🧾 2. PHYSICAL / LEGACY SERVER HISTORY
System
Type
Role
Notes
Status
Packard Bell Club 30
Physical
First domain/server system
NT4-era
🔴
HP ProLiant 1U server
Physical
First dedicated server
Pre-virtualisation era
🔴
HP DL380 G3 (×3)
Physical
ESX cluster
VMware + vCenter (RAM constrained)
🔴
HP DL360 G3
Physical
Support compute
Same generation
🔴
Whitebox PC
Physical
File Server
TrueNAS/FreeNAS era
🔴
IBM x3455 (×2–4 approx)
Physical
Linux/KVM experimentation
Early oVirt exploration
🔴
Dell 2U server
Physical
Unused
Too loud / rejected
🔴
HP DL160 G6 (×3)
Physical
Datacentre compute
~Jan 2017 – Jan 2018 (Datacentre)
🔴
IBM ThinkCentre
Physical
VM Host
~May 2018 (Post-Datacentre)
🔴
ThinkPad
Physical
VM Host
~May 2018 (Post-Datacentre)
🔴
Dell R620 (×1–2)
Physical
Consolidation era compute
~June 2018 (Consolidation era)
🔴
hetznur01
Virtual
Former Hetzner node
Moved from cloud July 2018
🔴
VPS Cluster (Hetz/OVH)
Virtual
HA Web/DB services
Tinc, BIND, Consul, HAProxy, MariaDB
🟢
pfserver-04
Physical
Baddesley Fileserver
4 x 3TB drives
🟢
Nomad Cluster (tinos, etc)
Virtual
Current orchestration plane
Debian 12 (tinos), Ubuntu 18.04 (admts)
🟢
🧾 3. STORAGE & BACKUP EVOLUTION
Era/Phase
Technology
Role
Notes
Early Era
Local HDD
Primary storage
Local disks on servers
ESX Era
iSCSI (Home-built)
VM storage
VMware -> iSCSI
Laptop Era
iPXE + NFS
Boot/Root drives
Fileserver -> PXE -> Laptop servers
Datacentre Era
iSCSI (NEXSAN)
Enterprise SAN
oVirt -> iSCSI -> NEXSAN
Post-Datacentre
iSCSI (Synology)
VM storage backend
Libvirt -> iSCSI -> Synology
SSD Era (~2020)
Software RAID (SSD)
VM performance layer
Libvirt + SSD RAID (with LVM cache LVs) -> iSCSI -> Synology
ZFS Era (2026+)
ZFS + Enterprise HDD
High-perf/High-cap
Libvirt + ZFS (10K drives) + SSD L2ARC/SLOG -> iSCSI -> Synology
💾 Backup Strategy Evolution
Era
Method
Flow / Tools
Notes
Tape Era (~2011)Symantec Backup Exec
Black server + LTO-2 Tape drive
Inherited server, found/cheap LTO-2 drive; physical tape rotation
Modern Tiered (Current)Rsync + Cloud
hans -> pfserver-04 -> Backblazehans-solo acts as MongoDB backup DB
🧾 4. NETWORKING EVOLUTION
System / Era
Type
Role
Notes
Status
Pre-2013 (Unmanaged) Hardware
Basic connectivity
3× 10/100 3Com switches + Netgear Fibre switch (unmanaged)
🔴
2013 – Datacentre (Managed) Hardware
Core switching
3× 3Com HP Baseline 2928-SFP Plus (VLAN/STP)
🔴
Interim Period Hardware
Core switching
4× Stacked D-Link DGS-3324SR
🔴
Post-Datacentre Hardware
Managed switching
Netgear 8-port managed switch
🔴
Tinc Mesh
Networking
Multi-provider VPN
Peer-to-peer mesh connectivity
🟢
OpenVPN
Networking
Home-lab interconnect
VPS ↔ Home-lab connection
🟢
Boundary
Networking
Zero-trust access
Identity-aware proxy/gateway
🟢
🧾 5. VIRTUALISATION / PLATFORM HISTORY
System
Type
Role
Notes
Status
VMware ESX (DL380 G3)
Virtualisation
First VM cluster
vCenter-managed
🔴
oVirt (DL160 G6 era)
Virtualisation
Datacentre cluster
Jan 2017 – Jan 2018 (Pre-R720xd)
🔴
ThinkCentre/ThinkPad Era
Virtualisation
VM Hosting
Pre-R720xd transition
🔴
HAProxy/Bind/Consul
Orchestration
HA VPS cluster
Multi-provider cluster
🟢
Rancher
Orchestration
Container platform
GitLab hosted here
🔴
Nomad
Orchestration
Current scheduler
Key migration target
🟢
The R720xd serves as the primary consolidation host for the majority of the service plane.
🟢 Running VM Catalog (Categorized)
Category
Key Systems
Infrastructure & Network cosh-fw-1, cosh-host-5 (Libvirt + Auth API), admts, apu, admin-vpn-interconnect, remote-interconnect, hetz-vpn-interconnect, init-connect-vpn-interconnect, netbird-gateway, inthetz, hetznur01, docker-man-1
Identity & Security marge, homer, keycloak-1, albali (Manual Vault), vault cluster (lemon, orange, lime), crypt (Nexus/Apt)
Consul Cluster banana, apple, cucumber
Nomad Cluster Servers : raspberry, strawberry, blueberry; Clients : thassos, syros, gordon, kasos, anafi, printer (Receipt API), coder (Coder dev environments), zante (Isolated Nomad)
CI/CD & Dev Platform fare-jenkins, pub-jenkins, ci-host-1, harbor, fare-docker-reg
Application/Service Layer cartman, miram (S3/Terrashine), hans (Rsync/Backup), hans-solo (MongoDB/Backup DB), sync-disc
🔴 Decommissioned / Legacy VMs
Category
Systems
Windows/Legacy w2k3-build-01, w2k3-dc-1, w2k3-dc-2, wilson, sharon (W2K3 Replacement), randy (W2K3 Replacement)
Replaced / Obsolete assassin-interconnect, auth-proxy-1, ff-proxy, cuddy, dockglu-1/2/3, dockheketi-1, fare-docker-1/2/3/4, fare-docker-store, fare-rancher, gatekeeper-1, tarvos (Test Firefly-III)
Other Legacy apu (Puppet Master), arnie (Hetzner Node), billet (R720xd VM), buid-osx-1 (OS X Build Slave), cameron (R720xd VM), che (Eclipse Che), chief-wiggum (Wazuh Security), cloud-dr-1 (Hetz Replication), corfu (Web App 4 - R720xd), cosh-fw-2 (Datacenter/Test Firewall), cosh-fw-3 (Test Firewall), crete (PHPipam/Portainer - R720xd), fare-fs-1 (Media Syncthing), kyle (Icinga2 Master), lemnos (R720xd VM), metis (R720xd VM), mob-vpn-interconnect (Mobile VPN), mr-bump (FreeIPA Client - R720xd), naxos (FreeIPA Self-Service - R720xd), ross (Icinga2 Distributed), samos (Rancher - R720xd), sec-scanner (Security Scanning?), sherlock (Location APIs)
🧾 7. IDENTITY / IAM SYSTEMS
System
Type
Role
Notes
Status
Windows NT Domain
Identity
Early auth system
Compaq era
🔴
FreeIPA (marge/homer)
Identity
Linux identity domain
CentOS, deployed Jun 2016
🟢
Keycloak-1
IAM
Legacy authentication
Deprecated VM instance
🟡
Keycloak (Nomad)
IAM
Primary auth system
login.dockstudios.co.uk
🟢
HashiCorp Vault (albali)
Secrets
Secrets management
Oct 13 2023 FS created
🟢
🧾 8. CI / DEVOPS SYSTEMS
System
Role
Notes
Status
pub-jenkins
Public CI
GitLab → GitHub sync, public builds
Ubuntu 18.04, Oct 2018 FS
fare-jenkins
Internal CI
Private pipelines (PrivateJenkins-Config)
🟢
ci-host-1
CI executor
Build worker
🟢
System
Platform
Role
Notes
Status
GitLab
Rancher
Source control platform
Jul 15 2019 initial deployment
🟡
GitLab
Nomad
Orchestrated platform
Migration ~Jun 6 2026
🟢
🧾 10. EDGE / EXTERNAL CLUSTER
Node
Provider
Role
Notes
Status
hetznur02
Hetzner
Cluster node
Debian 11 (updated May 2026 from Buster). Managed via PortainerConfigs
🟢
hetzfal01
Hetzner
Cluster node
Debian 11 (updated May 2026 from Buster). Managed via PortainerConfigs
🟢
ovhlim01
OVH
Cluster node
Debian 11 (updated May 2026 from Buster). Managed via PortainerConfigs
🟢
🧾 11. NETWORK / SECURITY INTERCONNECT
System
Role
Notes
Status
admin-vpn-interconnect
VPN
Admin access
Debian 9 (Stretch) created Nov 2018 (formerly pfsense fare-fw-1)
remote-interconnect
VPN
Remote access
Debian 9 (Stretch) created Jan 2018 (formerly pfsense fare-fw-1)
hetz-vpn-interconnect
VPN
External link
Debian 9 (Stretch) created Nov 2018 (formerly pfsense fare-fw-1)
init-connect-vpn-interconnect
VPN
ComboPod component
Debian 9 (Stretch) created Nov 2018
netbird-gateway
Mesh VPN
Zero-trust networking
hetz-haproxy
Proxy
Cluster Load Balancer
Hetzner cluster ingress
🧾 12. LEGACY BUILD / WINDOWS 2003 PIPELINE
System
Role
Notes
Status
w2k3-build-01
Build server
Jenkins dependency system
🟡
w2k3-dc-1
Domain controller
Build auth system
🟡
w2k3-dc-2
Domain controller
Redundant auth
🟡
🧾 13. KEY ARCHITECTURAL TRANSITIONS
Transition
From
To
Meaning
Identity
NT domain
FreeIPA
Linux identity foundation
IAM
VM Keycloak
Nomad Keycloak
Orchestrated identity
CI exposure
internal only
pub-jenkins
public build gateway
GitLab hosting
Rancher
Nomad
platform consolidation
Compute model
VM-centric
orchestrated services
infrastructure abstraction
Proxy
Nginx
HAProxy
Better HA and config management
VPN
OpenVPN
Boundary
From clunky VPN to zero-trust
Orchestration
(Internal) Swarm
Nomad
Core cluster migration to Hashicorp stack
Security/Firewall
Various (ISA/VM)
pfsense / Distributed
Persistent network security layer
Virtualisation
oVirt
Nomad/Containerized
Transition from VM cluster to orchestration
Storage
Local/NAS
ZFS + iSCSI
From local disks to high-perf distributed storage
This section traces the evolution of key application domains, showing how the tools and hosting environments have matured over the years.
Domain
Early / Legacy Era (Physical/ESX)
Consolidation / Rancher Era (K8s/VMs)
Modern Nomad Era (Orchestrated)
Identity & Auth Windows NT Domain
FreeIPA (marge/homer)
Keycloak (Nomad) + Vault
Orchestration Manual / ESX / KVM
Rancher (RKE/K8s)
Nomad + Consul
CI/CD Jenkins (sheila)
Private/Public Jenkins
GitLab (Nomad)
Version Control Manual / Local
GitLab (Rancher)
GitLab (Nomad) / Forgejo
DNS/Networking BIND (Manual)
BIND + Consul (Early)
BIND + Consul (Dynamic) + Netbird
Ingress/Proxy Apache / Nginx
Nginx Ingress (K8s)
Traefik (Nomad)
Storage Local HDD / NAS
iSCSI (Synology/Nexsan)
ZFS + iSCSI (Synology)
🧾 14. METHODOLOGICAL EVOLUTION
🛠️ Provisioning & Configuration Management
Era
Method
Key Systems
Notes
Manual Era (~2011–2014)Direct SSH / Manual
x3455, DL160Early manual provisioning
Config Management Era (~2015–2020)Puppet / Ansible
apu (Puppet Master)Automated configuration via Puppet/Ansible
IaC Era (~2021–Present)Terraform / Ansible
cosh-servers (Terraform)Declarative infrastructure via Terraform
📦 Artifact & Registry Management
Era
Method
Key Systems
Notes
Local Era (~2011–2015)Local FS / Manual
fare-fs-1Manual artifact management
Repository Era (~2016–2022)Nexus / Apt
cryptCentralized repository management
Registry Era (~2023–Present)Harbor / Container Registry
harborContainer image orchestration
👁️ Observability & Monitoring
Era
Method
Key Systems
Notes
Active Monitoring Era (~2012–2018)Icinga2 / Nagios
kyle, rossLegacy host-based monitoring
Centralized Era (~2018–2023)Checkmk / Monitoring
[TBC]
[TBC]
Modern Observability Era (~2024–Present)Prometheus / Consul / Nomad
[TBC]
[TBC]
💻 Development & Workspace Environments
Era
Method
Key Systems
Notes
Local/Jenkins Era (~2011–2017)Jenkins Slaves / Manual VMs
buid-osx-1, ci-host-1Early CI/CD execution
Containerized Era (~2018–2023)Rancher / Docker Swswarm
fare-docker-*, samos (Rancher)Transition from non-docker (cuddy) $\to$ fare-docker $\to$ Rancher
Ephemeral/Cloud-Native Era (~2024–Present)Coder / Nomad
coderDeveloper environment orchestration
🧠 15. CURRENT ARCHITECTURAL STATE (SUMMARY)
You currently operate a consolidated, orchestrated infrastructure based on the following layers:
🟢 Orchestration layer
Nomad (core scheduler and service plane)Consul (service discovery and KV store)
🟢 Identity & Security layer
Keycloak (primary identity provider, Nomad-hosted)FreeIPA (legacy identity dependency)HashiCorp Vault (secrets management and control plane)Boundary (zero-trust access gateway)
Jenkins (split between public and internal pipelines)GitLab (source control and orchestration, Nomad-hosted)Coder (ephemeral developer environments)
🟢 Compute Layers
Core Compute
R720xd Consolidation Host : Primary host for the majority of the service plane.
Edge Compute
3-node symmetric cluster : Multi-provider (Hetzner/OVH) footprint (HAProxy, BIND, Consul, Tinc).
🟢 Storage & Backup Layer
Storage : ZFS-based high-performance storage (10K RPM HDDs + SSD L2ARC/SLOG) via iSCSI/Synology.Backup : Tiered Rsync flow (hans $\to$ pfserver-04 $\to$ Backblaze).
🟢 Networking Layer
Cluster Interconnect : Tinc (decentralized mesh for Hetzner cluster linking)DNS : BIND (Public) + Consul (Dynamic Public)Mesh VPN : Netbird (zero-trust connectivity)Secure Access : Boundary + Netbird + Jump-hostsIngress/Load Balancing : HAProxy for cluster ingressSecurity : Distributed pfsense firewalling.