Blog - MattBits

Pre-amble This blog post was written in-flight during a quest to create a secure deployment mechanism for Terraform projects to Vault, Consul and Nomad. The beginning portion was written whilst attempting to use a technique that ended up failing. Feel free to skip this portion and jump to “Using JWT authentication” Intro For the Hashicorp stack of my homelab, I have: Vault cluster Consul cluster (single DC) Nomad servers Nomad clients using multiple datacenters An offline root CA, intermediate CA and terraform state are stored/managed by Minio (local S3-compatible alternative) and an seperated isolated Vault cluster.

Background I maintain a handful of open sources projects - most of which are of no interest to anyone. There are one or two, however, that have a small handful of users - and also a small number of contributors. Because of this, I spent a very inconsistent amount of time on each of these, occasionally fixing bugs and occasionally spending an hour or two each for a week to get a feature done.

AWS offers a broad variety of services - some of which are essential and unavoidable when using Amazon as a cloud provider. Others, on the other hand, provide solutions that, whilst on the surface are great as a quick start, do not scale. By scale, I don’t mean in the usual compute-power way, I mean in cost. If you use a service that provides scalability with a steady and predictable workload, then you are paying for flexibility that you aren’t using - and you do pay for this.

Goals I started off with a basic goal - host a small python website, which uses a MySQL-like database on a highly available cluster. After recently moving from a rack in a datacenter (and fortunately saving ~£250/month), I started looking at hosting some web applications on VPSs from a couple of providers. Of course, this means: Every instance costs money - I can no longer spin up 10 extra VMs because the hardware is already running… each instance costs.

Tinc appears to be one of the few open source mesh VPNs and, in my expierience, once working, performs incredible well. That said, the configuration of tinc is a little clunky and repetitive, nor does their documentation give much of a clue as to what is required for a minimal setup. Example The following example will connect 4 machines in a tinc mesh network; one machine has a direct internet connection (and have public IP addresses); two are behind a NAT gateway, with tinc ports being fowarded to one of the nodes; one machine is behind a different NAT gateway, again, unable to forward ports.

🧾 1. INFRASTRUCTURE EVOLUTION TIMELINE (HIGH LEVEL) Era Approx timeframe Key characteristics Core systems Early server era (pre-~2007 → unknown) First exposure to servers, NT domain Compaq Windows NT server Early homelab / ESX era ~2007–2013 Physical servers, early virtualization (RAM constrained) DL380 G3 (x3), DL360 G3, ESX/vCenter, Whitebox NAS Expansion / distributed experimentation ~2012–2016 Multi-server lab, PXE booting experiments, Jenkins (sheila) x3455 servers, DL160 G6 (later), fileserver VPS High-Availability Era ~2018 Multi-provider VPS cluster, Mesh VPN, HA MariaDB Tinc, OpenVPN, Docker Swarm, HAProxy Identity foundation era ~2016 First stable Linux identity system FreeIPA (marge/homer) Datacentre / colo era ~2017–2018 Formal cluster + SAN deployment DL160 G6 cluster, Nexsan SAN Consolidation era ~2018–2019 Transition (ThinkCentre) $\to$ R620/R720xd consolidation ThinkCentre, R620, R720xd, Synology, Keycloak Platform orchestration era ~2019–2024 Container orchestration, Rancher, early automation GitLab (Rancher), Jenkins systems Modern orchestration era ~2024–2026 Nomad-based unified service plane Nomad, Consul, Vault, Keycloak migration 🧾 1.