Another blog from another guy called Matt
Occasional blog posts from a random systems engineer

Kubernetes Learning Pledge

· Read in about 6 min · (1092 Words)

This is a pledge to myself and for any one else interested in following the same journey.

I don’t know jack about Kubernetes. I know about docker, integrated with it’s API, cgroups etc. I’ve used nomad and docker swarm heavily and rancher.

History

I have taken steps in the past to master technologies in various levels, for example Terraform, nomad

Terraform

  1. Used Terraform for about a decade, managed and supporting hundreds of projects using it
  2. Working knowledge to create agnostic re-usable modules
  3. Created and fixed bugs in Terraform providers
  4. Created a Terraform module and provider registry
  5. Created a working PoC of an open source Terraform cloud alternative:
    • implementing the Terraform cloud APIs,
    • reverse engineered the official Terraform cloud agent APIs to a fully functional state
    • implemented missing features (such as environments)
  6. working implementation of a custom state Vault state backend into OpenTofu (https://github.com/opentofu/opentofu/pull/1680)
  7. working as a maintainer for tfswitch

Hashicorp Nomad/Consul/Vault

I’ve written a lot about this before, but I have tried to use this stack in various ways to create flexible and secure patterns:

Others

Even non-career related instances, I have written a Gameboy emulator and a Gameboy game

For me, the fastest way to understand a system has always been to build it, integrate with it, break it, fix it and occasionally re-implement parts of it.

What I aim to achieve

My Mental Model

Coming from AWS and Terraform, Kubernetes fits into a fairly direct translation.

Existing mental model Kubernetes equivalent
AWS account (or homelab VM) Kubernetes cluster
Terraform state Desired state
IAM policies RBAC
EC2 / Auto Scaling Groups Nodes / Node pools
Load balancers Services / Ingress
Terraform modules Helm charts / Operators
Terraform apply Reconciliation loop

The key difference is simple:

Terraform applies a desired state when executed.

Kubernetes continuously reconciles that state in the background.

That single design decision drives almost every architectural property of the system.

Everything else is a consequence of that loop existing at cluster scale.

What I Am Not Trying To Do

This is not about certification or surface-level familiarity. I am not trying to:

  • Memorise kubectl commands
  • Pass CKA or CKAD exams
  • Learn YAML by repetition
  • Follow “deploy your first app” tutorials in isolation

Some of those things will happen naturally, but they are not the objective.

The objective is to understand how the system is built, why it behaves the way it does, and where the real design trade-offs are.

Planned Learning Path

Phase 1 – Core Kubernetes

Before anything else, the fundamentals need to be solid.

That means understanding:

  • API server and etcd
  • Controller manager and reconciliation model
  • Scheduler behaviour and decision logic
  • Pods, deployments, statefulsets, jobs, cronjobs
  • Services, ingress, and cluster networking
  • ConfigMaps and secrets
  • Persistent volumes, claims, and storage classes
  • Scheduling constraints (requests, limits, affinity, taints, tolerations)

This is not just about knowing what these objects do but about understanding why the system needs them at all.

This phase will involve deliberately breaking clusters and rebuilding them from the ground up.

Phase 2 – Homelab Integration

This system is not being built in a cloud environment.

It runs on a fully self-owned infrastructure stack:

  • libvirt for compute provisioning
  • ZFS for storage management
  • iSCSI-backed volumes where required
  • pfSense for routing, firewalling, and segmentation
  • phpIPAM for IP address management (temporary, but functional)
  • a custom metadata API for node identity, bootstrap state, and configuration
  • Vault for secrets, authentication, and identity flows
  • Terraform for VM provisioning and orchestration

And Kubernetes must operate within this model, meaning I will consider:

  • Provisioning:
    • custom node provisioning flows (Kubelet bootstrap tied into metadata API)
    • cluster autoscaling models that are not cloud-provider based
    • potential Cluster API-style control of libvirt-backed infrastructure
    • custom node labels/taints derived from physical topology or metadata API state
  • Storage:
    • custom CSI driver backed by ZFS datasets
    • snapshot-aware volume provisioning (ZFS snapshot → Kubernetes VolumeSnapshot)
    • iSCSI-based persistent volume lifecycle management
    • backup/restore controllers that operate at dataset level rather than pod level
    • topology-aware volume scheduling (matching workloads to storage pools)
    • CNI plugins that reflect physical network topology rather than cloud networking assumptions
  • Network:
    • custom IPAM integration for pod/service IP allocation tracking
    • reconciliation between Kubernetes Services and pfSense firewall rules (policy-as-code → firewall rules)
    • ingress integration that maps cleanly onto existing Traefik / reverse proxy layers
    • network policy enforcement mirrored into external firewall configuration
  • Identity:
    • Vault-integrated service account authentication (dynamic secrets per workload)
    • external secret operators syncing Vault → Kubernetes secrets (or avoiding Kubernetes secrets entirely)
    • SPIFFE-like workload identity models mapped onto Vault policies
    • admission controllers enforcing identity constraints at deployment time
    • metadata-driven bootstrap for pods (injecting runtime identity/config at scheduling time)

Phase 3 – Helm

Helm is the packaging layer for Kubernetes workloads. Rather than an astrction over kubernetes, it’s a system for templaing and releases.

Key areas:

  • chart structure and templating
  • values-driven configuration
  • release lifecycle (install, upgrade, rollback)
  • limits of templating as a control mechanism

Phase 4 – Controllers and Operators

This is the first real extension mechanism in Kubernetes.

A controller is:

A reconciliation loop that enforces desired state.

An operator is:

A controller combined with domain-specific knowledge encoded through CRDs.

Work here includes:

  • Custom Resource Definitions (CRDs)
  • Controller-runtime / kubebuilder
  • Reconciliation loops and idempotency
  • Informers and event-driven state tracking
  • Work queues, retries, and backoff models
  • Finalizers and lifecycle control
  • Admission webhooks (mutating and validating)
  • Status vs spec separation

My focus will be building controllers from scratch, rather than consuming prebuilt ones.

Phase 5 – Infrastructure as Kubernetes

At this stage Kubernetes starts overlapping directly with infrastructure provisioning systems.

Examples:

  • Crossplane
  • AWS Controllers for Kubernetes (ACK)
  • External Secrets Operator
  • Cert-Manager

This pushes Kubernetes beyond container orchestration and into infrastructure management for large-scale, multi-system deployments.

During reading it’s posed an important (and common) architectural question:

When does Kubernetes become a better control plane than Terraform, and when does it not? And finding this will be the goal.

Phase 6 – Platform Engineering

Once the primitives are understood, the focus shifts to system design.

This includes:

  • GitOps workflows (ArgoCD / Flux)
  • Multi-cluster architectures
  • Identity and tenancy models
  • Policy enforcement (OPA / Kyverno)
  • Observability systems (metrics, logs, tracing)
  • Internal platform APIs
  • Deployment and promotion workflows

What Success Looks Like

This will be considered successful when I can:

  • Read Kubernetes source code without relying on documentation for context
  • Build controllers and operators without following tutorials
  • Understand reconciliation, scheduling, and networking behaviour
  • Debug cluster behaviour without guessing
  • Clearly define when to use Terraform, Helm, or operators
  • Contribute meaningful fixes upstream