Homelab: reproducible infrastructure in one playbook

The goal of this project fits in one sentence: be able to wipe everything and rebuild it without thinking. No machine configured by hand, no setting that only exists in my memory.

The architecture

                 ┌─────────────────────┐
   Internet ───▶ │  Reverse proxy      │  (Caddy, automatic TLS)
                 └──────────┬──────────┘
                            │
        ┌───────────────────┼───────────────────┐
        │                   │                   │
   ┌────▼─────┐      ┌──────▼──────┐     ┌──────▼──────┐
   │ DNS      │      │ Monitoring  │     │ Services    │
   │ (LXC)    │      │ (LXC)       │     │ (VM / LXC)  │
   └──────────┘      └─────────────┘     └─────────────┘
          Proxmox VE: single host, backups on a dedicated disk

How it’s split

Layer	Tool	Responsibility
Provisioning	Terraform	Creates VMs and containers on Proxmox
Configuration	Ansible	Packages, users, services, firewall
Secrets	SOPS	Encrypts sensitive variables in the repo
Monitoring	Prometheus + Grafana	Metrics and alerting

The boundary is strict: Terraform creates machines and never configures them, Ansible configures machines and never creates them. The moment that line is crossed, you stop knowing which of the two is authoritative.

What was hard

Boot ordering. The reverse proxy needs DNS, monitoring needs the reverse proxy, and everything needs the hypervisor. Solved by declaring dependencies explicitly instead of hoping alphabetical order would do the job.

Secrets. First version: an uncommitted vars.yml that I copied from machine to machine. About as practical as a sticky note. Moved to SOPS with an age key: secrets live in the repository, encrypted, and the playbook decrypts them at runtime.

Being honest about idempotency. A playbook that “works” isn’t necessarily idempotent. Running it twice in a row and checking that the second run changes nothing (changed=0) is the only test that really counts.

Where it stands

Full rebuild measured at roughly 35 minutes, 10 of which are image downloads. Next step is triggering it from CI, so infrastructure changes go through a merge request like the rest of the code.