Homelab consisting of a 4-node Kubernetes cluster, Proxmox host, OPNsense router, + more.
The cluster runs on recycled thin clients and mini PCs mostly from ebay and facebook marketplace:
| Device | Specs | Role |
|---|---|---|
| HP Chromebook 14 G5 | Intel Celeron N3350, 4GB RAM, 24GB eMMC | NetBird gateway + random services. Used to be the entirety of my old homelab. |
| HP Mini | Intel i5-6500T, 16GB RAM, 500GB SSD + 1TB NVMe | Proxmox host (Control Plane VM, OPNsense VM, NetBird LXC, AdGuard LXC). |
| CLI ar9070 (×2) | AMD Ryzen V1605B, 16GB RAM, 60GB SSD | Talos worker nodes. |
| Dell WYSE 5070 | Intel J4105, 8GB RAM, 64GB SSD + 2TB HDD | Talos worker node. |
Github
Plans
Services I want to add:
- Harbor to host container images (replace dockerhub)
- git server - probably forgejo
- Restic, syncthing, and atuin (for nixos laptop)
Hardware to get:
- Custom NAS (to support SAS drives) + at least 1 more HDD
- GPU focused box for ML + AI stuff.
- Dedicated router box, instead of a VM in proxmox.
- Always need more RAM.
Other / Miscellaneous:
- Replace LXCs with NixOS LXCs. Should replace Ansible for LXCs. Use NixOS as much as i can. Plus impermanance. See Erase your darlings.
- Terranix
- Nixidy
Runtime / orchestration decisions (2026-07)
Investigated whether to de-containerize the cluster (run Nix closures as workloads instead of OCI images). Killed a project idea (“Klosure”, a NixOS-guest-under-kubelet runtime) after finding the whole design space is already covered by existing tools. Notes below so the reasoning isn’t re-litigated.
The layers, and what already solves each
“Containerization” is four separable things: (1) OCI image as artifact, (2) constructed rootfs, (3) isolation = namespaces + cgroups + seccomp/caps, (4) runtime machinery (containerd/CRI/runc). On NixOS the part worth removing is 1+2 (the closure already is the environment); the isolation (3) is not waste, it’s just kernel primitives.
nix-snapshotter(containerd CRI image service): makes workloads Nix store paths, no registry, store dedup across pods. Removes 1, does 2 from the store. v0.4.0, young/niche.nix-snapshotter+runcunder Kubernetes: isolated closures scheduled across nodes, no images. This is the closure-native cluster, and it’s the baseline that makes a custom runtime pointless.- Kata +
nix-snapshotter: adds a microVM (real kernel boundary) via RuntimeClass; virtiofs shares the store into the guest. Blocked only by a small in-flight Kata shim fix (PR kata-containers#12621 / nix-snapshotter#178). - Verified both by building and running them here (k3s+nix-snapshotter+runc, and Kata+nix-snapshotter with the patch): pods run, no registry, 97 store paths visible inside the microVM via virtiofs, redis serves. So this is real, not theoretical.
Node OS + isolation ladder
For “run an isolated workload from a closure”, lightest → heaviest, all share the host kernel except the VM rows:
- hardened
systemdunit (namespaces/cgroups/seccomp/caps on a process, no image, no rootfs) — most NixOS-native, single service. - NixOS
containers.*(systemd-nspawn) — full NixOS system, namespace-isolated, shared kernel/store, no image. - microVM NixOS guest (
microvm.nix) — full NixOS system + hardware kernel boundary; heavier (2nd kernel + full userland → more RAM/boot than a container-in-VM; the overhead is the systemd userland that provides the benefit). - Kata + nix-snapshotter — container-in-VM (what survives for k8s).
Isolation ceiling is the same for all shared-kernel options (a maxed hardened unit ≈ nspawn ≈ runc); only the VM rows give a real boundary against kernel exploits.
Orchestrator: k8s vs Nomad
Kubernetes is architecturally container-shaped (CRI/pods assume containers). Nomad is not: its exec2 driver schedules isolated processes natively — point it at a store path and you get “hardened systemd unit, scheduled across nodes” with no container model. Better fit for de-containerized Nix workloads; costs: DIY closure delivery (no polished equivalent of nix-snapshotter’s image service), smaller ecosystem, BSL license. k8s wins on ecosystem, turnkey closure delivery (nix-snapshotter), control-loop observability, RBAC, GitOps maturity.
Talos (current) vs NixOS + nix-snapshotter
- For NixOS switch: one artifact universe (OS + workloads are both closures, same flake/cache), no registry for own workloads, whole-stack reproducibility + generation rollback, store dedup, hackability.
- Against (keep Talos): Talos is turnkey/hardened/immutable/API-only (no shell, tiny attack surface, near-zero maintenance); NixOS is DIY and nix-snapshotter is a frontier component (betting the cluster on a v0.4.0 tool). Talos is more bounded for an autonomous agent to manage.
- Middle ground: build workload images with Nix (
nix2container/dockerTools) and keep Talos → Nix-reproducible workloads without leaving Talos. Loses only no-registry + store-dedup + node-is-Nix.
Decision lens: AI-agent-managed homelab (the actual goal)
Goal is an agent-managed homelab; reasoned that declarative > imperative suits agents. Correct instinct, but declarative ≠ Nix — Talos and k8s are also declarative, and for agents they have what Nix lacks: a reconciliation control loop with observable status (agent sees desired-vs-actual, self-heals), RBAC/admission to bound blast radius, and GitOps (agent proposes declarative diffs → CI validates → reconciler applies) as a mature agent-safety pattern. Talos is also more agent-safe at the node layer (immutable, no shell, API-scoped).
Nix’s genuine agent wins are at the artifact layer: whole-system pre-apply testing in a sandbox (proven this session — VM-tested infra before trusting it), one language for OS+workloads, atomic rollback, mcp-nixos grounding. Nix’s agent costs: cryptic errors, hallucination-prone option paths (hit several this session), no runtime control loop, weak blast-radius controls.
Conclusion / direction: the declarative-for-agents argument upgrades artifacts to Nix, not necessarily nodes away from Talos. Strongest agent-managed stack = keep Talos + k8s + GitOps + RBAC (the control loop + guardrails) and add Nix at the artifact layer (nix2container images, or nix-snapshotter+runc if moving nodes to NixOS for no-registry). Full NixOS-node switch only if the “agent owns one declarative repo that is the whole homelab, build+test before apply” model is the priority — elegant, but trades away Talos’s agent guardrails and k8s’s control-loop observability. See Nixidy/GitOps + Erase your darlings for the hybrid path.