Currently, I run Unraid and have all of my services’ setup there as docker containers. While this is nice and easy to setup initially, it has some major downsides:
- It’s fragile. Unraid is prone to bugs/crashes with docker that take down my containers. It’s also not resilient so when things break I have to log in and fiddle.
- It’s mutable. I can’t use any infrastructure-as-code tools like terraform, and configuration sort of just exist in the UI. I can’t really roll back or recover easily.
- It’s single-node. Everything is tied to my one big server that runs the NAS, but I’d rather have the NAS as a separate fairly low-power appliance and then have a separate machine to handle things like VMs and containers.
So I’m looking ahead and thinking about what the next iteration of my homelab will look like. While I like unraid for the storage stuff, I’m a little tired of wrangling it into a container orchestrator and hypervisor, and I think this year I’ll split that job out to a dedicated machine. I’m comfortable with, and in fact prefer, IaC over fancy UIs and so would love to be able to use terraform or Pulumi or something like that. I would prefer something multi-node, as I want to be able to tie multiple machines together. And I want something that is fault-tolerant, as I host services for friends and family that currently require a lot of manual intervention to fix when they go down.
So the question is: how do you all do this? Kubernetes, docker-compose, Hashicorp Nomad? Do you run k3s, Harvester, or what? I’d love to get an idea of what people are doing and why, so I can get some ideas as to what I might do.
Thanks for this, I’ve been sort if interested in both Nomad and NixOS for the exact reasons it seems like you use them. Thanks for linking that repo, I’ll check it out for inspiration!
Do you find that you sometimes struggle to get things working in Nomad? My one worry is that, because it’s not as well established as kubernetes or docker, there won’t be good compatibility or documentation. For example most services in their docs will show how to deploy with kubernetes or docker, but rarely Nomad. Do you find that it’s easy enough to translate these instructions that it doesn’t matter?
Good question! So it depends, but TLDR: imo it’s worth it, or it’s fine, but it’s easy to try yourself and see
You are absolutely correct, but I do find that for the large large majority of things, either you can find an online Nomad config, or the Nomad config is easy enough to translate from Docker compose. Only some complicated larger deployments (think Immich) are harder to translate, but even then it just takes some trial and error. I really do think that extra trouble of translating is very much worth the pain you save yourself in terms of deploying k8s though. You might spend a bit longer typing out the Nomad job file yourself, but in exchange you are thankfully not maintaining the k8s cluster.
As far Nomad-specific documentation goes, I think it the official one is more than good enough.
You mentioned compatibility. So far I have not found anything I really wanted that was not possible to set up in Nomad. Nomad does CNI and CSI, which is the same API k8s uses, so thinkgs working there will work for Nomad. Other things you would use with docker compose or k8s don’t work with Nomad, but you don’t need them (for example: portainer or metrics exporters) because Nomad has them natively already (this blog discusses that).
As you can see I am pretty opinionated towards Nomad - I have been using it in my previous job in prod, and in my home-lab for a year now, and I am very happy with it. If you would like to read more I recommend this blog post. For Nomad on NixOS I wrote this one.
For now my advice is: just try nomad yourself (as simple as running
nomad agent -devon your laptop), run the tutorial, and see if it was easy enough that you see yourself using it for the rest of your containers. If you need more help you are welcome to DM me :)