Reading earlier comments in this community made me consider documenting the workings of my homelab to some extent, ie. docker configuration, credentials, ports and links of my services. I’ve tried it to make it consistent and organised but it still feels half baked and insufficient. Everyone suggests documenting everything you do in your homelab but don’t state how. Since I’ve hardly had experience running my own server, I would really appreciate observing the blueprint of some other fellow selfhoster for copying or taking inspiration from rather than considering the note taking to be ‘left as an exercise for the reader’.
You must log in or # to comment.
Ansible and Nix. Code is the document.
I’ve been in the process of migrating everything over do Nix. Love it so much.
What hole does Ansible fill for you? I haven’t looked into it in the past really, so just curious. I have a single Paoxmox node so don’t really need horizontal scaling orchestration.
Libreoffice calc/MS Excel. Old school tracking and extremly flexible for documentation. I have been doing this for the last decade, both at home and at my workplace. My team loves it, tho YMMV.
This is what I like about git ops and infra/config as Code personally.
Ideally everything is an a tofu/ansible/helm chart and git lab pipeline/Fleet job. I add comments for anything that I had to learn to make work to those files. Follow good commit hygenine (most of the time). And bam I can almost a year later half asleep stumble back into a thing I did.
Do you use this for physical machines too?
Yep! Metal3 for servers with BMCs Tinkerbell for everything else.
I also have an ansible playbook that templates everything into a cloud init scripts as a boot strap server.
About 12 nodes in total now, from new servers to freebee junk laptops in it.
Netbox
Use Ansible or some such solution like Puppet, Salt or Chef, just like the big boys do. If you don’t have a unified editable config for your machines, you don’t really have a homelab, you just have a pile of hardware instead.
I created something similar to this. It got a lot of love during interviews later down the line. https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fi.redd.it%2Fvmd34mabi4r91.jpg&f=1&ipt=2dde77fd04d48156bc514ad4b1f090c8473f4e666ead0e16906eeed55a79aca6
That is a behemoth of a homelab you have set up there. My jaw would’ve dropped out if it could.
Dude that is a respectable lab you have there! Much envy
I believe it is traditional to do so written in blood in the style of an apocalypse log, dealer’s choice for who’s blood. Make sure it’s disjointed and nearly incomprehensible, but that everything is there.
Bonus points if you print the config files and write your documentation on them after stapling them to the walls
Whenever I set something up I usually make a markdown file listing the commands and steps to take. I do this as I am setting things up and familiarizing myself, so once I’m done, I have a start to finish guide.
Raw text/markdown files will be readable until the end of time.
I agree with the advice that says “Document your setup such that you could recreate it from your notes from scratch” but I’d take it another step further — consider that someone may have to do some work on your system when you are unable or unavailable. The kind of thing you’d keep with your will, or power of attorney. Just a suggestion.
…and to my family I bequeath my entire collection of Linux iso’s
You jest but if I left my wife my Home Assistant setup undocumented she would pee on my grave.
LOL, well I’m single tho I’ve known my ladyfriend for over 40 years. I offered to set up a server at her house, and connect the two, but she has no interest rifling through all my lab for anything of interest in the case of my passing.
I’m happily married with a kid, and we recently went through the estate planning process. When I brought up IP stuff and digital properties, their advice was pretty much “Hmm… you should pick someone who understands what you’re talking about, get their approval in advance, and then add them as your legacy contacts and document the heck out of everything”. Realistically nobody is going to want my GitHub stuff or anything like that, but I would like my kid to have access to most* of my files after I pass. I am of course excluding the kind of content that “real friends” delete while your body is still warm.
It’d be nice to donate all my equipment to some kid who is very interested. That would be something I’d be interested in.
Ansible is my config and documentation in one.
It’s reproducible, idempotent and I don’t need anything else.
I write all code myself, that makes it even easier to read.
I “document” everything by forcing myself to create ansible runbooks for new services and configs. I have some gaps, definitely, but the more of them I create, the easier new services are to deploy.
(Bookmarked for when I have the mental capacity to …)
Do y’all also document backup/restore procedures?
How often do you test it?Frankly, with my screwed up brain, I document everything. I can turn around twice in my lab and my brain will flat line. When I first started, I would always tell myself that I’d remember stuff. Not anymore.
I created a script for Linux that automatically backs up to a NAS drive, once every two weeks, as a complete image, and I keep 5 on deck. Testing usually happens once every 3 months or so. I also have Duplicati backups that are stored offsite on my VPS.
That’s the neat part, I don’t!
I have a docker-compose file, which is somewhat self-documenting, especially since I give everything descriptive names. Creds go in bitwarden anyway.
But then, my environment isn’t that complex, and I don’t have anything so custom that I need notes to replicate it.
I have two systems that sort of work together.
The first system involves a bunch of text files for each task. OS installation, basic post OS installation tasks and a file for each program I add (like UFW, apparmor, ddclient, docker and so on). They basically look like scripts with comments. If I want to I can just copy/paste everything into a terminal and reach a a specific state that I want to be at.
The second system is a sort of “skeleton” file tree that only contains all the files that I have added or modified.
Here's an example of what my server skeleton file tree looks like
. ├── etc │ ├── crontabs │ │ └── root │ ├── ddclient │ │ └── ddclient.conf │ ├── doas.d │ │ └── doas.conf │ ├── fail2ban │ │ ├── filter.d │ │ │ └── alpine-sshd-key.conf │ │ └── jail.d │ │ └── alpine-ssh.conf │ ├── modprobe.d │ │ ├── backlist-extra.conf │ │ └── disable-filesystems.conf │ ├── network │ │ └── interfaces │ ├── periodic │ │ └── 1min │ │ └── dynamic-motd │ ├── profile.d │ │ └── profile.sh │ ├── ssh │ │ └── sshd_config │ ├── wpa_supplicant │ │ └── wpa_supplicant.conf │ ├── fstab │ ├── nanorc │ ├── profile │ └── sysctl.conf ├── home │ └── pi-user │ ├── .config │ │ └── ash │ │ ├── ashrc │ │ └── profile │ ├── .ssh │ │ └── authorized_keys │ ├── .sync │ │ ├── file-system-backup │ │ │ ├── .sync-server-fs_01_root │ │ │ └── .sync-server-fs_02_boot │ │ └── .sync-caddy_certs_backup │ ├── .nanorc │ └── .tmux.conf ├── root │ ├── .config │ │ └── mc │ │ └── ini │ ├── .local │ │ └── share │ │ └── mc │ │ └── history -> /dev/null │ ├── .ssh │ │ └── authorized_keys │ ├── scripts │ │ ├── automated-backup │ │ └── maintenance │ ├── .ash_history -> /dev/null │ └── .nanorc ├── srv │ ├── caddy │ │ ├── Caddyfile │ │ ├── Dockerfile │ │ └── docker-compose.yml │ └── kiwix │ └── docker-compose.yml └── usr └── sbin ├── containers-down ├── containers-up ├── emountman ├── fs-backup-quick └── rtransferThis is useful to me because I can keep track of every change I make. I even have it set up so I can use
rsyncto quickly chuck all the files into place after a fresh install or after adding/modifying files.I also created and maintain a “quick install” guide so I can install a fresh OS,
rsyncall the modified files from my skeleton file tree into place, then run through all the commands in my quick install guide to get myself back to the same state in a minimal amount of time.Document everything as if it were a step by step tutorial you will give to someone so that they can duplicate your deployment without any prior knowledge. I’ll even include urls to sites I consulted with to achieve production deployment.
