Repo layout¶

One private repo (here called homelab) holds everything that defines the servers. The layout is boring on purpose: the path tells you which host a file belongs to and which Komodo Stack deploys it.

homelab/
├── README.md                 # hosts, roles, deploy flow, where things are written down
├── renovate.json             # update policy (see Renovate page)
├── .gitignore                # .env, keys, data dirs
└── hosts/
    ├── control-host/
    │   ├── komodo/           # compose.yaml + README: applied BY HAND, not a Komodo stack
    │   ├── forgejo/          # compose.yaml, ca-bundle.crt
    │   ├── renovate/
    │   ├── ntfy/
    │   └── <stack>/compose.yaml
    ├── server1/
    │   ├── mediastack/compose.yaml
    │   └── immich/{compose.yaml, hwaccel.transcoding.yml}
    ├── server2/
    │   └── <stack>/compose.yaml
    ├── workstation/          # reference only: run by hand, ignored by Renovate
    └── _fleet/               # host-level config, not containers
        ├── install-host.sh
        ├── bin/notify        # tiny ntfy publisher used by every script below
        ├── updates/          # unattended-upgrades, reboot-if-required timer, apt report
        ├── smart/            # smartd.conf + alert hook
        ├── diskspace/        # hourly fill check + per-host thresholds
        ├── system/           # sysctl, boot-time fixes
        ├── docker/           # daemon.json
        └── backup/           # borgmatic configs, own install.sh, restore runbook

Stacks: hosts/<host>/<stack>/¶

• One directory = one Komodo Stack. The directory name is the Stack name (unique across hosts, e.g. monitoring-server1 if two hosts have a monitoring stack), and the Stack's run directory is that path.
• The compose file is always compose.yaml (Compose v2 default, no version: key).
• Anything the stack reads (config files, a Dockerfile for a local build, scripts) sits next to it and is listed in the Stack's config files so changes to it trigger a redeploy.
• Persistent data is never in the repo directory. Bind mounts are absolute: /mnt/appdata/<service>/.... Komodo runs compose from its own clone, so a relative ./data ends up inside that clone (see Gotchas).
• Images are pinned to exact tags. Databases can additionally be pinned by digest (postgres:17.6@sha256:...).
• Every service has a healthcheck. When the image has no curl/wget, a comment says why the chosen command works.
• Comments explain why, with a link to the upstream doc. The "what" is already in the YAML.

Adding a new service:

1. Create hosts/<host>/<stack>/compose.yaml. Run docker compose config -q on it locally (catches YAML and interpolation mistakes before Komodo does).
2. Create the secret Variables it needs in Komodo, then the Stack (copy an existing one's settings, change server and run directory, add the Stack Environment).
3. Push. The webhook deploys it. Check the Stack shows Running on the latest commit.

Removing one: Destroy the Stack in Komodo first (that runs compose down), then delete the directory and push. The other order leaves containers running with nothing tracking them.

Host-level config: hosts/_fleet/¶

Some things aren't containers: unattended-upgrades, smartd, a disk-space timer, sysctl tweaks, the script that sends host alerts to ntfy. They still belong in git, so they're reviewable and a rebuilt host gets the same setup.

There's no config-management tool here (yet). Instead, one idempotent shell script copies the files over SSH with install -D -m <mode> -o root -g root and enables the units:

# From a checkout on the workstation
hosts/_fleet/install-host.sh <ssh-target|local> <host> [component...]

hosts/_fleet/install-host.sh server1 server1                 # all default components
hosts/_fleet/install-host.sh server2 server2 updates smart   # just these

How the script is built:

• put <src> <dest> <mode> pipes each file over SSH into sudo install, so no temp files are left on the target and ownership/mode are always reset.
• Every step is safe to re-run: re-running after a git pull is how changes are deployed.
• No secrets in the repo: the ntfy token is read by the caller (from Komodo or a password manager) and piped on stdin into a root-only file on the target.
• Per-host differences are files named after the host (diskspace/<host>.conf, timer.d/<host>.conf), picked up when present.

The tiny publisher everything uses:

#!/bin/sh
# notify <topic> <priority 1-5> <tags> <title> [message|-]
topic="$1"; prio="${2:-3}"; tags="$3"; title="$4"; msg="$5"
[ -z "$msg" ] || [ "$msg" = "-" ] && msg="$(cat)"
tok="$(cat /etc/ntfy/host-publisher.token 2>/dev/null)"
[ -n "$tok" ] || { logger -t notify "no token, dropped: $title"; exit 0; }
# LAN address first so host alerts don't depend on DNS or the tunnel, then the public name
for base in http://<YOUR_SERVER_IP>:88 https://ntfy.<YOUR_DOMAIN>; do
  if printf '%s' "$msg" | curl -fsS --max-time 15 -o /dev/null \
      -H "Authorization: Bearer $tok" -H "Title: [$(hostname -s)] $title" \
      -H "Priority: $prio" -H "Tags: $tags" --data-binary @- "$base/$topic"; then
    exit 0
  fi
done
logger -t notify "delivery failed: $title"
exit 0   # never fail the caller (a backup hook, a timer) because ntfy was down

Backups have their own installer (hosts/_fleet/backup/install.sh) because they also create repos and keys.

.gitignore¶

# Secrets
**/.env
**/.env.*
**/secrets/
*.pem
*.key
*.ovpn
*.wg

# Docker data that should never be in the repo anyway
**/data/
**/logs/
**/cache/

# Local CA: never commit keys
**/rootCA*.pem
**/*-key.pem

A public CA bundle (system CAs + your local root certificate, no key) is fine to commit when a stack needs it at deploy time, like Forgejo's ca-bundle.crt. If you ignore *.crt globally, add a !path/to/ca-bundle.crt exception.

Sources & further reading¶

• Komodo: Docker Compose stacks
• Docker: Compose file reference
• Debian wiki: UnattendedUpgrades
• ntfy: Publishing