Skip to content

Backups: lessons learned

Most homelab backup setups don't fail loudly. They fail by never quite running, by copying a database mid-write, or by getting deleted along with the thing they were protecting. These are the lessons from getting it wrong first. How the backup system is built is in Backups.

Several "backup systems", none of them working

Symptom. You audit the hosts and find the remains of three tools (a Kopia install, an unconfigured borgmatic, a Restic UI behind a proxy host pointing at a dead server) and old repositories nobody has written to in months. Nothing is actually backing up.

Cause. Each attempt was set up, not monitored, and quietly broke: a cron line with a syntax error, a target host that was retired, a password that changed. Without a success signal, "set up" and "working" look the same.

Fix.

  • Pick one tool and delete the others' leftovers (repos, cron lines, proxy hosts), so nobody mistakes them for a current backup.
  • Give every job a dead-man switch from day one: a push monitor that goes DOWN on failure or after ~25 h without a report (see Monitoring).
  • Don't call it done until a restore worked (below).

A restore test is the only proof

Symptom. Backups have been "green" for months. The first real restore fails: a database file won't open, a dump is empty, the passphrase isn't where you thought.

Cause. A file-level copy of a live database (SQLite, Postgres, MongoDB) can be torn mid-write. Restic and Borg aren't database-aware; they copy whatever bytes are on disk at that moment.

Fix.

  • Dump databases before the snapshot and back up the dump: pg_dump / pg_dumpall for Postgres, SQLite's own .backup (or borgmatic's sqlite_databases hook), mongodump or the app's own export for Mongo, the app's built-in backup for things like Home Assistant and network controllers. borgmatic has hooks for most of these.
  • Do a restore drill and write it down: restore the dump into a throwaway container (postgres:<same major>), run an integrity check (PRAGMA integrity_check), boot the real app against the restored data once, and pull at least one file from the offsite copy. Repeat it every few months.
  • Store the repository passphrases and keys in your password manager and somewhere that doesn't depend on the backed-up hosts. A vault that lives on the server you're restoring can't help you restore it.

A restore drill also tells you things about the live system: one found that a password manager everyone assumed was in use had zero accounts.

Backups that a compromised host can delete

Symptom. None yet, which is the point. Each server pushes backups to the others over SSH, and each server's key can do anything on the target.

Cause. If a host is compromised (or a script goes wrong), the same credentials it uses to write backups can delete them.

Fix. Make the receiving side append-only and lock each key to one repository with a forced command in the target's authorized_keys:

~borg/.ssh/authorized_keys (on the backup target)
command="borg serve --append-only --restrict-to-repository /srv/borg/server1 --storage-quota 500G",restrict ssh-ed25519 AAAA... borg@server1
  • --append-only means a client can add archives but not really delete them. A prune from the client only marks segments; the space is freed when you run borg compact on the target (a weekly timer there).
  • --restrict-to-repository stops one host's key reading or writing another host's repo. Test it: the wrong key should get Repository path not allowed.
  • Any other key that lands in the same account (for a different purpose) gets the tightest restriction it can work with: start from restrict and add back only what it needs. One unrestricted shell key in the same account as your offsite repos undoes all of the above.
  • A backup UI that only reads repos gets its own append-only key, too.

Offsite backups slow down everything else (or never finish)

Symptom. While the offsite job runs, streaming stutters and the internet feels slow. Or the very first offsite run of a large dataset (a photo library) takes days and gets killed by the next night's schedule.

Cause. An uncapped upload saturates the uplink. A first full seed of hundreds of GB can't finish in one night, and an unplanned reboot or a second job interrupts it.

Fix.

  • Cap the upload in the backup tool (borgmatic upload_rate_limit, Restic --limit-upload) to a share of your uplink you can live with.
  • Give large, slow-changing datasets their own configuration and schedule, and start that seed first, by hand, days before you rely on it. Make the nightly job skip or wait while the seed runs.
  • Keep LAN and offsite as separate configs with separate monitors, so a provider outage doesn't hide a working LAN backup or the other way round.
  • Have automatic reboots (unattended upgrades) wait while a backup is running or about to start.

Things that aren't in the backup even though "/mnt/appdata is"

  • Config that lives outside the data directory. Router plugins that keep their settings outside the main config file (AdGuard Home on OPNsense), host files in /etc (fstab, systemd units, daemon.json), and the deploy tool's own bootstrap compose and secrets. Back these up deliberately, and check a restore would bring them back.
  • Services whose data sits in the deploy tool's clone (relative bind mounts), see GitOps & updates.
  • Network gear with auto-backup switched off. Controllers like Omada or UniFi only write automatic backups if you enable them; then make sure the backup directory is inside what you back up.
  • The previous copy of a migrated directory. After moving data, keep the old copy for a couple of weeks of good archives, then delete it. And remember the old path is still in older archives until they age out, including anything sensitive it held.

Also exclude what you don't need (thumbnail caches, transcodes, model caches), so every night isn't spent re-uploading regenerable data.

borgmatic config passes validation, then a check fails

Symptom. borgmatic config validate is happy, the create step works, but the spot check aborts with data_tolerance_percentage must be less than or equal to the data_sample_percentage.

Cause. The spot check compares a sample of files; the allowed tolerance can't be larger than the sample. The validator doesn't check the relationship between the two.

Fix. Keep data_tolerance_percentage ≤ data_sample_percentage (e.g. sample 2, tolerance 1) in every config, including the ones for separate datasets, and run borgmatic check --only spot once by hand after changing them.

Sources & further reading