Skip to content

Restore & restore testing

A backup that has never been restored from is a hypothesis. This is the runbook I use for the borgmatic mesh: how to get a file, a SQLite database or a Postgres database back, the order to rebuild a dead server in, and the quarterly restore test that proves all of it still works.

Role names as on the borgmatic page: server1/server2/server3 back each other up, plus an offsite host.

Rules (for tests and real restores)

  • Restore to scratch paths and throwaway containers. Never restore over live data until you've looked at what came back.
  • Never run borgmatic restore for a database without an explicit target.
    • The configured pg_restore_command / psql_command point at the live containers.
    • A SQLite restore deletes the file at path before writing, unless you pass --restore-path.
  • Work as root in a private scratch directory on a disk with room:
DATE=$(date +%F); T=/var/tmp/restore-test/$DATE; install -d -m 0700 $T

0. Repository health

borgmatic repo-list                       # archives exist, newest is from last night, in EVERY repo
borgmatic info --archive latest           # sizes look plausible (compare with the last test)
borgmatic check --only repository --force --repository server1   # --repository = the repo's label
journalctl -u borgmatic --since -8d | grep -iE 'error|warn|fail' | tail

Pass: the newest archive is under 26 h old in a LAN repo and the offsite repo, there was an ntfy "OK" every night, and all push monitors are green.

1. Single files

Paths inside an archive are absolute paths without the leading /.

borgmatic list --archive latest --find 'etc/fstab'           # find it first
borgmatic extract --archive latest --path etc/fstab --path etc/komodo/stacks --destination $T
diff -rq $T/etc/komodo/stacks /etc/komodo/stacks | grep -v '\.git' | head   # only expected drift

Or browse and restore in the Borg UI web interface. For whole directories, borgmatic mount (FUSE, needs python3-pyfuse3/llfuse) lets you look around before extracting.

2. SQLite

# borgmatic stores dumps under borgmatic/sqlite_databases/... in the archive
borgmatic list --archive latest --find 'borgmatic/sqlite_databases/*'
# --restore-path is MANDATORY: without it the live DB file is deleted first
borgmatic restore --archive latest --data-source vaultwarden --restore-path $T/vw/db.sqlite3
sqlite3 $T/vw/db.sqlite3 'PRAGMA integrity_check;'        # -> ok
sqlite3 $T/vw/db.sqlite3 'select count(*) from users;'    # compare with the live DB

Boot test (proves the app accepts the DB, not just SQLite): start the app's image on the restored data with no network and signups off, check its health endpoint, then delete it.

docker run -d --name vw-restore-test --network none -v $T/vw:/data -e SIGNUPS_ALLOWED=false vaultwarden/server:<TAG>
sleep 15; docker exec vw-restore-test curl -fsS http://localhost:80/alive && echo "VW OK"
docker rm -f vw-restore-test

For git hosting (Forgejo/Gitea): restore its DB the same way, extract one bare repository and run git -C <repo>.git fsck --no-dangling and git log -1 --oneline main.

3. Postgres into a throwaway container

Custom-format pg_dump (one database)

borgmatic extract --archive latest --path borgmatic/postgresql_databases --destination $T
DUMP=$(find $T/borgmatic/postgresql_databases -type f -name app_db | head -1); ls -la "$DUMP"
docker run -d --name pg-restore-test --network none -e POSTGRES_HOST_AUTH_METHOD=trust postgres:17-alpine
sleep 10
docker exec -i pg-restore-test pg_restore -U postgres --create --dbname postgres --no-owner < "$DUMP"
docker exec pg-restore-test psql -U postgres -d app_db -c '\dt' | head
docker rm -f pg-restore-test

pg_dumpall (plain SQL, e.g. Immich)

Use the same image as production so extensions (Immich's vector extensions) are present:

DUMP=$(find $T/borgmatic/postgresql_databases -type f -name all | head -1)
IMG=$(docker inspect immich_postgres --format '{{.Config.Image}}')    # exact pinned image
docker run -d --name immich-pg-restore-test --network none -e POSTGRES_PASSWORD=restoretest -e POSTGRES_USER=postgres "$IMG"
sleep 20
docker exec -i immich-pg-restore-test psql -U postgres -v ON_ERROR_STOP=0 -q < "$DUMP" 2>&1 | grep -iE '^ERROR' | sort | uniq -c
docker exec immich-pg-restore-test psql -U postgres -d immich -c '\dx'     # extensions present
docker exec immich-pg-restore-test psql -U postgres -d immich -tc 'select count(*) from asset'
docker rm -f immich-pg-restore-test

Pass: no ERROR lines except "role/database already exists", extensions present, row counts match live to within the day's changes.

Real restore: stop the app, restore into the real container (that's what the configured psql_command / pg_restore_command are for: borgmatic restore --archive <name> --data-source immich), start the app.

4. Large separate repositories (photo library)

For a big repo, test a random sample instead of everything: list regular files, pick 20, extract them and compare checksums with the live copies.

C=/etc/borgmatic.d/photos.yaml; R=<repo label>; S=$T/photos; install -d -m 0700 $S
borgmatic -c $C list --archive latest --repository $R --path <library path without leading /> \
  --format '{type} {path}{NL}' | awk '$1=="-" {sub(/^- /,""); print}' | shuf -n 20 > $S/sample.txt
while read -r p; do borgmatic -c $C extract --archive latest --repository $R --path "$p" --destination $S; done < $S/sample.txt
while read -r p; do [ "$(sha256sum < "$S/$p")" = "$(sha256sum < "/$p" 2>/dev/null)" ] && echo "OK   $p" || echo "DIFF $p"; done < $S/sample.txt

Pass: 20/20 OK, or DIFF only for files you know were deleted or edited since.

5. Offsite

Once a quarter, on one host in rotation, repeat steps 1 and 2 with --repository offsite. This proves the offsite key, passphrase, SSH key and the provider account all still work, which is exactly the scenario where the LAN is gone.

6. Break-glass rehearsal (yearly)

On a desktop or a spare VM, using only the off-fleet copies of a passphrase and SSH key (not the servers' /etc/borgmatic/secrets):

sudo apt install borgbackup
export BORG_RSH='ssh -i ~/offsite-key'
read -rs BORG_PASSPHRASE; export BORG_PASSPHRASE   # typed, not in shell history
borg list ssh://<OFFSITE_USER>@offsite.example.com/./borg/server1
borg extract ssh://<OFFSITE_USER>@offsite.example.com/./borg/server1::<archive> etc/komodo opt/docker

If this fails, your backups can't be restored after losing the site. Fix that before anything else.

7. Clean up and record

docker ps -a --format '{{.Names}}' | grep -E 'restore-test' | xargs -r docker rm -f
rm -rf /var/tmp/restore-test/$DATE

Log each test (date, host, repos, steps passed, time taken, follow-ups) somewhere you'll see it next quarter. I keep a table at the bottom of the runbook in the infra repo, plus a recurring tracker issue.

Date Host / config Repos (LAN/offsite) Steps passed Time Follow-ups
YYYY-MM-DD server1 server2 + offsite 0, 1, 2, 3, 5 ~1 h

Rebuilding a whole server

Restore in dependency order. For the control-plane server (GitOps controller, git server, reverse proxy, alerting) that's:

  1. Base host: Debian 13, your user, Docker CE with the fleet's daemon.json, the fleet's host config (notifications, updates, SMART, free-space checks).
  2. Restore files from a peer's repo (or offsite): /opt/docker, /etc (deploy checkouts, firewall rules, /etc/borgmatic), /mnt/appdata, the systemd units.
  3. The deploy controller first (Komodo here), by hand with docker compose up -d, and wait for it to be healthy. If its database won't open the restored data directory, start it empty and load its own nightly export.
  4. The git server next, by hand: the controller deploys stacks from git, so it can't deploy the git server.
  5. Remote access (reverse proxy, tunnel), then alerting (ntfy, Uptime Kuma): every other alert flows through these.
  6. Everything else from the controller, one stack at a time.
  7. Re-enable the borgmatic timer and check the push monitors go green the next night.

Sources & further reading