Borg / Borgmatic¶

borgmatic is simple, configuration-driven backup software for servers and workstations. Protect your files with client-side encryption. Backup your databases too. Monitor it all with integrated third-party services.

https://torsion.org/borgmatic/

This page is the pattern every one of my servers follows. It's written with role names: three servers (server1, server2, server3) that back each other up, plus one offsite host somewhere else.

The design¶

server1 ──► server2:/srv/borg/server1   + server3:/srv/borg/server1   + offsite:~/borg/server1
server2 ──► server1:/srv/borg/server2   + server3:/srv/borg/server2   + offsite:~/borg/server2
server3 ──► server1:/srv/borg/server3   + server2:/srv/borg/server3   + offsite:~/borg/server3

• borgmatic on every server, config in git, run nightly by a systemd timer. Start times are staggered per host so two servers aren't busy with each other's backups at once.
• Three copies of every server: the two other servers (fast restores over the LAN) and one offsite host (the "house burned down" copy).
• Each receiving host has a borg user that only runs borg serve. Every client's SSH key is pinned to append-only access to its own repository, with a storage quota. A compromised server can't read the other servers' backups, can't delete its own history, and can't fill a peer's disk.
• Every run reports to ntfy, and a dead-man switch in Uptime Kuma notices a run that never happened.
• Databases are dumped, not copied as live files: Postgres through docker exec, SQLite with .dump.
• Restores are tested every quarter: see Restore & restore testing.
• A web UI, Borg UI, browses and restores from all repos. It never runs backups.

Why this shape. My earlier attempts (a restic/Backrest setup, Kopia, a single NAS share) all had one thing in common: they died silently, and I found out months later. So the requirements became: no single box holds the only copy, a compromised box can't destroy the backups, and silence is an alert.

Installation¶

Debian 13 has Borg 1.4 in its main repo. borgmatic comes from PyPI through pipx --global, pinned to one version, because the configs below use borgmatic 2.x features (commands: hooks, database labels, {credential ...}) that Debian's own borgmatic package is too old for.

sudo apt update
sudo apt install borgbackup sqlite3 pipx
# --global installs to /usr/local/bin for every user, including root and systemd
sudo pipx install --global 'borgmatic==2.1.9'   # pin; bump deliberately
borg --version && borgmatic --version

Other ways to Install¶

You can also install via docker/podman. The official docs cover this. I chose to just run it natively as it is a root privileged system backup that manages files. Running via a container can work fine, just a bit extra work. I like to run infrastructure services outside of a container, if those services manage containers. Either way works.

Receiving side (every server, and the offsite host)¶

The borg user¶

sudo useradd --system --create-home --home-dir /var/lib/borg --shell /bin/sh borg
sudo usermod -p '*' borg          # no password, but not "locked" (sshd refuses key logins for '!' accounts)
sudo install -d -o borg -g borg -m 0700 /srv/borg /srv/borg/server2 /srv/borg/server3 /var/lib/borg/.ssh

Put the repos on a disk that isn't the thing being protected: on the NAS that's the boot NVMe, not the media pool. And exclude the repo directory from that host's own backup: a repo is never backed up into another repo.

authorized_keys: append-only, one repo, a quota¶

command="borg serve --append-only --restrict-to-repository /srv/borg/server2 --storage-quota 40G",restrict ssh-ed25519 AAAA... root@server2 borgmatic
command="borg serve --append-only --restrict-to-repository /srv/borg/server3 --storage-quota 150G",restrict ssh-ed25519 AAAA... root@server3 borgmatic

sudo install -m 600 -o borg -g borg authorized_keys /var/lib/borg/.ssh/authorized_keys

Compaction on the server¶

Because clients are append-only, their prunes free no space. The server compacts its repos weekly, as the borg user, which needs no passphrase:

#!/bin/bash
set -u; rc=0
for repo in /srv/borg/*/; do
  [ -f "$repo/config" ] || continue
  borg compact --threshold 10 "$repo" || rc=1
done
exit $rc

[Unit]
Description=Compact borg repositories served to other hosts

[Service]
Type=oneshot
User=borg
Group=borg
ExecStart=/usr/local/sbin/borg-compact-served
Nice=19
IOSchedulingClass=idle
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=full

[Unit]
Description=Weekly compaction of served borg repositories

[Timer]
# Weekly, at a time clear of the nightly backups and update reboots
OnCalendar=Sat *-*-* 14:00:00
RandomizedDelaySec=30min
FixedRandomDelay=true
Persistent=true

[Install]
WantedBy=timers.target

The offsite host¶

Any box you can SSH into and run borg serve on works: a small VPS, a friend's server, a storage provider with Borg support. If you can't install packages there, use the official standalone borg binary from the releases page, verify its GPG signature, put it in ~/bin/borg, and use command="~/bin/borg serve --append-only ..." in that account's authorized_keys the same way. Keep the same Borg major version as the clients (1.4.x here). If you don't control the offsite account's other keys, edit only your own lines and back the file up first.

With no root there, nothing compacts the offsite repos on a timer. Run borg compact on them by hand every few months, from a trusted machine with full (non-append-only) access.

Client side (every server)¶

Secrets¶

Generated on each host, never copied between hosts:

sudo install -d -m 0700 /etc/borgmatic /etc/borgmatic/secrets /etc/borgmatic.d
# A random passphrase per host
sudo sh -c 'umask 077; head -c 48 /dev/urandom | base64 -w0 > /etc/borgmatic/secrets/passphrase'
# An SSH key used only for borg
sudo ssh-keygen -q -t ed25519 -N '' -C "root@$(hostname -s) borgmatic" -f /etc/borgmatic/secrets/id_ed25519_borg

Pinned host keys¶

Peers are addressed by IP, with their SSH host keys pinned in a borgmatic-only known_hosts (StrictHostKeyChecking=yes). A DNS outage then can't skip a night, and a changed host key fails loudly instead of being accepted.

192.0.2.11 ssh-ed25519 AAAA...   # server1
192.0.2.12 ssh-ed25519 AAAA...   # server2
offsite.example.com ssh-ed25519 AAAA...

Take the keys from /etc/ssh/ssh_host_ed25519_key.pub on each peer over a connection you already trust, not from a first ssh prompt.

The configuration¶

One file per host in /etc/borgmatic.d/. This is server2's, trimmed of host-specific paths:

source_directories:
    - /mnt/appdata          # container configs + data
    - /etc                  # includes the GitOps checkouts with their .env files
    - /opt/docker
    - /usr/local/sbin
    - /root
    - /home/<YOUR_USER>

source_directories_must_exist: true
one_file_system: true

exclude_patterns:
    # Repos that the other servers back up INTO this host. Never back up a repo inside another repo.
    - /mnt/appdata/borg
    # Git object stores of the deploy tool's clones (re-cloned from git)
    - /etc/komodo/stacks/*/.git
    - /etc/komodo/repos/*/.git
    # Live database directories: dumped below instead
    - /mnt/appdata/immich/db
    # Regenerable caches
    - /mnt/appdata/immich/machinelearning
    - /mnt/appdata/lost+found
    - /home/<YOUR_USER>/.cache
    - /root/.cache

exclude_caches: true
exclude_if_present:
    - .nobackup

repositories:
    - path: ssh://borg@192.0.2.11/srv/borg/server2
      label: server1
      encryption: repokey-blake2
    - path: ssh://borg@192.0.2.13/srv/borg/server2
      label: server3
      encryption: repokey-blake2

encryption_passcommand: cat /etc/borgmatic/secrets/passphrase
ssh_command: ssh -i /etc/borgmatic/secrets/id_ed25519_borg -o BatchMode=yes -o IdentitiesOnly=yes -o UserKnownHostsFile=/etc/borgmatic/known_hosts -o StrictHostKeyChecking=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=6
compression: auto,zstd,3
lock_wait: 300
retries: 1
retry_wait: 300

keep_daily: 7
keep_weekly: 4
keep_monthly: 6

checks:
    - name: repository
      frequency: 1 week
    - name: archives
      frequency: 1 month
    - name: spot
      frequency: 1 week
      count_tolerance_percentage: 10
      data_sample_percentage: 5
      data_tolerance_percentage: 2
check_last: 3

# (database dumps, hooks and notifications: see the sections below)

What's deliberately not backed up: the media library (it has SnapRAID parity, and it can be re-acquired), Plex's regenerable metadata/thumbnails, NVR recordings, model and download caches, logs. A large photo library gets its own repository and key with a bigger quota, so it can't crowd out the small, critical config backups. Each exclude carries a comment with its reason in the real file.

sudo chmod 600 /etc/borgmatic.d/*.yaml
sudo borgmatic config validate
sudo borgmatic repo-create --encryption repokey-blake2   # once, creates the repo on every target

(borgmatic before 1.9.0 calls this action init.)

Database dumps¶

borgmatic streams each dump straight into the archive, so there's no dump file left lying on disk.

# Postgres in a container: run the client tools INSIDE the container over its local socket.
# No password, no published port, and the tools always match the server version.
postgresql_databases:
    - name: all                 # pg_dumpall: every DB + roles (the method Immich documents)
      label: immich
      username: postgres
      pg_dump_command: docker exec immich_postgres pg_dumpall
      psql_command: docker exec -i immich_postgres psql
    - name: app_db              # one DB, custom format (pg_restore-able)
      label: app
      username: app_user
      pg_dump_command: docker exec app-database pg_dump
      pg_restore_command: docker exec -i app-database pg_restore
      psql_command: docker exec -i app-database psql

# SQLite: borgmatic runs the host's `sqlite3 <path> .dump`, a consistent snapshot even while the app writes (WAL).
# The raw files are in the archive too, but restore from these dumps.
sqlite_databases:
    - name: vaultwarden
      path: /mnt/appdata/vaultwarden/data/db.sqlite3
    - name: sonarr
      path: /mnt/appdata/sonarr/config/sonarr.db

Waiting for the NAS's parity run¶

On the NAS, SnapRAID pauses the containers during its nightly sync/scrub. A docker exec pg_dumpall against a paused container hangs, so the NAS's borgmatic is scheduled after the SnapRAID window and checks for it:

commands:
    - before: configuration
      run:
          # Wait up to 90 min for SnapRAID, then give up (the failure is reported)
          - >-
            for i in $(seq 1 90); do
            if pgrep -x snapraid >/dev/null || pgrep -f "[s]napraid-aio-script" >/dev/null;
            then sleep 60; else break; fi; done;
            if pgrep -x snapraid >/dev/null; then echo "SnapRAID still running after 90 min" >&2; exit 1; fi
          # Refuse to dump a paused database instead of hanging
          - >-
            if docker ps --filter status=paused --format '{{.Names}}' | grep -qx immich_postgres;
            then echo "immich_postgres is paused" >&2; exit 1; fi

The offsite copy¶

A second config file includes the first and only swaps the repository, upload limit and notification titles. It's separate because borgmatic's upload_rate_limit applies to the whole config: this way the LAN copies run at full speed and only the WAN upload is capped (to about a third of the uplink, so remote access stays usable).

# "zz-" sorts after server2.yaml, so borgmatic does the LAN copies first.
<<: !include /etc/borgmatic.d/server2.yaml

repositories: !retain
    - path: ssh://<OFFSITE_USER>@offsite.example.com/home/<OFFSITE_USER>/borg/server2
      label: offsite
      encryption: repokey-blake2

upload_rate_limit: 6250        # KiB/s, ~51 Mbit/s

ntfy:
    finish:
        title: "borgmatic OK: server2 (offsite)"
    fail:
        title: "borgmatic FAILED: server2 (offsite)"

# Its own dead-man monitor, so a good offsite run can't mask a failed LAN run (and vice versa)
uptime_kuma:
    push_url: !include /etc/borgmatic/uptime-kuma-url-offsite

Scheduling: systemd timer¶

Earlier versions of this page used the sample cron file. The servers now use systemd units, based on upstream's sample with the sandboxing loosened where the dumps need it:

[Unit]
Description=borgmatic backup
Wants=network-online.target
After=network-online.target docker.service

[Service]
Type=oneshot
RuntimeDirectory=borgmatic
RuntimeDirectoryMode=0700
StateDirectory=borgmatic
StateDirectoryMode=0700
PrivateTmp=yes
NoNewPrivileges=yes
LockPersonality=yes
# docker exec dumps need the Docker socket; sqlite3 .dump on WAL DBs must write -shm files,
# so no ProtectSystem=strict, no PrivateNetwork, no dropped CAP_DAC_OVERRIDE.
ProtectHome=no
UMask=0077
# Be gentle with the host's real workload
Nice=19
CPUSchedulingPolicy=batch
IOSchedulingClass=best-effort
IOSchedulingPriority=7
Restart=no
LogRateLimitIntervalSec=0
# A first offsite seed can take days; the dead-man switch catches a real hang
TimeoutStartSec=infinity
# Let network, DNS and containers settle after boot (Persistent=true catch-up runs)
ExecStartPre=/bin/sleep 60
# Block suspend/shutdown during a run
ExecStart=/usr/bin/systemd-inhibit --who="borgmatic" --what="sleep:shutdown" --why="Prevent interrupting scheduled backup" \
    /usr/local/bin/borgmatic --verbosity -2 --syslog-verbosity 1
# No [Install]: only the timer starts this

[Unit]
Description=Nightly borgmatic backup

[Timer]
# Per-host start time comes from a drop-in (borgmatic.timer.d/schedule.conf); stagger the hosts
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=10min
FixedRandomDelay=true
# If the host was off at the scheduled time, run once it's back up
Persistent=true
AccuracySec=1min

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl start borgmatic.service      # first run by hand; watch it
journalctl -u borgmatic -f
sudo systemctl enable --now borgmatic.timer # after a good first run

Line the night up so jobs don't collide: app-level DB exports first, then each host's borgmatic, the NAS's after its SnapRAID run. Update reboots wait for a running borgmatic and for incoming borg serve sessions, so a peer rebooting for updates can't cut another server's backup off.

Notifications and the dead-man switch¶

Three layers, because each one catches a failure the others can't:

ntfy:
    server: https://ntfy.<YOUR_DOMAIN>
    topic: backups
    # A write-only token for this publisher, read from a root-only file
    access_token: "{credential file /etc/borgmatic/secrets/ntfy-token}"
    finish:
        title: "borgmatic OK: server2"
        message: "Backup, prune and checks finished"
        tags: white_check_mark
        priority: min
    fail:
        title: "borgmatic FAILED: server2"
        message: "Check: journalctl -u borgmatic -n 200"
        tags: rotating_light
        priority: high
    states:
        - finish
        - fail

# The push URL carries a token, so it lives in a root-only file. borgmatic doesn't resolve
# {credential ...} in push_url, hence !include.
uptime_kuma:
    push_url: !include /etc/borgmatic/uptime-kuma-url
    states:
        - finish
        - fail

commands:
    # Consecutive-failure escalation: urgent alert from the 2nd failed run in a row, one "recovered"
    # message after. The script must always exit 0, or borgmatic marks the backup itself failed.
    - after: configuration
      when: [create]
      states: [finish]
      run:
          - /usr/local/sbin/backup-streak ok {configuration_filename}
    - after: configuration
      when: [create]
      states: [fail]
      run:
          - /usr/local/sbin/backup-streak fail {configuration_filename}

Why the streak hook: an offsite run once failed six nights in a row. Each night produced one high-priority message, and the next good LAN run looked like a recovery, so nothing stood out. Now the second consecutive failure of the same config is urgent.

A morning summary (a small timer that reads the Kuma push monitors' last heartbeats) sends one message listing every backup job with OK / FAILING / overdue, problems first. One glance instead of reading the night's ntfy feed.

apprise:
    services:
        - url: discord://webhook_id/webhook_token
          label: discord
    send_logs: true
    states:
        - finish
        - fail

Operating it¶

sudo borgmatic repo-list                       # archives in every repo
sudo borgmatic info --archive latest
sudo borgmatic create --verbosity 1 --stats    # an extra run now
journalctl -u borgmatic -n 200                 # last nightly run

Restores and the quarterly restore test: Restore & restore testing. Browsing in a browser: Borg UI.

Sources & further reading¶

• borgmatic documentation and configuration reference
• borgmatic: back up your databases
• borgmatic: monitor your backups
• borgmatic: provide your passwords
• borg serve (--append-only, --restrict-to-repository, --storage-quota)
• Borg: append-only mode notes
• Borg releases (standalone binaries + signatures)