Monitoring & alerts¶

How the lab notices that something is wrong, and how it tells me. Everything here is self-hosted and runs as Docker stacks on the servers, except the host-level checks (SMART, disk space, update reboots), which are systemd units on each host. Those are covered in Host maintenance.

What alerts where¶

Every source publishes to ntfy. ntfy is the only thing that talks to my phone and desktop, so there's one app to configure and one place to look.

flowchart LR
    subgraph sources[Publishers]
        deploy[Deploy tool<br/>host/service down, failed deploys]
        git[Git server webhooks<br/>update PRs, merges]
        kuma[Uptime Kuma<br/>monitor DOWN/UP]
        beszel[Beszel<br/>metric thresholds]
        borg[borgmatic<br/>every backup run]
        host[Host scripts<br/>SMART, disk space,<br/>reboots, apt, SnapRAID]
    end
    subgraph ntfy[ntfy]
        alerts([alerts])
        updates([updates])
        backups([backups])
        hosts([hosts])
    end
    deploy --> alerts
    deploy --> updates
    git --> updates
    kuma --> alerts
    beszel --> alerts
    borg --> backups
    borg --> alerts
    host --> alerts
    host --> hosts
    host --> backups
    host --> updates
    alerts --> phone[Phone app]
    updates --> phone
    backups --> phone
    hosts --> phone
    alerts --> desk[Desktop client<br/>KDE notifications]
    updates --> desk
    backups --> desk
    hosts --> desk
    borg -. push .-> kuma
    host -. push .-> kuma

The dotted lines are the dead-man switches: nightly jobs also push their result to an Uptime Kuma push monitor, and Kuma alerts when a push doesn't arrive. See Uptime Kuma.

Topic design¶

Four topics, split by what I need to do about the message, not by which tool sent it:

Why separate topics instead of one "homelab" topic:

• Notification settings per topic. alerts can be set to urgent and break through do-not-disturb, while updates and hosts stay silent and are just there when I look. With one topic, either everything is loud (and I start ignoring it) or nothing is.
• Alert fatigue is the real enemy. "Upgraded 4 packages" in the same stream as "disk failing" trains you to swipe both away. Keeping alerts for things that need action keeps it worth reading.
• Least-privilege publishing. Each publisher gets a token that can only write, and only to the topics it needs (Uptime Kuma → alerts only, borgmatic → backups + alerts). A leaked token can't read anyone's messages or spam topics it was never given. See ntfy.
• Priorities still matter inside a topic. borgmatic's OK messages are priority min, its FAILED ones high. Uptime Kuma sends DOWN at 5 and UP at 3.

Design rules that came out of things breaking¶

• Alert on the job, not only the service. A nightly sync once stopped for a month without anyone noticing, because nothing was down, it just never finished. Every nightly job now reports to a Kuma push monitor that goes DOWN after ~25 h of silence.
• Use LAN addresses for alert delivery. Publishers post to ntfy's LAN address (or over the shared Docker network) first, so an alert about DNS or the tunnel being down can still be delivered. Hostnames and the public URL are only the fallback.
• Silence planned work, but only exactly that. Maintenance windows in Uptime Kuma and the deploy tool cover the nightly parity sync for the services it pauses, and nothing else. A run that overruns the window alerts normally. (Lessons learned)
• Report level changes, not states. Disk-space and firewall checks alert when something goes from ok → warning → critical (and back), not every hour it stays bad.
• Alerting has a single point of failure. If the host running ntfy and Kuma dies, nothing reports it. An external dead-man switch (a free healthchecks.io check pinged every few minutes, or a second Kuma on separate hardware) closes that gap.

Sources & further reading¶

• ntfy documentation
• Uptime Kuma wiki
• Beszel documentation
• Dozzle documentation
• syslog-ng admin guide
• healthchecks.io