Skip to content

Monitoring & alerts: lessons learned

When the alerting itself is the problem: noise during scheduled jobs, monitors that check the wrong thing, or silence when something is actually down. How the monitoring stack is built is in Monitoring. Each entry is symptom → cause → how to diagnose → fix.

Alerts fire every night during a scheduled job (the maintenance window expired)

Symptom. Every night at the same time you get a burst of "service down" alerts, for example while SnapRAID sync/scrub pauses your containers, or while a backup stops a database. You did set up a recurring maintenance window in Uptime Kuma to silence exactly this, and it used to work.

Cause. One or more of:

  • The window has an end date. Uptime Kuma's recurring maintenance has an optional effective date range. Once the current time is past the end date, the maintenance shows as Ended and stops suppressing alerts, even though the recurring schedule is still defined. Nothing warns you when that happens.
  • The window doesn't line up with the job. The job's start time drifted (a new timer, a job that now runs after another one), or it takes longer than the window.
  • Timezone mismatch. The maintenance has its own timezone setting (or "same as server"), and the server, the container and your cron/timer may all disagree.
  • Not every monitor is in the window. New monitors, or monitors in a different group, aren't covered.

Diagnose.

  1. In Uptime Kuma open Maintenance and check the status badge. Ended or Scheduled (instead of active/under maintenance at the right time) is the giveaway.

  2. Compare the window with the job's real run times:

    systemctl list-timers                         # when timers actually fire
journalctl -u <job>.service --since -7d | grep -E 'Started|Finished|Deactivated'
timedatectl                                   # host timezone

  3. Check which monitors or groups are attached to the maintenance.

Fix.

  • Remove the end date (leave the effective date range open) for windows that cover a permanent recurring job.
  • Use a cron schedule that starts a few minutes before the job, with a duration comfortably longer than the slowest run you've seen.
  • Set the maintenance timezone explicitly to the same zone your timers run in.
  • Attach the maintenance to a group (e.g. "Media server") rather than individual monitors, so new monitors in that group are covered automatically.
  • If other alert sources also notify (a deployment tool, a container watchdog), give them the same quiet window.

Alert on the job, not just the services

Silencing services during maintenance is only half of it. Have the job itself report success (for example an Uptime Kuma push monitor the job pings when it finishes) so you're alerted when the nightly sync didn't run, not only when it did.

The monitor says UP, but the service is dead

Symptom. A service has been down for days, yet its uptime monitor is green. Or a whole group of monitors on public hostnames never alerts, even during an outage.

Cause. The monitor checks something that answers regardless of the backend:

  • An SSO login page. Behind Cloudflare Access, Authelia or Authentik, an unauthenticated request gets a redirect to the login page. HTTP monitors follow redirects by default, the login page returns 200, and the check is "UP" whether or not the app behind it exists.
  • The reverse proxy itself. The proxy answers (a 502 page, a default page) when the backend is gone, and a monitor that accepts any status passes.
  • A port that's always open on a host that runs more than one thing.

Diagnose. Run the monitor's request by hand and look at where it ends:

curl -sSL -o /dev/null -w '%{http_code} %{url_effective}\n' https://<SERVICE>.<YOUR_DOMAIN>/

An url_effective on your SSO provider's domain means you're monitoring the login page.

Fix.

  • Monitor the internal URL (http://<YOUR_SERVER_IP>:<PORT>/<health-path>) for each service, and use a keyword check ("status":"ok", MediaContainer, "healthy":true) rather than "any 2xx".
  • Check the public path separately, end to end, with one monitor against an endpoint that legitimately bypasses SSO (a health endpoint of an app that does its own auth) or with an SSO service token. One good end-to-end check beats ten that only see the login page.
  • Turn off "follow redirects" on monitors where a redirect means something is wrong.

Monitors pointing at hosts that no longer exist

Symptom. A handful of monitors are permanently red, or throw thousands of ENOTFOUND errors a day. Everyone has learned to ignore them.

Cause. A host was decommissioned and its services moved, but the monitors still point at the old name. Permanent red is worse than no monitor: it trains you to ignore the dashboard, and it hides the next real failure.

Fix. When you retire or move anything, update the monitors in the same change. Repoint moved services at their new endpoints. Pause (rather than delete) monitors for things that are gone if you want to keep their history. Review the monitor list now and then for anything that hasn't been green in weeks.

A job failed for weeks and nobody heard about it

Symptom. You discover by accident that a nightly job (parity sync, backup, offsite copy, certificate renewal) hasn't succeeded in a month. It was set up to notify.

Cause. One of these, usually more than one:

  • The notification only fires on failure, and the failure mode was "never ran" or "refused to run" (a safety threshold blocked it, the timer was disabled, the host was off). No run, no failure, no message.
  • The notification path depended on the thing that broke. The alert needed DNS, the internet, a tunnel, or a mail server, and that was exactly what was down. A webhook to a chat service times out on DNS; smartd -m root mails a host with no mail transfer agent; a healthcheck URL is still a placeholder.
  • The alerts went to an address nobody reads.

Fix.

  1. Dead-man switches for every scheduled job. The job pings a push monitor (Uptime Kuma push, healthchecks.io) on success and failure. The monitor goes DOWN on a failure report or when no report arrives within the expected interval plus a margin (for a nightly job, about 25 hours).
  2. Separate monitors per destination. A backup that goes to a LAN target and an offsite target gets two monitors, so a good run to one can't mask a failed run to the other.
  3. Alert over the LAN, by IP. Point hooks at your notification server's LAN address (http://<YOUR_SERVER_IP>:<PORT>), not its public name through a tunnel, so alerts still arrive when DNS, the tunnel or the internet are the problem.
  4. Report every run, not only failures, for the jobs you care most about. A quiet "OK" on a low-priority topic tells you it ran.
  5. Test the alert path itself: send a test notification from each source (smartd has -M test, most tools have a "send test" button).

The same failure every night, and you still didn't notice

Symptom. A backup failed on most nights of a week. Each time you got one notification, followed a day later by an automatic "recovered" when a run succeeded. Each looked like a blip.

Cause. One-off alerts that clear themselves don't convey a pattern. Seven "failed" + seven "recovered" messages read as noise.

Fix.

  • Escalate on streaks. Keep a per-job counter of consecutive failures. From the second failure in a row, send at the highest priority, and send one "recovered after N failures" when it clears.
  • Send one daily summary of every scheduled job: last result, when, current streak, problems first, with a title like Backups: 2 FAILING, 1 overdue. Reading the push monitors' state from your monitoring tool's database (read-only) means new jobs show up automatically.

One blip, four notifications

Symptom. A single failed check produces a burst of messages: the monitor went down, its group went down, the monitor came up, the group came up.

Cause. Notifications are attached to both the monitor and the group it belongs to, and the check alerts on the first failure.

Fix. Attach notifications to monitors (or to groups, not both). Give checks that cross the internet a couple of retries (retries: 2, short retry interval) before they count as DOWN, so a single dropped request doesn't page you.

The monitoring host is the single point of failure

Symptom. Nothing alerts during the worst outage of all: the host that runs your monitoring and your notification server is down (or the whole site lost power).

Cause. Every alert path ends at one box. If it's off, nothing can report that it's off. Silent reboots are related: if a host reboots and comes back, nothing notices unless something is watching uptime.

Fix.

  • Add an external dead-man switch: the monitoring host pings a hosted service (healthchecks.io has a free tier) every few minutes, and that service emails or pushes when the pings stop. A second monitoring instance on a different machine (a Pi) works too.
  • Alert on reboots: a boot-time job that posts "host is back up" (with kernel and container health), and a poller that notices uptime going backwards on devices that can't run your scripts (the router).
  • Monitor the deploy tool's own failure events, not just state changes (see GitOps & updates).
  • Don't let a permanent warning (a crash-report banner, an unhealthy container nobody fixes) sit there. A status that is always red can't tell you anything new.

Sources & further reading