Skip to content

Troubleshooting

Problems I actually hit, written up so the next person (or me, six months later) doesn't have to rediscover them. Every entry follows the same shape: symptom → cause → how to diagnose → fix.

  • Hardening & hygiene checklist

    Everything worth making sure is done: pinned images, healthchecks, append-only and restore-tested backups, dead-man switches, secrets out of git, SSO and MFA, cert expiry, UPS.

  • Networking & DNS

    AAAA records with no IPv6 route, boot-time DNS races, AdGuard rate limits, CrowdSec banning your own server, desktop VPNs tunnelling the LAN, Tailscale rules that match nobody, wildcard DNS, proxy cert drift, real client IPs, rebind protection, discovery across VLANs.

  • Storage

    Failing SMART long tests and baselining, disk letters that change between boots, network mounts that fail at boot, SnapRAID syncs blocked by thresholds, pools filling up, TRIM behind an HBA.

  • Docker Docker & systemd

    Stale or wedged DNS inside containers, restart loops from leftover files in /tmp, timers that fire as soon as you enable them, wrong port mappings, broken healthchecks, secrets in ps, watchdogs vs nightly pauses, database stop windows, empty compose variables.

  • Monitoring & alerts

    Maintenance windows that silently expire, monitors that only see an SSO login page, dead-man switches, failures that repeat every night, alert noise, the monitoring host as a single point of failure.

  • GitOps & updates

    Deploys aborted by registry rate limits, service_healthy timeouts, data inside the deploy tool's clone, pins that downgrade, floating tags, Renovate quirks, untracked config files.

  • Backups

    Backup tools that never worked, restore drills, append-only targets and restricted keys, offsite bandwidth and first seeds, things that silently aren't in the backup.

  • Hosts, reboots & power

    Unexplained resets and kernel.panic, what breaks when the whole rack loses power, and why you want a UPS with NUT.

  • Security pitfalls

    Secrets in git, password reuse, "no login for local addresses" behind a proxy, admin tools that are root in disguise, ports on 0.0.0.0, WAN forwards to the main proxy.

  • Home Assistant Smart home

    Frigate API debugging, masks after a model change, tracker identity swaps, Zigbee button automations.

Elsewhere in these docs