Skip to content

Hosts, reboots & power: lessons learned

What happens when machines go down unexpectedly, and what makes them come back cleanly. Each entry is symptom → cause → how to diagnose → fix. Routine patching and reboot windows are in Host maintenance.

A server rebooted and nobody knows why

Symptom. journalctl --list-boots shows a boot that just ends: no shutdown sequence, a corrupted journal, filesystem recovery on the next boot. Other machines on the same circuit stayed up.

Cause. Either a kernel panic or a hardware-level reset (power supply, board, a loose connector, a brief power dip on one outlet). On Debian, kernel.panic defaults to 0, which means a panic hangs forever with nothing on screen you'll ever see on a headless box, and nothing in the journal. So if the machine came back by itself within seconds, it probably wasn't a panic.

Diagnose.

journalctl --list-boots | tail -5
journalctl -b -1 -n 50                       # last lines before the reset
ls /sys/fs/pstore/ /var/lib/systemd/pstore/  # kernel crash logs, if any were saved
sysctl kernel.panic
sudo smartctl -A /dev/nvme0 | grep -i unsafe # lifetime count of unsafe shutdowns

Check whether other hosts rebooted at the same moment. All of them within seconds = a site power event. Only one = suspect that machine's power or hardware.

Fix. Make the next one leave evidence:

/etc/sysctl.d/90-panic.conf
# Reboot 10 s after a kernel panic instead of hanging forever
kernel.panic = 10

sudo sysctl --system
systemctl status systemd-pstore.service   # archives pstore crash logs on the next boot

With that in place, a panic reboots the box and its log lands in pstore. If the next unexplained reset leaves pstore empty, look at hardware: the BIOS/IPMI event log, reseating power connectors, an overnight memtest86+. Without ECC memory, RAM errors aren't logged anywhere.

Everything comes back half-broken after a power cut

Symptom. After the power comes back, the servers boot, but containers can't resolve names, network mounts are missing, the tunnel is down, the router's own services (IDS, NTP, VPN) log DNS failures, and a few databases need recovery. One host may have given up waiting for the network and booted "degraded".

Cause. A site-wide power loss restarts everything at once, and the pieces come back at different speeds. The switch takes most of a minute to bring ports up. The router takes another minute before it answers DHCP and DNS. Servers boot faster than both, so network-online.target is reached (it only means "the network service started") long before there's a usable network. Docker starts containers in that gap and they keep the broken DNS config they started with. Databases that were killed mid-write need recovery.

Diagnose. Line up the boot logs of the router, the switch (its controller's event log) and each server for the same few minutes: link up, DHCP lease, resolv.conf written, dockerd started, containers started.

Fix. Make each host tolerant of a slow network, instead of trying to perfect the boot order across devices:

  • Bring the NIC up synchronously (auto in /etc/network/interfaces) — see allow-hotplug vs auto.
  • Pin Docker's DNS in /etc/docker/daemon.json ("dns": [...]) so containers forward to the router directly instead of a snapshot of the host's boot-time resolv.conf.
  • Optionally hold docker.service until the gateway's DNS answers: a small oneshot unit ordered Before=docker.service that loops on a dig against the router, with a timeout of several minutes (a router cold boot can take longer than you think). Make it give up rather than block boot forever.
  • Use automounts with nofail for network shares — see network mounts that fail at boot.
  • Give databases a longer stop window (stop_grace_period, daemon shutdown-timeout) — see Docker & systemd.
  • Alert on reboots ("host is back up" from each server, uptime-went-backwards for the router), so a power event isn't something you discover days later.

Power blips several times a day: you need a UPS

Symptom. Every machine in the rack cold-boots several times in a few hours, one reset even interrupting a boot that was still in progress. Each time, journals are corrupted and filesystems on the data disks run recovery.

Cause. No UPS. Every brownout or flicker becomes a hard power-off, which is the worst case for disks, databases and parity arrays.

Fix. A UPS sized for the router, switch and servers, plus software that shuts things down in order before the battery runs out:

  • NUT (Network UPS Tools): the machine the UPS's USB cable plugs into runs the NUT server (upsd), the others run upsmon as clients. OPNsense has an os-nut plugin, so the firewall can be the server and shut down last.
  • Shut down the servers when the battery reaches a threshold, not when it's empty, and test it once by pulling the plug.
  • Put the switch and the router on the UPS too. Servers that survive a power cut but lose their network for two minutes still end up with the problems above.
  • Until there is a UPS, don't turn on unattended firmware updates for the router: a power cut during an update is the one you can't easily recover from.

Sources & further reading