Storage: lessons learned¶

Disk health, device names, network mounts, parity and pools. For backups see Backups. Each entry is symptom → cause → how to diagnose → fix.

A drive fails its SMART long test¶

Symptom. smartd (or your monitoring) alerts that a scheduled long self-test failed, usually with a read error at a specific LBA. The drive is otherwise working and the filesystem looks fine.

Cause. The long (extended) test reads every sector. A failure means at least one sector couldn't be read even after the drive's internal retries. That sector is now pending: the drive will remap it next time it's written. One pending sector on an old drive is a warning, not an emergency. A number that keeps growing means the drive is on its way out.

Diagnose.

# Self-test log: which test failed, at what power-on hour, and the first bad LBA
sudo smartctl -l selftest /dev/sdX

# The attributes that matter
sudo smartctl -A /dev/sdX | grep -Ei 'Reallocated|Pending|Offline_Uncorrectable|CRC|Power_On'

To see whether the bad sector holds any data, map the LBA to a filesystem block (for ext4: (LBA - partition start sector) × 512 / block size, then debugfs -R "icheck <block>"). If it's free space, nothing was lost.

Fix.

1. Write down a baseline now: the failing LBA, power-on hours, and the counts for 5/197/198/199. Next time it alerts, compare against these numbers instead of guessing whether things got worse.
2. Run another long test in a week or two (sudo smartctl -t long /dev/sdX) and compare. Stable = keep an eye on it. Growing pending/reallocated counts = replace it now.
3. Decide urgency by what's on the drive. A disk holding rolling camera recordings or cache can be replaced on your schedule. A disk with your only copy of anything is an emergency regardless of the numbers: back it up first.

Disk letters change between boots¶

Symptom. A disk that was /dev/sdc yesterday is /dev/sdd today. Scripts, SMART alerts or notes that refer to sdc now point at the wrong drive, which is dangerous right before a wipefs, mkfs or "replace the failing disk".

Cause. Kernel names (sda, sdb, …) are assigned in the order devices are detected, which can change with every boot, controller, firmware update or added USB drive. They were never stable.

Diagnose / fix. Identify disks by serial number or stable ID, never by letter:

lsblk -o NAME,SIZE,MODEL,SERIAL,MOUNTPOINT
ls -l /dev/disk/by-id/     # model+serial based, stable
ls -l /dev/disk/by-uuid/   # filesystem UUIDs, stable (use these in fstab)

• Use UUID= in /etc/fstab and /dev/disk/by-id/... in SnapRAID, smartd and scripts.
• Label the physical drive (or the bay) with the last few characters of its serial, so the one you pull is the one that's failing.
• Before any destructive command, re-check with lsblk -o NAME,SERIAL right then.

Network mounts that fail at boot¶

Symptom. A CIFS/NFS share in /etc/fstab isn't mounted after a reboot, or boot hangs for a minute and a half and drops to emergency mode, usually when the NAS that serves it boots more slowly than the client, or was down.

Cause. systemd tries to mount the share during boot. If the server isn't reachable yet, the mount fails. Without nofail a failed fstab mount can block boot. _netdev alone only orders it after the network is "up", not after the server is actually reachable.

Fix. Mount it on first access instead of at boot:

//<NAS_HOSTNAME>/<SHARE>  /mnt/<SHARE>  cifs  credentials=/etc/samba/creds-<SHARE>,uid=<PUID>,gid=<PGID>,_netdev,nofail,noauto,x-systemd.automount,x-systemd.mount-timeout=30  0  0

sudo systemctl daemon-reload
sudo systemctl restart remote-fs.target
ls /mnt/<SHARE>        # triggers the mount
findmnt /mnt/<SHARE>

Check journalctl -b | grep -i cifs if it still fails. Add x-systemd.idle-timeout=600 if you want it unmounted again after ten idle minutes.

sudo umount /mnt/<SHARE>; sudo systemctl stop mnt-<SHARE>.automount
sudo chattr +i /mnt/<SHARE>
sudo systemctl start mnt-<SHARE>.automount

SnapRAID sync silently blocked for weeks (deleted-file threshold)¶

Symptom. You find out by accident that the nightly SnapRAID sync hasn't run in a month. The log shows the same warning every night: too many deleted files. Parity is weeks old, so anything added since has no protection.

Cause. Wrapper scripts such as snapraid-aio-script refuse to sync when snapraid diff reports more deletions than DEL_THRESHOLD. That's a good safety net (mass deletions can mean a failed disk, a bad mount or ransomware, and syncing would write the loss into parity). But a legitimate clean-up triggers it too, and if the notification for "blocked" never arrives, it blocks forever. With SYNC_WARN_THRESHOLD=-1 it never forces a sync on its own.

Diagnose.

grep -E 'WARN|COMPLETED|threshold' /var/log/snapraid.log | tail -20
sudo snapraid diff | tail -20          # counts: equal, added, removed, updated, moved
sudo snapraid diff | grep '^remove' | less   # the actual deleted files

Look at which files were removed, not just how many. A tidy-up of one folder is fine; deletions scattered across every disk are not.

Fix.

1. If the deletions are legitimate, run one sync by hand (sudo snapraid sync) or use the script's own override for one run. Don't disable the check.
2. Set the thresholds to fit your normal churn: a DEL_THRESHOLD that covers a typical clean-up, plus the ratio-based ADD_DEL_THRESHOLD so a big reorganisation with as many additions as deletions can pass. A small SYNC_WARN_THRESHOLD (e.g. 5) forces a sync after that many blocked nights.
3. Make every run report somewhere that can't die with the problem: a quiet "OK" or a loud "needs attention" per run, plus a dead-man push monitor that goes DOWN when no run reports for 25 hours (see Monitoring).
4. Make the config root-owned if root's cron sources it.

A pooled filesystem fills up faster than the percentage suggests¶

Symptom. A mergerfs pool shows 95% used, which has been "fine" for months, then imports and downloads start failing with No space left on device while df still shows hundreds of GB free.

Cause. mergerfs stops placing new files on a branch once its free space drops below minfreespace, and on large pools a few percent is still a lot of data. Percentage alerts are the wrong unit: 85% of 20 TB leaves 3 TB, 95% leaves 1 TB, and at a growth rate of tens of GB a day that's weeks, not months. A parity disk always reads nearly full by design, so it needs to be excluded from percentage alerts.

Fix. Alert on free space in GiB for pools (e.g. warning at 500 GiB, critical at 250 GiB), percentage for ordinary filesystems, and skip the individual branches and the parity disk. Alert only on a level change, with a daily reminder while critical. Set free-space floors in the apps that write to the pool too (the *arr apps and qBittorrent have them), so they stop cleanly before the filesystem does.

Weekly fstrim errors on an SSD behind an HBA¶

Symptom. Every week the journal shows fstrim failing with critical target error … DISCARD for one SSD. The drive is healthy.

Cause. The SSD is attached through a SAS HBA (an LSI/mpt2sas card in IT mode, for example) that doesn't pass TRIM through to SATA SSDs.

Fix. It's noise, not damage. Either move the SSD to a motherboard SATA port, or leave it and stop fstrim.timer from touching that filesystem (fstrim only trims filesystems listed in fstab, so a drop-in for fstrim.service with an explicit list of mountpoints works).

Sources & further reading¶

• smartctl(8) and smartd.conf(5) — smartmontools
• Arch Wiki — S.M.A.R.T.
• Backblaze — Hard drive SMART stats
• Arch Wiki — Persistent block device naming
• systemd.mount(5) — x-systemd.automount
• Arch Wiki — fstab: automount with systemd
• chattr(1)
• snapraid-aio-script
• mergerfs — minfreespace
• Arch Wiki — Solid state drive: TRIM