Skip to content

Docker & systemd: lessons learned

Container and service-manager gotchas, mostly the kind that only show up after a restart, a reboot or a few weeks of nobody looking. Each entry is symptom → cause → how to diagnose → fix.

A container keeps stale DNS settings

Symptom. A container can't resolve names (or uses an old DNS server) even though the host resolves fine. Often right after a reboot, a DHCP change, or connecting a VPN on the host.

Cause. Docker writes the container's /etc/resolv.conf from the host's when the container starts. Later changes on the host aren't pushed into running containers.

Diagnose.

cat /etc/resolv.conf                        # host
docker exec <container> cat /etc/resolv.conf  # container

Fix. docker restart <container> picks up the current settings. To stop it happening, make the host's network fully up before Docker starts (see Networking & DNS → after a reboot), or pin DNS servers for containers with "dns": ["<YOUR_DNS_SERVER_IP>"] in /etc/docker/daemon.json (then restart Docker) or dns: per service in compose.

Redis (or another container) restart-loops: "Permission denied" on a file in /tmp

Symptom. After a container restart (not a fresh create), it crash-loops with Permission denied writing a file in /tmp or another world-writable sticky directory, even though it worked the first time.

Cause. The entrypoint writes a config file to /tmp (e.g. /tmp/redis.conf), then drops privileges. On restart the old file is still there, owned by the unprivileged user. Debian and most current distros set fs.protected_regular = 2, which stops root from opening (with O_CREAT) an existing file in a sticky world-writable directory when that file is owned by someone else. So the entrypoint, running as root, can't overwrite its own leftover file.

Diagnose.

docker logs --tail 20 <container>
sysctl fs.protected_regular           # 2 = strict
docker exec <container> ls -l /tmp    # if it stays up long enough

Fix. Remove the stale file before writing it. In a custom entrypoint or command::

command: ["sh", "-c", "rm -f /tmp/redis.conf && <original command that writes /tmp/redis.conf>"]

Even better, write the file somewhere that isn't world-writable, or mount the config read-only from the host. Don't "fix" it by lowering fs.protected_regular on the host. That protection exists to stop a whole class of /tmp attacks.

A new systemd timer fires immediately when you enable it

Symptom. You add a timer with Persistent=true (e.g. a nightly backup at 03:00), enable it in the afternoon, and the job runs right away, or at the next boot, instead of waiting for 03:00.

Cause. Persistent=true records the last trigger time in a stamp file (/var/lib/systemd/timers/stamp-<name>.timer) and, when the timer is activated, runs the job once if it "missed" a scheduled time while it wasn't active. A brand-new timer has no stamp, so systemd may decide it missed today's run.

Diagnose.

systemctl list-timers <name>.timer
ls -l /var/lib/systemd/timers/
journalctl -u <name>.service --since today

Fix. If an immediate catch-up run would be a problem (a heavy job, or one that alerts), create the stamp before enabling:

sudo touch /var/lib/systemd/timers/stamp-<name>.timer
sudo systemctl enable --now <name>.timer
systemctl list-timers <name>.timer     # NEXT should be the scheduled time

Keep Persistent=true anyway. It's what makes a job that was due while the machine was off run when it comes back.

Docker's embedded DNS returns SERVFAIL for everything external

Symptom. One container (or several on different networks) suddenly can't reach anything on the internet: Resource temporarily unavailable, EAI_AGAIN, or an app that reports its upstreams as "unavailable due to failures". Container-to-container traffic still works, and the host resolves fine.

Cause. Containers on user-defined networks ask Docker's embedded resolver at 127.0.0.11, which forwards to the upstream servers the daemon knows about. That forwarder can get wedged (often right after a dockerd restart, or when the upstream timed out during boot) and keep answering SERVFAIL until the container is restarted.

Diagnose. Compare the embedded resolver with your real DNS server from inside the container:

docker exec <container> cat /etc/resolv.conf                # nameserver 127.0.0.11
docker exec <container> nslookup example.com                 # fails
docker exec <container> nslookup example.com <YOUR_DNS_SERVER_IP>   # works
journalctl -u docker --since today | grep -i 'resolver'      # "i/o timeout" to the upstream

Fails via 127.0.0.11 but works direct = the embedded forwarder, not your network.

Fix. docker restart <container> clears it, no daemon restart needed. To make it less likely, set an explicit upstream in /etc/docker/daemon.json ("dns": ["<YOUR_DNS_SERVER_IP>"]) so the forwarder never depends on whatever the host's resolv.conf looked like when the container started. Changing dns needs a systemctl restart docker, which restarts every container, so do it in a quiet window. Apps that cache "indexer failed" state (the *arr apps, Prowlarr) may need a manual re-test afterwards before they try again.

Container is "Up" but nothing answers on its port

Symptom. docker ps shows the container Up, but the published port refuses connections or resets, even from the host itself.

Cause. The ports: mapping points at the wrong container port. "8072:3000" publishes host 8072 to container port 3000. If the app listens on 8080, Docker happily forwards to a closed port. Without a healthcheck, nothing ever turns red.

Diagnose.

docker inspect -f '{{json .Config.ExposedPorts}}' <image>    # what the image says it listens on
docker exec <container> ss -ltn 2>/dev/null || docker exec <container> netstat -ltn
curl -sI http://127.0.0.1:<HOST_PORT>/

Fix. Map to the port the app really listens on (check the image's docs, not an old example), and give the service a healthcheck that hits that port, so a wrong mapping or a dead app shows up as unhealthy instead of Up.

A healthcheck that can never pass (and nobody notices)

Symptom. A container has been unhealthy for days. docker inspect shows a FailingStreak in the thousands. The app itself works fine.

Cause. The healthcheck calls something that isn't in the image (a binary that was renamed between versions, curl in an image that only has wget, a path from another project's example). Docker runs it, it exits non-zero, forever. If the same service is defined slightly differently on each host, one copy can be broken while the others are fine.

Diagnose.

docker inspect --format '{{json .State.Health}}' <container> | jq '{Status, FailingStreak, last: .Log[-1].Output}'
docker exec <container> sh -c 'command -v curl wget /agent'   # what actually exists

Fix. Use the probe the image documents (many ship one, e.g. /agent health, /healthz, /ping). For minimal images without curl or wget, a bash TCP check works:

healthcheck:
  test: ["CMD", "bash", "-c", "</dev/tcp/127.0.0.1/10400"]
  interval: 30s
  timeout: 5s
  retries: 3

Define a shared service (agents, exporters) once and copy it, rather than letting each host grow its own variant. And alert on unhealthy, or the healthcheck is only decoration (see Monitoring).

Passwords on the command line leak into ps and docker inspect

Symptom. Not a failure, a leak: ps -eo args on the host, or docker inspect, shows a password, for example Redis started with --requirepass <password>, or a healthcheck running redis-cli -a <password>.

Cause. Command-line arguments are visible to every user on the host via /proc, and they're stored in the container's config.

Fix. Put the secret in a config file or an environment variable that the app reads itself. For Redis, write the config from the environment with a shell builtin (so no process carries the value) and point the healthcheck at REDISCLI_AUTH:

services:
  redis:
    image: redis:7.4-alpine
    environment:
      REDIS_PASSWORD: ${REDIS_PASSWORD}
    command:
      - sh
      - -c
      - |
        rm -f /tmp/redis.conf
        umask 077
        printf 'requirepass %s\n' "$$REDIS_PASSWORD" > /tmp/redis.conf
        exec docker-entrypoint.sh redis-server /tmp/redis.conf
    healthcheck:
      test: ["CMD-SHELL", "REDISCLI_AUTH=$$REDIS_PASSWORD redis-cli ping | grep -q PONG"]
      interval: 30s
      timeout: 5s
      retries: 3

The rm -f matters: see the restart-loop entry above. Verify with ps -eo args | grep -c requirepass on the host (should be 0).

Note

The value is still in the container's environment, readable with docker inspect. That needs root or Docker-group access, which is already equivalent to root.

A watchdog kills containers after a nightly pause

Symptom. Every morning some containers have been force-restarted, the deployment tool sends a flood of running → paused → running alerts, and a backup that uses docker exec (database dumps) fails if it runs during the same window.

Cause. A nightly job pauses containers (SnapRAID helper scripts commonly docker pause every container while they sync and scrub). Paused containers fail their healthchecks. A watchdog such as autoheal sees unhealthy and restarts them, and docker stop gives up after 10 s on a frozen process and kills it. Pausing databases or the watchdog itself makes it worse, and clients that keep polling a paused service pile up connections that all get answered at once on resume (which can look like a port scan to an IDS, see CrowdSec banning your own server).

Fix.

  • Pause only the services that write to the array, and only for the sync step. A scrub only reads, so leave containers running for it.
  • Keep databases, monitoring agents, VPN containers and the watchdog itself running.
  • For services that have clients connected (media servers), stop them cleanly and start them afterwards, instead of freezing them.
  • Schedule backups that docker exec into containers outside the pause window, and give the alerting side a matching quiet window (see Monitoring).

Databases get killed on every reboot (10-second stop window)

Symptom. After reboots or redeploys, MongoDB-based apps (Omada controller, Komodo, UniFi) or other databases log unclean-shutdown recovery, take ages to start, or occasionally come back with a damaged database.

Cause. docker stop sends SIGTERM and waits 10 seconds before SIGKILL. The daemon uses a similar default at shutdown. An embedded MongoDB flushing to disk can take longer than that.

Fix. Give stateful services more time, per service and daemon-wide:

services:
  omada-controller:
    stop_grace_period: 60s
/etc/docker/daemon.json
{
  "shutdown-timeout": 60
}

shutdown-timeout can be reloaded with systemctl reload docker (SIGHUP). Check it took with docker inspect -f '{{.Config.StopTimeout}}' <container>. A UPS that gives the host time to shut down cleanly matters just as much (see Hosts, reboots & power).

Compose variables that are silently empty

Symptom. A container runs as root although you set user: "${UID}:${GID}", or the LinuxServer image ignores your group, or docker compose up prints The "GUID" variable is not set. Defaulting to a blank string. once and nobody reads it.

Cause. Compose interpolates ${VAR} from the shell environment and the project's .env file. UID is a bash shell variable, not an exported environment variable, so Compose sees nothing. A typo (PGID=${GUID}) does the same. An empty user: or PGID doesn't fail; it falls back to the image default, which is often root.

Diagnose. Render the file exactly as Docker will run it:

docker compose config | grep -nE 'user:|PUID|PGID|TZ'
docker compose config 2>&1 | grep -i 'not set'
docker exec <container> id

Fix. Put the values in .env (PUID=<PUID>, PGID=<PGID>, TZ=<TZ>) and reference those names, or use the image's own non-root user. Use the ${VAR:?message} form for anything that must be set, so a missing value stops the deploy instead of running as root.

Do / avoid: compose conventions

Small inconsistencies are what make a fleet hard to reason about later.

  • Do pin every image to a version tag (or tag@sha256:… when upstream re-pushes tags) and let a bot propose updates. See GitOps & updates.
  • Do use one restart policy everywhere. always and unless-stopped only differ after you docker stop a container and then the daemon restarts, but mixing them makes "why is this up?" harder to answer.
  • Do set container_name consistently (or never). A unique project name per stack also matters: three stacks all called monitoring on different hosts confuse tools that group by project name.
  • Do set TZ the same way on every service, from one variable.
  • Do use absolute bind-mount paths for state (/mnt/appdata/<service>), never ./data. See GitOps & updates.
  • Do create external networks somewhere you can see (a bootstrap script or a README), or a fresh host can't start any stack that uses them.
  • Avoid leaving old copies of stacks in /opt/docker/... after migrating them. A stray docker compose up in the old directory starts a duplicate (or collides on container_name) and may point at old data. Archive, then delete.
  • Avoid mounting /var/run/docker.sock into dashboards, even :ro. Read-only on a socket doesn't limit the API; the container can still start, stop and create containers. Put a socket proxy in front that only allows the read endpoints a widget needs.

Sources & further reading