Autoheal¶

docker-autoheal

Docker's restart: policy only acts when a container exits. A container that is still running but stuck (failing its healthcheck) just sits there marked unhealthy. Autoheal fills that gap: it polls the Docker API and restarts any container that carries a chosen label and has turned unhealthy.

In this stack it watches the two containers that hang rather than crash: qBittorrent and qbit_manage. Both have a healthcheck: and the autoheal=true label. Everything else relies on its restart policy.

Container¶

From the stack compose:

### --- Autoheal: restarts containers labelled autoheal=true when they turn unhealthy --- ###
  autoheal:
    image: willfarrell/autoheal:latest@sha256:5faf6731257cd57316c2863e40b56bbd148760e867ac689311c08924a266312c
    container_name: autoheal
    restart: always
    environment:
      - AUTOHEAL_CONTAINER_LABEL=autoheal
      - AUTOHEAL_INTERVAL=30
      - AUTOHEAL_START_PERIOD=60
    volumes:
      # Docker socket access is root-equivalent on the host. Mount it only into containers you trust.
      - /var/run/docker.sock:/var/run/docker.sock

And on each watched service:

    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8080 >/dev/null 2>&1 || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    labels:
      - autoheal=true

Why the image is pinned by digest

Autoheal only publishes a latest tag. To keep "every image pinned" true, it's pinned as latest@sha256:<digest>, and Renovate opens a pull request when the digest changes.

Key settings¶

Variable	Default	Here	Meaning
`AUTOHEAL_CONTAINER_LABEL`	`autoheal`	`autoheal`	Only containers with this label set to `true` are watched. `all` watches every container.
`AUTOHEAL_INTERVAL`	`5`	`30`	Seconds between checks. 30 s matches the healthcheck interval; checking faster gains nothing.
`AUTOHEAL_START_PERIOD`	`0`	`60`	Seconds autoheal waits after its own start before acting, so a host boot doesn't trigger a restart storm while everything is still coming up.
`AUTOHEAL_DEFAULT_STOP_TIMEOUT`	`10`	default	Seconds Docker waits for a clean stop before killing the container. Per container: label `autoheal.stop.timeout=30`.
`AUTOHEAL_ONLY_MONITOR_RUNNING`	`false`	default	`true` makes autoheal ignore paused containers. See the gotcha below.
`WEBHOOK_URL`	empty	not set	Posts a message on every restart (or failed restart), e.g. to ntfy.

Gotchas¶

Nightly pauses look like failures¶

If something pauses containers for a long time, autoheal sees it as a failure. A SnapRAID helper script that docker pauses the stack during the nightly sync is the usual culprit. By default autoheal also watches paused containers, and a long pause leaves the healthcheck failing. When the stack is unpaused, autoheal finds qBittorrent and qbit_manage unhealthy and restarts them. They don't stop within 10 s, so Docker kills them (failed to exit within 10s ... using the force in the dockerd log). That's a hard kill every night, which puts qBittorrent's resume data at risk.

Ways to avoid it, from least to most change:

Only pause the containers that write to the array during the sync, and don't pause autoheal or the things it watches unless they write there.
Set AUTOHEAL_ONLY_MONITOR_RUNNING=true so paused containers are left alone.
Give qBittorrent more time to shut down cleanly: label autoheal.stop.timeout=60 (and stop_grace_period: 60s for normal stops).

It needs the Docker socket¶

Autoheal works by calling the Docker API, so it mounts /var/run/docker.sock read-write. That's root on the host. Use the upstream image, pin it, and keep it on a host you control. If you already run a Docker socket proxy, give autoheal access through it with only the container list/inspect/restart endpoints allowed.

It only sees containers with a healthcheck¶

No healthcheck: (in the image or the compose) means the container never becomes unhealthy, and autoheal never touches it. Check with docker inspect --format '{{json .State.Health}}' <container>.

A restart loop hides the real problem¶

If a container keeps going unhealthy, autoheal keeps restarting it. Check docker logs autoheal now and then. A container that restarts every few minutes needs fixing, not healing.