GitOps & updates: lessons learned¶

What goes wrong once compose files live in git and a deployment tool (Komodo, Portainer, Dockge, a webhook script) applies them for you, and once images are pinned and updated by a bot. Each entry is symptom → cause → how to diagnose → fix, plus a do/avoid list at the end. The setup itself is in GitOps.

The fix is in git, but the old config is still running (registry 429)¶

Symptom. You pushed a change, the webhook fired, and days later you notice the container still runs the old config. The deploy tool shows the stack's deployed commit behind the latest one. Nothing alerted.

Cause. The deploy ran docker compose pull first and Docker Hub answered 429 Too Many Requests (anonymous pulls are rate-limited per IP, and a homelab behind one public IP shares that budget). The pull failed, so compose up never ran. The stack then stays undeployed until the next push that touches it, which may be weeks away. Deploys that pull every image on every run, even when only an env var changed, burn through the limit fastest.

Diagnose.

• In the deploy tool, compare the stack's deployed commit/hash with the repo's latest.
• Read the failed run's log for 429, toomanyrequests or rate limit.
• Check what's actually running: docker inspect -f '{{.Config.Image}} {{.Created}}' <container> and the env/labels the change was supposed to set.

Fix.

1. Authenticate pulls on every host (docker login with a Docker Hub access token, or the deploy tool's registry account). Authenticated pulls get a much higher limit.
2. Don't pull on every deploy. With pinned tags, an image only has to be pulled when its tag changed. Turn off "always pull" / auto-pull in the deploy tool, and let Compose use the local image otherwise.
3. Alert on failed deploys, not just on containers going down. Many tools only forward "server unreachable" and "container state changed" by default. Check which alert types your notifier actually receives and send it a test.
4. Retry on a schedule. A periodic "redeploy changed stacks" job (every 15 min or hourly) is idempotent and picks up anything a transient failure left behind.

A deploy fails with "dependency failed to start: container is unhealthy"¶

Symptom. A stack deploy aborts and the dependent service is left in Created. The dependency (a VPN container such as Gluetun, a database) is perfectly healthy a minute later.

Cause. depends_on: condition: service_healthy fails as soon as the dependency is marked unhealthy. The image's built-in healthcheck may have a very short start period (10 s), while the service legitimately needs longer, e.g. a VPN client cycling through a few servers before one connects.

Diagnose.

docker inspect --format '{{json .Config.Healthcheck}}' <dependency>   # StartPeriod, Interval, Retries
docker logs --timestamps <dependency> | head -50                       # how long it really took

Fix. Override the healthcheck with the same probe and a realistic start period. Failures inside start_period don't count, and start_interval checks more often while starting, so a fast start still turns healthy quickly:

services:
  gluetun:
    healthcheck:
      test: ["CMD", "/gluetun-entrypoint", "healthcheck"]
      interval: 5s
      timeout: 5s
      retries: 3
      start_period: 120s
      start_interval: 5s

A service comes back empty after the deploy tool moves its files¶

Symptom. You switch a stack to a new checkout (a shared repo clone, a renamed stack, a different host path) and the service starts as a fresh install: a Git server with no repositories, a dashboard asking for first-time setup, an agent with a new identity.

Cause. The compose file uses a relative bind mount (./data:/data). Relative paths resolve against the directory the deploy tool runs Compose from, which is usually somewhere inside its own clone of your repo. Move the clone and ./data points at a new, empty directory. Worse, the real data lives inside a directory the deploy tool considers disposable: a re-clone, a rename or a "delete stack" takes it with it.

Diagnose. Before migrating anything, look at where the running container's data actually is:

docker inspect -f '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}' <container>
docker inspect -f '{{index .Config.Labels "com.docker.compose.project.working_dir"}}' <container>

Any Source under the deploy tool's stacks/repos directory is at risk.

Fix. Move state to a fixed path outside the clone and reference it absolutely:

1. Take a backup, then stop the service.
2. Copy cold: rsync -aHAX <old>/ /mnt/appdata/<service>/ and confirm with a checksum dry run (rsync -anc --delete <old>/ /mnt/appdata/<service>/ prints nothing).
3. Change the compose file to /mnt/appdata/<service>:/data, deploy, verify.
4. Keep the old directory (renamed) for a couple of weeks of good backups, then delete it.

This matters most for the Git server your GitOps depends on. It can't redeploy itself from a repo it no longer has.

Pinning images accidentally downgrades them¶

Symptom. You replace :latest with "the current version" across your compose files, deploy, and a service comes up older than it was, or refuses to start because its database was already migrated by the newer version.

Cause. The pinned tag was written down from memory, an old example or a staged branch, not from what's actually running. :latest had already moved past it.

Diagnose / fix. Pin to the running build. Compare the running image's digest with the registry's digest for the tag you're about to write:

docker inspect -f '{{join .RepoDigests "\n"}}' <container-image>
docker buildx imagetools inspect <image>:<tag> | grep Digest

If the digests match, the pin changes nothing. If upstream has since re-pushed that tag (common for database and LinuxServer images), pin tag@sha256:<running digest>. Roll out one stack at a time with automatic deploys paused, checking each is healthy before the next.

A floating tag upgrades something you didn't touch¶

Symptom. A redeploy for an unrelated change also upgraded the deploy tool itself, the database under it, or a stateful app. A major tag like :2 or a bare mongo had quietly moved to a new build.

Cause. Any tag except an exact version (and even those, if upstream re-pushes) means "whatever the registry says today". Auto-update features in deploy tools (and Watchtower) apply that blindly, often at night.

Fix.

• Pin everything to exact versions, including the deploy tool's own bootstrap compose, which often lives outside git.
• Turn off auto-update for stateful services (password manager, Git server, home automation, photo library, databases, reverse proxy). Keep update detection on.
• Let Renovate (or a similar bot) open a pull request per update, and merge on your schedule. See the policy below.

Renovate never opens the PRs you expect¶

Symptom. Renovate runs, finds your images, but updates sit on the dashboard forever as "pending", or some LinuxServer images flip between found and no-result.

Cause and fix.

Changing a file next to the compose file doesn't redeploy anything¶

Symptom. You edit a config file that sits beside compose.yaml (an extends: fragment, a CA bundle, an nginx config), push, and nothing happens. Or a stack only works because of a host-local .env that isn't in git, and a redeploy elsewhere picks a different image.

Cause. Deploy tools detect changes to the compose file. Other files are only watched if you list them. A value that only exists in a .env on one host silently falls back to a default (for example an image tag variable falling back to a different release channel).

Fix. Declare every extra file the stack reads (Komodo calls these config_files), keep all variables in git or the deploy tool's environment, and make "no hand edits on hosts" a rule. Scan once for drift: compare each host's on-disk .env keys with what the tool would deploy.

Do / avoid¶

• Do write down a bootstrap order for the control host: base OS → restore files → deploy tool → Git server by hand (it can't deploy itself) → reverse proxy → tunnel → notifications and monitoring → everything else. Keep the deploy tool's own compose in git too, even if it's applied by hand.
• Do keep large binaries (model caches, media, databases) out of the repo. Every host clones the full history; a few hundred MB of blobs gets multiplied by every stack and every host.
• Do prefer one shared checkout per host over one clone per stack, if your tool supports it. One git pull per push instead of twenty.
• Do protect the main branch once a bot opens PRs, and use a token scoped to read the one repo for the deploy tool.
• Avoid secrets in compose files. Everything in the repo ends up on every host that clones it, and in git history forever. Use the tool's secret variables or a gitignored .env. See Security pitfalls.
• Avoid pushing a big change with automatic deploys on. Pause the webhook and schedule, deploy stacks one at a time, then re-enable.
• Avoid deploy tools and Ansible both managing the same file. Check for # Managed by … headers before editing.

Sources & further reading¶

• Docker docs — Docker Hub usage and rate limits
• Docker docs — Compose depends_on and healthcheck
• Gluetun wiki — Healthcheck
• Docker docs — docker buildx imagetools inspect
• Renovate — Docker
• Renovate — minimumReleaseAge
• Renovate — Dependency Dashboard
• Komodo docs