Gotchas¶

Things that went wrong in this setup, or nearly did, and what the fix was.

Relative bind mounts point into Komodo's clone¶

Symptom: after a redeploy through Komodo, a service comes up empty: a fresh install page, no users, no data.

Cause: Komodo runs docker compose from its own checkout (/etc/komodo/repos/<repo>/hosts/<host>/<stack> or /etc/komodo/stacks/<stack>/...). ./data:/data resolves relative to that directory, not wherever the container used to run from. The old data is still where it was; the container just isn't looking there.

This happened to Forgejo, the worst possible service for it: it is the git source of every deploy, so a reclone of its stack could take the deploy source down with it.

Fix: absolute paths for anything stateful, /mnt/appdata/<service>/.... Before moving a stack under Komodo (or to a different clone), compare docker inspect <container> --format '{{json .Mounts}}' with the compose file.

Registry rate limits fail deploys¶

Symptom: a push changed a stack, but it is still on the old commit. The procedure log shows 429 Too Many Requests from Docker Hub, and compose never got to up.

Fixes (all three):

Retry schedule: run Redeploy Changed Stacks every 15 minutes, not only on webhook. A failed stack still differs from git, so the next run deploys it.
Registry account on every Stack: authenticated Docker Hub pulls don't share the anonymous per-IP limit.
auto_pull = false: a deploy only pulls when a tag in the file changed. Before this, a config-only commit pulled every image of the stack and could fail on a registry blip that had nothing to do with the change.

Failed deploys nobody heard about¶

The default alerters only forwarded server and container state changes. A failed procedure is a different alert type (ProcedureFailed), so broken deploys sat there silently until the next push. Add ProcedureFailed and ActionFailed to an alerter on your loudest ntfy topic and test it.

A schedule that silently stopped¶

Komodo schedules can be written in English ("Every 15 minutes"). Komodo turns that into cron internally, and after an upgrade the generated expression can be rejected (single number steps like "0/15" are not allowed). The procedure still exists and the webhook still works, so nothing looks broken, but the retry loop is gone. Use Cron format (0 */15 * * * ?, six fields with seconds first) and check the procedure page shows a next run after every Komodo upgrade.

Dependency health vs. slow starters¶

depends_on: condition: service_healthy fails the whole compose up as soon as the dependency is marked unhealthy. A VPN container that needs 40 s to find a working server, with an image healthcheck that allows 10 s, fails the deploy and leaves the dependent containers in Created. Give slow starters an explicit healthcheck with a long start_period (failures inside it don't count) and a short start_interval.

Non-compose files don't trigger deploys¶

Editing settings.yaml, a Dockerfile or a CA bundle next to a compose file is not a change as far as DeployStackIfChanged is concerned, unless the file is listed in the Stack's config files with Requires: Redeploy. List every file a stack reads.

Host-local `.env` drift¶

A stack worked only because someone once left a .env on the host inside an old clone. The repo and Komodo didn't know about it; a re-clone would have changed which image ran. Rule: nothing is edited on hosts. Every value lives in git or in the Stack Environment. Compare on-disk .env keys with the Stack Environment when you take over a host.

Linked repo vs. per-stack clones¶

Per-stack clones mean one full clone and one git pull per stack per push, and a copy of the whole repo under every stack directory. Switching to one linked_repo per host fixes that, but a stack with relative mounts switched without moving its data first starts empty (see the first gotcha). Move data, then switch. The old clone directories under /etc/komodo/stacks/ are left behind; delete them once you're sure.

Forgejo webhooks fail with x509 errors¶

Forgejo is a Go program. Go reads SSL_CERT_FILE, not the store that update-ca-certificates manages. If the CA bundle mount is lost (for example a container recreated from a compose that never had it), webhooks to internal HTTPS names fail with x509: certificate signed by unknown authority while pushes keep working. The only trace is the webhook's Recent Deliveries. Keep the bundle mount in the compose in git.

Renovate can't clone¶

gitUrl = endpoint. Forgejo advertises clone URLs built from ROOT_URL; if that host isn't reachable from the Renovate container, every run fails at clone. See Renovate.

Renovate never merges anything¶

minimumReleaseAge set while the registries return no release timestamps: every update is held forever. See Renovate.
No CI checks in the repo and ignoreTests not set: Renovate waits for checks that never arrive.
Outside the automergeSchedule window: it merges on the next run inside it.
Rate limits: config:recommended opens 2 PRs per hour. The dashboard issue lists what's waiting.

Upgrading Komodo halfway¶

Core on the new version, a systemd Periphery on the old one: that server goes not-OK or behaves oddly. Core and every Periphery move together, in one window. The Renovate rule groups core and periphery into one manual PR as a reminder, but the systemd binaries still have to be upgraded on each host.

Validate before you push (and before you apply by hand)¶

docker compose config -q            # in the stack directory: syntax + interpolation
renovate-config-validator --strict  # after editing renovate.json (run from the Renovate image)

docker compose config -q is mandatory before applying Komodo's own compose by hand: there is no webhook retry for that one, and a typo there takes down the control plane.

Don't let the automation deploy the thing you're standing on¶

If you access everything remotely through a tunnel and reverse proxy, those images are manual-review in Renovate, and changes to them happen when you're at home. A bad automerge on the access path means no way in to fix it.