Skip to content

Rebuild from scratch

The order matters because of one chicken-and-egg problem: Komodo deploys from Forgejo, but Forgejo itself is deployed by Komodo. So the first Forgejo and Komodo come up by hand, and everything after that comes from git.

Two starting points:

  • Restore: you still have backups of /mnt/appdata/forgejo, /opt/docker/komodo and the other hosts' appdata. Steps 1–6 bring back the repo, Komodo's database (servers, stacks, procedures, variables) and the rest follows.
  • Fresh: only the repo exists (a workstation clone). Same steps, but you recreate Komodo resources and variables by hand from the tables in these pages.

0. Before you start

  • Every server on Debian 13 with Docker CE and the Compose plugin (Docker's Debian install).
  • DNS names for git.<YOUR_DOMAIN> and komodo.<YOUR_LOCAL_DOMAIN> pointing at the reverse proxy on control-host.
  • Your password manager open: Komodo compose.env values, the secret Variables list, bot tokens, Docker Hub token.
  • The shared Docker network for the proxy: docker network create proxy on control-host.
  • /mnt/appdata mounted on each host.

1. Forgejo (by hand, first time only)

  • Copy hosts/control-host/forgejo/ from your workstation clone to a temporary directory on control-host.
  • Restore: put the backup of /mnt/appdata/forgejo/data back in place (and restore gitea.db from the SQLite dump if the file copy wasn't consistent).
  • docker compose config -q && docker compose up -d, then curl -f http://localhost:3030/api/healthz.
  • Fresh: run the install page, create your admin user, create the empty repo <YOUR_USER>/homelab, push your workstation clone to it: git remote set-url origin https://git.<YOUR_DOMAIN>/<YOUR_USER>/homelab.git && git push -u origin main.
  • Create the komodo (Read) and renovate (Write) bot users and their tokens (Forgejo).

2. Komodo Core (by hand, always)

  • mkdir -p /opt/docker/komodo && cd /opt/docker/komodo, copy hosts/control-host/komodo/compose.yaml there.
  • Create compose.env from the example, chmod 600 compose.env, ln -s compose.env .env. Copy your local root CA to certs/rootCA.crt.
  • Restore: put back mongo/, core/keys/ and core/backups/ from the host backup. If only the DB exports survived, start with an empty Mongo and run km database restore -y.
  • sudo docker compose config -q && sudo docker compose up -d && sudo docker compose ps.
  • Log in, change the initial admin password, remove KOMODO_INIT_ADMIN_PASSWORD from compose.env. Enable 2FA/OIDC. Turn off user registration.

3. Periphery on the other hosts

  • Create an onboarding key in Komodo.
  • On each of server1, server2 (and the workstation if you want it visible): run the install script, systemctl enable --now periphery.
  • Set disable_terminals = true in /etc/komodo/periphery.config.toml unless you need them, restart periphery.
  • Servers page: every host Ok, all on the same version as Core.

4. Providers, repo and variables

  • Settings → Providers: git account (Forgejo host + komodo token), registry account (Docker Hub user + token).
  • Repo resource homelab → <YOUR_USER>/homelab, branch main. Clone it once on each server (Komodo does this on first deploy, or use the Repo's Clone action).
  • Settings → Variables: every secret, marked Secret (Secrets). Do this before creating stacks, so no stack ever deploys with empty values.

5. Stacks

  • One Stack per hosts/<host>/<stack>/ directory, with the settings from Stack resources (linked repo, run directory, config files, environment, registry account, auto pull off, auto update off).
  • Bring Forgejo under Komodo: create its Stack pointing at hosts/control-host/forgejo, then docker compose down the hand-started copy and Deploy the Stack. Its data is at an absolute path, so nothing moves.
  • Deploy stacks one at a time the first time and check each is healthy. Order: proxy/tunnel, ntfy, then the rest.

6. Deploy automation

  • Procedure Redeploy Changed Stacks: one stage, BatchDeployStackIfChanged with pattern *, schedule format Cron 0 */15 * * * ?, webhook enabled, failure alert on (Komodo). Confirm the page shows a "next run" time and no schedule error.
  • Forgejo push webhook to the procedure URL, branch main, secret = KOMODO_WEBHOOK_SECRET (Forgejo). Test Delivery → 200.
  • Alerters to ntfy for host-down, service-down, deploy-failed (ProcedureFailed, ActionFailed) and updates. Test Alerter on each.
  • Forgejo → ntfy hooks for pushes and PRs.
  • Make sure Backup Core Database exists and is scheduled.

7. Renovate

  • Stack renovate with RENOVATE_TOKEN and RENOVATE_HOST_RULES in its environment (Renovate).
  • renovate label exists in the repo. renovate.json passes renovate-config-validator --strict.
  • First run: docker logs -f renovate. The dashboard issue appears; first PRs arrive (2 per hour).

8. Host-level config

  • hosts/_fleet/install-host.sh <target> <host> for every host (Repo layout).
  • Backups: hosts/_fleet/backup/install.sh, then run one backup by hand and check it arrives everywhere.

9. Prove it end to end

  • Change a comment in one stack's compose, push. Within seconds: ntfy "pushed to main", Komodo runs the procedure, only that stack redeploys.
  • Break something on purpose (an image tag that doesn't exist), push: ntfy alerts shows the failed procedure. Revert, push: it recovers.
  • Wait 15 minutes without pushing: the procedure ran on schedule (Komodo Updates list).
  • Merge a Renovate patch PR: the stack redeploys on the new tag.

Sources & further reading