Skip to content

Secrets in a GitOps repo

The rule is simple: no secret value ever goes into git. Not in a compose file, not in a committed .env, not "just for now". The repo is cloned onto every server (Komodo clones the whole repo, not one directory), and a value in git history stays there even after you delete the line. Once a secret is committed, the only fix is rotating it at the source.

Compose files only ever contain ${NAME}. The values come from Komodo.

How a value reaches a container

flowchart LR
    v["Komodo Variable<br/>NTFY_TOKEN_KUMA (secret)"]
    e["Stack Environment<br/>KUMA_NTFY_TOKEN=[[NTFY_TOKEN_KUMA]]"]
    f[".env next to compose.yaml<br/>(written at deploy time)"]
    c["compose.yaml<br/>${KUMA_NTFY_TOKEN}"]
    v --> e --> f --> c
  1. Variables (Komodo → Settings → Variables): create one per secret value, tick Secret. A secret variable's value is hidden in the UI for non-admins and masked in update logs.
  2. Stack Environment: in the Stack's config, add lines like DB_PASSWORD=[[IMMICH_DB_PASSWORD]]. Komodo replaces [[NAME]] with the variable's value.
  3. On deploy, Periphery writes the environment to .env in the stack's run directory and passes it with --env-file. Compose substitutes ${DB_PASSWORD} from it, or the service reads it with env_file: .env.
compose.yaml (in git)
services:
  app:
    image: ghcr.io/example/app:1.2.3
    environment:
      DB_PASSWORD: ${DB_PASSWORD}
Stack Environment (in Komodo, not in git)
DB_PASSWORD=[[APP_DB_PASSWORD]]
TZ=<TZ>

Non-secret settings (TZ, a version string, a hostname) can go straight into the Stack Environment or the compose file. Anything that would let someone log in, decrypt or impersonate goes into a secret Variable.

Set the variables before the first deploy

If a stack is deployed while a referenced variable doesn't exist, Compose substitutes an empty string and the service starts misconfigured (an empty password, a broken push key). For anything on the alerting path, create the variables first, then push the compose change.

Keep .env out of git

The repo's .gitignore should block the generated files even though nobody should be creating them in the checkout:

.gitignore
.env
.env.*
!.env.example
*.pem
*.key
*.ovpn

Add a secret scanner as a pre-commit hook or run it before pushing:

docker run --rm -v "$(pwd)":/repo zricethezav/gitleaks:latest git /repo -v

Per-publisher tokens (ntfy example)

Give every thing that sends notifications its own token, with write access to only the topics it needs:

Publisher Topics it may write
Komodo alerts, updates
Forgejo updates
backups backups, alerts
host scripts hosts, alerts, backups, updates
Uptime Kuma alerts

Each token is a secret Variable (NTFY_TOKEN_KOMODO, NTFY_TOKEN_FORGEJO, ...). If one leaks, you revoke one token and nothing else breaks, and nobody can read your topics with it. Anonymous access to ntfy is off. See ntfy's access control docs.

The same idea applies to git: Komodo gets a read-only Forgejo token, Renovate gets a write token on the infra repo only, and your own token is separate from both.

Secrets that can't live in Komodo

A few things exist before Komodo can provide anything:

Secret Where it lives
Komodo's own DB password, JWT secret, webhook secret compose.env next to Komodo's compose on the control host, mode 600
Periphery onboarding key / keys /etc/komodo/ on each host, root-only
Host-level tokens (e.g. the notify script's ntfy token) a root-only file such as /etc/ntfy/<publisher>.token, written by the install script from stdin
Backup passphrases your password manager, plus a copy offline

Keep a copy of all of these in a password manager. Rebuilding Komodo from its DB export brings back the Variables, but only if you can still start Komodo.

Don't put tokens in argv

curl -H "Authorization: token abc..." or git -c http.extraHeader=... puts the token in the process list and often in shell history and logs. Read it from a file instead:

curl -H @<(printf 'Authorization: token %s\n' "$(cat ~/.config/forgejo/token)") https://git.<YOUR_DOMAIN>/api/v1/user

If a secret was committed

  1. Rotate it at the source (new password, new token, new keypair). This is the actual fix.
  2. Move the new value into a secret Variable and reference it with [[NAME]].
  3. Remove the literal from the file and push.
  4. Rewriting history (git filter-repo) is optional for a private repo, and it forces every clone (including Komodo's) to be reset. Rotation matters; rewriting usually doesn't.

Sources & further reading