Provisioning a New Host with Ansible¶

A complete walkthrough of bringing a bare machine into a managed homelab using Ansible. This guide uses server3 as the example, but the process applies to any new Debian host.

By the end, you will understand what Ansible does, how it works, how to provision a host from nothing, and how to use Ansible day-to-day for routine management.

Official Ansible documentation: https://docs.ansible.com/ansible/latest/getting_started/index.html

What is Ansible¶

Ansible is a tool that configures remote machines over SSH. You describe what you want the machine to look like (packages installed, files in place, services running) and Ansible makes it happen. You do not install Ansible on the remote machines. It runs from a single control node and reaches out to each managed host via SSH.

The core idea is declarative configuration: you describe the desired state, not a sequence of steps. If a package is already installed, Ansible skips it. If a config file already matches the template, Ansible leaves it alone. This property is called idempotency, and it means you can safely run the same playbook over and over. The first run changes things. Subsequent runs confirm nothing drifted.

Ref: https://docs.ansible.com/ansible/latest/getting_started/get_started_ansible.html

Core Concepts¶

Before touching any files, here is the vocabulary. Every term maps to a real file or action you will encounter in this guide.

Control node is the machine that runs Ansible. Ansible is only installed here.

Managed node is a machine Ansible configures. server3, server1, server2, and nas are all managed nodes. They do not need Ansible installed. They only need SSH access and sudo.

Inventory is a file that lists your managed nodes and organizes them into groups. When you run a playbook, Ansible reads the inventory to know which machines to target.

Playbook is a YAML file that says "run these roles against these hosts." It is the entry point for any Ansible run. You execute playbooks from the command line.

Role is a reusable bundle of work. The base role installs packages and hardens SSH. The docker role installs Docker. Each role has its own folder with tasks, templates, handlers, and files.

Task is a single action inside a role. "Install curl" is a task. "Copy the SSH config template" is a task. Tasks run in order, top to bottom.

Handler is a special task that only runs when triggered. For example, the SSH config template task notifies a handler called restart sshd. If the template did not change, the handler never fires. This prevents unnecessary service restarts.

Template is a file with placeholder variables (Jinja2 syntax, using {{ variable_name }}). Ansible fills in the variables and copies the result to the managed node. Your sshd_config.j2 is a template.

Module is a built-in Ansible function. ansible.builtin.apt installs packages. ansible.builtin.template copies templates. ansible.builtin.service manages services. You do not write these yourself; you call them in your tasks.

Facts are information Ansible automatically collects about each managed node when a playbook runs. OS version, IP addresses, CPU count, disk layout, hostname. You can reference facts in templates and tasks. For example, {{ ansible_facts['distribution_release'] }} resolves to trixie on Debian 13.

Vault is Ansible's built-in encryption for secrets. Passwords and tokens go in a vault-encrypted YAML file so they never sit in plain text on disk.

Tags are labels you attach to roles or tasks so you can run subsets of a playbook. --tags docker runs only the docker role. --skip-tags borgmatic runs everything except borgmatic.

Idempotent means running the same playbook twice produces the same result. Ansible only changes things that need changing.

How the pieces call each other:

You run a playbook
  - playbook targets hosts and calls roles
    - role runs tasks in order (tasks/main.yml)
      - task uses a template (.j2) to place a config file
        - template pulls values from group_vars/host_vars
          - task notifies a handler if something changed
            - handler runs once at the end (e.g., restart service)

On disk:

playbooks/base.yml              # you run this
  calls -> roles/docker/
             tasks/main.yml      # what to do, in order
             templates/daemon.json.j2  # config files with variables
             handlers/main.yml   # triggered reactions

Playbook calls role. Role runs tasks. Tasks use templates. Tasks notify handlers.

Ref: https://docs.ansible.com/ansible/latest/getting_started/basic_concepts.html

How It All Fits Together¶

Here is the flow when you run a playbook:

You run: ansible-playbook playbooks/base.yml --limit server3.example.com

Ansible reads ansible.cfg
  -> finds inventory file (inventory/hosts.ini)
  -> finds server3.example.com in the [standby] group
  -> [standby] is a child of [homelab], which matches hosts: homelab in the playbook
  -> loads variables from group_vars/all.yml (applies to all hosts)
  -> loads variables from group_vars/vault.yml (secrets, loaded via vars_files)
  -> loads variables from host_vars/server3.example.com.yml (if it exists)
  -> connects to server3.example.com via SSH using ~/.ssh/ansible key
  -> gathers facts about server3 (OS, hostname, IPs, etc.)
  -> runs each role in order: base, docker, tailscale, mounts, borgmatic
  -> within each role, runs tasks top to bottom
  -> if a task changes something, it may notify a handler
  -> handlers fire at the end of all tasks
  -> reports summary: ok=X, changed=Y, failed=Z

The --limit flag restricts which hosts the playbook targets. Without it, the playbook runs against every host that matches its hosts: directive.

Ref: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_intro.html

The Existing Setup¶

Ansible is already installed and configured on your control node. The project lives at ~/ansible/ and currently manages server1. Here is the structure:

~/ansible/
  ansible.cfg                    # Global settings (SSH key, inventory path, sudo config)
  .vault_pass                    # Vault decryption password (chmod 600, never commit)
  inventory/
    hosts.ini                    # Host inventory with groups
    group_vars/
      all.yml                    # Variables shared across all hosts
      vault.yml                  # Encrypted secrets (samba password, tokens)
    host_vars/                   # Per-host variable overrides (empty for now)
  playbooks/
    base.yml                     # Provisions base system (packages, SSH, Docker, backups)
    stacks.yml                   # Deploys Docker Compose stacks per host role
    update.yml                   # Pulls and restarts containers
  roles/
    base/                        # Packages, SSH hardening, timezone, directories
    docker/                      # Docker CE installation and daemon config
    tailscale/                   # Tailscale install (left stopped)
    mounts/                      # CIFS mount to the NAS
    borgmatic/                   # Borg backup config and cron
    codeserver/                  # Native code-server install (on infra hosts)
    stacks_infra/                # Docker stacks for server1 (NPM, monitoring, etc.)
    stacks_apps/                 # Docker stacks for server2 (empty, future use)
    update_containers/           # Pull latest images and recreate

The key files that control behavior:

ansible.cfg tells Ansible where to find the inventory, which SSH key to use, which user to connect as, and that it should use sudo without a password prompt.

inventory/hosts.ini lists all managed hosts and their groups. Groups determine which playbooks and variables apply to which machines.

inventory/group_vars/all.yml defines shared variables like package lists, directory paths, and DNS settings. Every host gets these.

inventory/group_vars/vault.yml holds encrypted secrets (samba password, service tokens). Decrypted automatically at runtime using .vault_pass.

Prepare the Network¶

Before the machine exists on the network, it needs two things in OPNsense: a DHCP static mapping so it always gets the same IP, and a dnsmasq host entry so server3.example.com resolves across the lab.

DHCP Static Mapping¶

In the OPNsense web UI:

1. Go to Services > ISC DHCPv4 > [your server VLAN interface]
2. Scroll to DHCP Static Mappings for this Interface
3. Click Add
4. Set the MAC address of server3's NIC (find it on the machine's BIOS screen or from a live boot)
5. Set the IP address (pick the next available in your subnet, e.g. 192.0.2.x)
6. Set the hostname to server3
7. Save and Apply

dnsmasq Host Entry¶

1. Go to Services > Dnsmasq DNS > Host Overrides
2. Add an entry: Host = server3, Domain = example.com, IP = the IP you just assigned
3. Save and Apply

Why This Matters¶

Ansible connects to hosts by name. The inventory file says server3.example.com, so DNS has to resolve that to an IP. The static DHCP mapping ensures the IP never changes. Without these two pieces, Ansible cannot reach the machine.

Install Debian 13 (Trixie)¶

Install Debian 13 Trixie on the NVMe drive. During the installer:

• Set hostname to server3
• Create user <YOUR_USER>
• Use the NVMe as the OS disk
• Choose minimal install (no desktop environment, no extra packages)

Since this is a test bed, skip setting up the SATA SSD for now. Ansible will create /mnt/appdata as a regular directory on the NVMe. If you later want a dedicated appdata disk, you can format the SSD, add an fstab entry, and re-run the playbook.

Ref: https://www.debian.org/releases/trixie/

Pre-Ansible Setup on server3¶

Ansible needs two things to work: an SSH key that lets it log in without a password, and sudo that does not prompt for a password. Fresh Debian minimal installs do not include sudo, so this step is manual.

Install sudo and configure passwordless sudo¶

SSH into server3 from your control node using whatever method works (password auth is still enabled at this point):

ssh <YOUR_USER>@server3.example.com

If DNS is not propagated yet, use the IP directly:

ssh <YOUR_USER>@192.0.2.12

Once logged in, switch to root (the root password was set during the Debian install):

su -

Install sudo:

apt update && apt install -y sudo

Configure passwordless sudo for <YOUR_USER>:

echo '<YOUR_USER> ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/<YOUR_USER>
chmod 440 /etc/sudoers.d/<YOUR_USER>

Exit back to your normal user, then exit SSH entirely:

exit
exit

Why passwordless sudo¶

Ansible runs most tasks as root (installing packages, writing config files to /etc, managing services). It connects as <YOUR_USER> then uses sudo to elevate. If sudo prompted for a password, Ansible would hang waiting for input. The become_ask_pass = False setting in ansible.cfg relies on this sudoers config being in place.

Ref: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_privilege_escalation.html

Copy the Ansible SSH key¶

Back on your control node, copy the dedicated Ansible key to server3:

ssh-copy-id -i ~/.ssh/ansible.pub <YOUR_USER>@server3.example.com

This adds the public key to ~<YOUR_USER>/.ssh/authorized_keys on server3. From this point forward, Ansible can log in without a password using the private key at ~/.ssh/ansible.

Test it:

ssh -i ~/.ssh/ansible <YOUR_USER>@server3.example.com "hostname"

If that prints server3 without asking for a password, you are good. If it prompts for a password, something went wrong with the key copy.

Add server3 to the Ansible Inventory¶

Now we tell Ansible that server3 exists. This happens in one file on your control node.

Edit inventory/hosts.ini¶

Open ~/ansible/inventory/hosts.ini on your control node. It currently looks like this:

[infra]
server1.example.com

[apps]
server2.example.com

[standby]
server3.example.com

[nas]
nas.example.com

[homelab:children]
infra
apps
standby

server3 is already listed in the [standby] group. The [homelab:children] section includes standby, which means server3 is part of the homelab group. Since playbooks/base.yml targets hosts: homelab, server3 is already included. No changes needed here.

If server3 were not already in the inventory, you would add it under the appropriate group. The group name determines which group_vars apply and which playbooks target it.

How groups and variable loading work¶

For this repo's inventory layout, think of variable loading like this (later entries override earlier ones):

group_vars/all.yml          <- applies to every host, loaded first
group_vars/<group>.yml      <- applies to hosts in that group
host_vars/<hostname>.yml    <- applies to one specific host, loaded last (wins)

The full Ansible variable precedence rules are more extensive than this (there are 22 levels). For this repo, the simplified model above covers every case you will encounter.

Ref: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html#understanding-variable-precedence

For server3: - group_vars/all.yml loads because all.yml applies to everything - group_vars/standby.yml would load if it existed (it does not currently) - host_vars/server3.example.com.yml would load if it existed (it does not currently)

Since server3 is a test bed with no special configuration, the defaults in all.yml are sufficient. You do not need to create any host_vars or group_vars files for it.

Ref: https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html

Verify Ansible can reach server3¶

From ~/ansible/ on your control node:

ansible server3.example.com -m ping

Expected output:

server3.example.com | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

This ping is not an ICMP ping. It is an Ansible module that connects via SSH, runs a tiny Python script on the remote host, and returns "pong" if everything works. It validates: SSH connectivity, key authentication, Python availability on the remote host, and sudo access.

Dry Run¶

Before making any changes to server3, do a dry run. This shows you exactly what Ansible would do without actually doing it.

cd ~/ansible
ansible-playbook playbooks/base.yml --check --diff --limit server3.example.com

The --check flag is "check mode." Ansible connects to server3, evaluates every task, and reports what would change. Nothing is actually modified on the remote host.

The --diff flag shows a line-by-line comparison of what changed in templated files. Combined with --check, you see exactly what Ansible will modify without actually modifying it.

The --limit flag restricts execution to server3 only. Without it, the playbook would target every host in the homelab group (server1, server2, and server3).

You will see output like:

TASK [base : Set timezone] ****************************************************
changed: [server3.example.com]

TASK [base : Install base packages] *******************************************
changed: [server3.example.com]

TASK [docker : Install Docker CE] *********************************************
changed: [server3.example.com]

Everything shows changed because server3 is a fresh install with none of this configured yet. That is expected.

Ref: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_checkmode.html

Run the Playbook¶

This is where Ansible actually configures server3.

ansible-playbook playbooks/base.yml --limit server3.example.com

Ansible will connect to server3 and execute each role in the order defined in the playbook. Here is what each role does and why.

Role: base¶

What it does: Sets the timezone, generates the locale, updates the apt cache, installs all base packages (curl, git, htop, jq, tmux, rsync, etc.), creates the standard directory structure (/opt/docker, /mnt/appdata, /mnt/backups), deploys a hardened SSH config, and ensures passwordless sudo is configured.

Why: Every host in the lab needs the same foundation. Instead of manually running apt install on each machine and hoping you remember the same list, the role guarantees consistency. The SSH hardening (key-only auth, no root login) is enforced on every Ansible run, so even if someone manually edits sshd_config, the next run reverts it.

How it works internally: The ansible.builtin.apt module checks each package. If already installed, it skips it. If not, it installs it. The ansible.builtin.template module compares the rendered template against the file on disk. If they match, no change. If they differ, it writes the new version and notifies the restart sshd handler.

Role: docker¶

What it does: Removes any old Docker packages (distro-provided ones that conflict), adds Docker's official GPG key and apt repository, installs Docker CE, the CLI, containerd, and the compose plugin. Adds <YOUR_USER> to the docker group so you can run docker commands without sudo. Configures log rotation (10MB max, 3 files) via /etc/docker/daemon.json.

Why: Docker from the official repo is more current than what Debian ships. The daemon.json log config prevents container logs from filling up the disk, which is a common problem on long-running hosts.

How the creates argument works: The GPG key task uses args: creates: /etc/apt/keyrings/docker.gpg. This tells Ansible "if this file already exists, skip the task." It is a simple idempotency guard for shell commands that would otherwise always show as changed.

Ref: https://docs.docker.com/engine/install/debian/

Role: tailscale¶

What it does: Adds the Tailscale apt repository and installs tailscale. Then explicitly stops and disables the service.

Why: Tailscale is an emergency backdoor. If OPNsense goes down and you lose access to the VLANs, you can SSH into a host via Tailscale's overlay network. It is installed but not running so it does not consume resources or create unexpected network paths during normal operation. To use it: sudo tailscale up.

Role: mounts¶

What it does: Creates /etc/samba/ with restricted permissions, writes a credentials file containing the NAS samba username and password (pulled from the vault), and mounts //nas.example.com/backups at /mnt/backups via CIFS.

Why: Borgmatic needs somewhere to write backup archives. The NAS's samba share is the target. The credentials file has mode 0600 so the password is not exposed. The ansible.posix.mount module both mounts the filesystem and adds it to /etc/fstab so it persists across reboots. This module is from the ansible.posix collection (not ansible-core), included with the full ansible package.

Ref: https://docs.ansible.com/ansible/latest/collections/ansible/posix/mount_module.html

Role: borgmatic¶

What it does: Installs uv (a fast Python package manager), then uses uv tool install to install borgmatic as recommended by upstream. Creates the config directory, templates the borgmatic config (backup sources, exclude patterns, retention policy), sets up a cron job to run at 03:00 daily, checks if a Borg repo exists on the NAS, and initializes one if not.

Why: Automated daily backups of /opt/docker (compose files), /mnt/appdata (container data), and /etc (system config). The retention policy keeps 7 daily, 4 weekly, and 6 monthly archives. The --encryption=repokey flag means the repo is encrypted, deduplicated and checksummed - the passphrase is set via BORG_PASSPHRASE: "{{ vault_borg_passphrase }}" which is a vault variable.

Note on install method: Borgmatic's upstream documentation now recommends uv tool install borgmatic over pip. The uv tool isolates borgmatic into its own virtual environment automatically, avoiding conflicts with system Python packages. This replaces the older pip install --break-system-packages borgmatic approach.

Ref: https://torsion.org/borgmatic/how-to/install-borgmatic/

Verify¶

After the playbook completes, verify everything is in place. You can do this with Ansible ad-hoc commands from your control node, which is a good skill to practice.

Check base packages¶

ansible server3.example.com -a "which htop jq tmux git"

The -a flag runs a raw command. This is the command module (Ansible's default). It runs the command on the remote host and returns the output.

Check Docker¶

ansible server3.example.com -a "docker --version"
ansible server3.example.com -a "docker compose version"

Check Docker is running¶

ansible server3.example.com -m service_facts

This returns JSON with every service's state. To check just Docker:

ansible server3.example.com -a "systemctl is-active docker"

Check the NAS mount¶

ansible server3.example.com -a "df -h /mnt/backups"
ansible server3.example.com -a "ls /mnt/backups"

Check borgmatic¶

ansible server3.example.com --become -a "/root/.local/bin/borgmatic --version"
ansible server3.example.com -a "cat /etc/borgmatic/config.yaml"
ansible server3.example.com -a "cat /etc/cron.d/borgmatic"

Check Tailscale is installed but stopped¶

ansible server3.example.com -a "tailscale version"
ansible server3.example.com -a "systemctl is-active tailscaled"

The first should print a version number. The second should print inactive.

Check SSH hardening¶

ansible server3.example.com -a "grep PasswordAuthentication /etc/ssh/sshd_config"
ansible server3.example.com -a "grep PermitRootLogin /etc/ssh/sshd_config"

Expected: PasswordAuthentication no and PermitRootLogin no.

Check directory structure¶

ansible server3.example.com -a "ls -la /opt/docker /mnt/appdata /mnt/backups"

Verify idempotency¶

Run the playbook again:

ansible-playbook playbooks/base.yml --limit server3.example.com

This time, the summary should show changed=0 (or close to it). A few tasks may always show changed (like shell commands without creates), but the majority should be ok. This confirms the playbook is idempotent: running it twice does the same thing as running it once.

Test Deploying a Stack¶

Since server3 is a test bed, let's deploy a simple container to practice the compose workflow. This mimics what stacks_infra does on server1 but in a manual, one-off way so you can see the mechanics.

The pattern for Ansible-managed compose stacks¶

Every stack follows the same four steps:

1. Create the compose directory at /opt/docker/<stack-name>/
2. Create the appdata directories at /mnt/appdata/<service>/
3. Template the compose file from a .j2 template to the compose directory
4. Run docker compose up -d if the compose file changed

The compose templates use Jinja2 variables like {{ appdata_dir }} and {{ docker_dns }} so the same template works on any host. Ansible fills in the values from your group_vars/all.yml.

Create a test stack template¶

On your control node, create a simple compose template. This deploys IT-Tools, a lightweight utility container that is harmless and easy to verify:

Create the file at ~/ansible/roles/stacks_infra/templates/test-compose.yml.j2:

# Managed by Ansible -- test stack for server3
services:
  it-tools:
    image: corentinth/it-tools:latest
    container_name: it-tools
    restart: unless-stopped
    ports:
      - "8071:80"
    dns:
      - {{ docker_dns }}

Add tasks to deploy it¶

You could add this to the existing stacks_infra role, but since it is a test, let's run it as an ad-hoc set of commands to understand what each step does:

# Step 1: Create the compose directory on server3
ansible server3.example.com -m file -a "path=/opt/docker/test state=directory owner=<YOUR_USER> group=<YOUR_USER>"

# Step 2: Copy the rendered template to server3
ansible server3.example.com -m template -a "src=roles/stacks_infra/templates/test-compose.yml.j2 dest=/opt/docker/test/docker-compose.yml owner=<YOUR_USER> group=<YOUR_USER>"

# Step 3: Start the stack
ansible server3.example.com -a "docker compose -f /opt/docker/test/docker-compose.yml up -d"

Each of these is an ad-hoc command using a different module:

• file creates directories (like mkdir -p)
• template renders a Jinja2 template and copies it to the remote host
• The default command module runs docker compose

Verify it is running¶

ansible server3.example.com -a "docker ps"

You should see it-tools running and mapped to port 8071. You can also visit http://server3.example.com:8071 in your browser.

Clean up when done¶

ansible server3.example.com -a "docker compose -f /opt/docker/test/docker-compose.yml down"
ansible server3.example.com -m file -a "path=/opt/docker/test state=absent"

The state=absent on a file module deletes the directory.

Day-to-Day Ansible Usage¶

Now that server3 is provisioned, here is how you use Ansible for routine management.

Checking on your hosts¶

# Are all hosts reachable?
ansible homelab -m ping

# What is the uptime on all hosts?
ansible homelab -a "uptime"

# How much disk space is left on all hosts?
ansible homelab -a "df -h /"

# What containers are running on server1?
ansible server1.example.com -a "docker ps"

# Check memory usage on server3
ansible server3.example.com -a "free -h"

These are ad-hoc commands. They run a single task against one or more hosts without needing a playbook. The -m flag specifies a module (defaults to command if omitted). The -a flag passes arguments.

Ref: https://docs.ansible.com/ansible/latest/command_guide/intro_adhoc.html

Gathering host information (facts)¶

Ansible collects a large set of system facts every time a playbook runs. You can query them directly:

# All facts for a host (long output)
ansible server3.example.com -m setup

# Filter to specific facts
ansible server3.example.com -m setup -a "filter=ansible_distribution*"
ansible server3.example.com -m setup -a "filter=ansible_memtotal_mb"
ansible server3.example.com -m setup -a "filter=ansible_default_ipv4"
ansible server3.example.com -m setup -a "filter=ansible_hostname"

Useful facts you can reference in templates and tasks:

The setup filter still uses the ansible_ prefix; in tasks and templates use the ansible_facts['…'] form (see Facts).

Ref: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_vars_facts.html

Editing a compose file and pushing it¶

This is the main workflow once services are deployed. The typical cycle:

1. Edit the template on your control node:

cd ~/ansible
micro roles/stacks_infra/templates/npm-compose.yml.j2

2. See what would change (dry run):

ansible-playbook playbooks/stacks.yml --check --diff --limit server1.example.com

The --diff flag shows a line-by-line comparison of what changed in templated files. Combined with --check, you see exactly what Ansible will modify without actually modifying it.

3. Push the change:

ansible-playbook playbooks/stacks.yml --limit server1.example.com

If the compose file changed, the stacks_infra role's task for that stack detects it (via register: npm_compose) and runs docker compose up -d to apply the change. If the file did not change, Docker is not touched.

4. Verify:

ansible server1.example.com -a "docker ps"

Pulling updated container images¶

To update containers across hosts, use the update playbook:

# Update all containers on server1
ansible-playbook playbooks/update.yml --limit server1.example.com

# Update on all hosts
ansible-playbook playbooks/update.yml

Or do it ad-hoc for a specific stack:

# Pull latest images for the monitoring stack
ansible server1.example.com -a "docker compose -f /opt/docker/monitoring/docker-compose.yml pull"

# Recreate containers with the new images
ansible server1.example.com -a "docker compose -f /opt/docker/monitoring/docker-compose.yml up -d"

Running a manual backup¶

# Trigger borgmatic on a specific host
ansible server3.example.com -a "borgmatic --verbosity 1 --list --stats" --become

# Trigger on all hosts
ansible homelab -a "borgmatic --verbosity 1 --list --stats" --become

The --become flag ensures the command runs as root, which borgmatic needs to read /etc and other protected directories.

Re-converging after a drift¶

If someone manually changes something on a host (edits sshd_config by hand, removes a package, deletes a cron job), re-running the playbook fixes it:

ansible-playbook playbooks/base.yml --limit server3.example.com

Ansible compares the current state to the declared state and corrects any differences. This is the power of idempotency. You do not need to figure out what changed. Just re-run and Ansible puts everything back.

Understanding the Playbook File¶

Here is playbooks/base.yml with annotations:

---
# The triple dash marks the start of a YAML document

- name: Base system configuration        # Human-readable name, shows in output
  hosts: homelab                          # Target group from inventory
  become: yes                             # Use sudo for all tasks
  vars_files:                             # Additional variable files to load
    - ../inventory/group_vars/vault.yml   # Load encrypted secrets
  roles:                                  # Roles to apply, in order
    - { role: base, tags: ['base'] }
    - { role: docker, tags: ['docker'] }
    - { role: tailscale, tags: ['tailscale'] }
    - { role: mounts, tags: ['mounts'] }
    - { role: borgmatic, tags: ['borgmatic'] }

The vars_files line is included by design. In this repo, vault.yml is loaded explicitly so secrets are clearly separated from regular variables. Ansible auto-loads files in group_vars/ that match a group or host name, but vault.yml does not match any group, so it requires explicit loading.

Ref: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_reuse_roles.html

Using tags¶

Tags let you run specific parts of the playbook:

# Only run the docker role
ansible-playbook playbooks/base.yml --tags docker --limit server3.example.com

# Run everything except borgmatic
ansible-playbook playbooks/base.yml --skip-tags borgmatic --limit server3.example.com

# Run base and docker only
ansible-playbook playbooks/base.yml --tags base,docker --limit server3.example.com

This is useful when you are iterating on a single role and do not want to wait for all five roles to run every time.

Ref: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_tags.html

Understanding a Role in Detail¶

Let's walk through the base role task by task to see how roles work internally. This is roles/base/tasks/main.yml:

- name: Set timezone
  community.general.timezone:
    name: "{{ timezone }}"

Module: community.general.timezone. Sets the system timezone. Like locale_gen, this module is from the community.general collection, not ansible-core. It is included with the full ansible package. Variable: {{ timezone }} comes from group_vars/all.yml where it is set to <TZ>. Idempotency: If the timezone is already <TZ>, this task reports ok and changes nothing.

Ref: https://docs.ansible.com/ansible/latest/collections/community/general/timezone_module.html

- name: Set locale
  community.general.locale_gen:
    name: "{{ locale }}"
    state: present

Module: community.general.locale_gen. Generates the specified locale so programs can use it. Note: the role file may reference this as ansible.builtin.locale_gen, which works because ansible-core redirects that old name to the collection. The technically correct fully qualified collection name (FQCN) is community.general.locale_gen since this module is part of the community.general collection, not ansible-core.

Ref: https://docs.ansible.com/ansible/latest/collections/community/general/locale_gen_module.html

- name: Update apt cache
  ansible.builtin.apt:
    update_cache: yes
    cache_valid_time: 3600

Module: apt. The cache_valid_time: 3600 means "only run apt update if the cache is older than 1 hour." This prevents hitting the apt mirrors on every single run.

- name: Install base packages
  ansible.builtin.apt:
    name: "{{ base_packages }}"
    state: present

Variable: {{ base_packages }} is a list defined in group_vars/all.yml. The apt module accepts a list and installs everything in one transaction. state: present means "make sure it is installed." It does not upgrade if already present. Use state: latest if you want upgrades (but that makes runs less predictable).

- name: Create directory structure
  ansible.builtin.file:
    path: "{{ item }}"
    state: directory
    owner: "{{ main_user }}"
    group: "{{ main_user }}"
    mode: "0755"
  loop:
    - "{{ docker_compose_dir }}"
    - "{{ appdata_dir }}"
    - "{{ backup_mount }}"

The loop keyword: This runs the task once for each item in the list. {{ item }} refers to the current item. So this creates three directories: /opt/docker, /mnt/appdata, and /mnt/backups.

- name: Configure SSH daemon
  ansible.builtin.template:
    src: sshd_config.j2
    dest: /etc/ssh/sshd_config
    owner: root
    group: root
    mode: "0644"
    validate: "sshd -t -f %s"
  notify: restart sshd

Module: template. Renders roles/base/templates/sshd_config.j2 with variables filled in and places it at /etc/ssh/sshd_config. validate: Before writing the file, Ansible runs sshd -t -f <tempfile> to check the config syntax. If validation fails, the file is not written and the task fails. This prevents locking yourself out with a bad SSH config. notify: If the file changed, the restart sshd handler is triggered. If the file was already identical, no restart happens.

- name: Ensure passwordless sudo for main user
  ansible.builtin.copy:
    content: "{{ main_user }} ALL=(ALL) NOPASSWD:ALL\n"
    dest: "/etc/sudoers.d/{{ main_user }}"
    owner: root
    group: root
    mode: "0440"
    validate: "visudo -cf %s"

Module: copy with inline content instead of a source file. Writes a single line to /etc/sudoers.d/<YOUR_USER>. The validate runs visudo -cf to check sudoers syntax before committing.

Working with Ansible Vault¶

The vault stores secrets like the NAS samba password, the borg passphrase and service tokens. It is decrypted automatically at runtime using ~/ansible/.vault_pass, which should be chmod 600 and never committed to version control.

The commands you will use (view, edit, rekey), adding a new secret, and encrypting a single value are on the Vault page.

Troubleshooting and Quick Reference¶

• Something failed? "Permission denied", "Missing sudo password", "Could not resolve host", tasks that show changed on every run, template errors and verbose output are all on the Troubleshooting page.
• Commands and flags: the playbook and ad-hoc commands used in this guide, plus a table of the common flags (--limit, --check, --diff, --tags, -v…), are in the Quick Reference. The reasoning behind check mode and the safe-run workflow is in Ad-hoc commands & running playbooks.

Gotchas and Lessons Learned¶

Fresh Debian minimal does not include sudo. You must install it manually before Ansible can connect. This is the only manual step on a new host.

Jinja2 braces in heredocs. Never use cat << EOF in shell scripts to write files containing {{ }} syntax. The shell interprets the braces and breaks everything. Use a text editor (micro, nano, vim) instead, or use Ansible's template or copy module.

Vault file naming. In this repo, group_vars/vault.yml is loaded explicitly via vars_files in the playbook. This is by design. Ansible auto-loads files in group_vars/ that are named after a group in the inventory (e.g., group_vars/homelab.yml would auto-load for hosts in the homelab group). Since vault.yml does not match any group name, it must be loaded explicitly.

Ref: https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html

Docker apt repo on new Debian releases. The Docker role uses {{ ansible_facts['distribution_release'] }} to build the repo URL (the Tailscale role does the same). Docker officially supports Debian 13 Trixie, so this works today. However, when future Debian releases come out (e.g. Forky), Docker may not have packages ready on day one. In that case, you would temporarily hardcode the previous release name in the template until Docker catches up.

Ref: https://docs.docker.com/engine/install/debian/

The command module always shows changed. Read-only commands like lsb_release -cs still report changed. Use changed_when: false to suppress this.

Ansible connects via the key in ansible.cfg. If you rotate SSH keys, update private_key_file in ansible.cfg and re-copy the new public key to all hosts.

The --limit flag is your safety net. When testing changes, always use --limit to target a single host. Running against the whole lab when you meant to test on server3 can cause unintended changes.

Checklist: server3 from Nothing to Complete¶

For quick reference, here is the entire process as a condensed checklist:

• Create DHCP static mapping in OPNsense for server3
• Create dnsmasq host entry for server3.example.com
• Install Debian 13 Trixie on server3 (hostname: server3, user: <YOUR_USER>)
• SSH in, install sudo, configure passwordless sudo
• Copy Ansible SSH key from your control node: ssh-copy-id -i ~/.ssh/ansible.pub <YOUR_USER>@server3.example.com
• Verify server3.example.com is already in inventory/hosts.ini under [standby]
• Test connectivity: ansible server3.example.com -m ping
• Dry run: ansible-playbook playbooks/base.yml --check --limit server3.example.com
• Run for real: ansible-playbook playbooks/base.yml --limit server3.example.com
• Verify: Docker running, NAS mounted, borgmatic configured, SSH hardened
• Verify idempotency: re-run playbook, expect changed=0
• (Optional) Deploy a test stack to practice the compose workflow
• (Optional) Clean up the test stack when done