Networking & DNS: lessons learned¶

Real problems that looked like "the app is broken" but turned out to be DNS or routing underneath. Each entry is symptom → cause → how to diagnose → fix.

AAAA records with no IPv6 route¶

Symptom. .NET apps (Sonarr, Radarr, Prowlarr, Jellyfin, …) time out talking to indexers, update checks or metadata sites. The web UI feels like it hangs. Logs show SocketException (11): Resource temporarily unavailable or Prowlarr's hint "This is typically caused by DNS/SSL issues. Check DNS settings, ensure IPv6 is working or disabled". Later, docker pull for Docker Hub images stalls for many minutes while images from other registries pull fine.

Cause. Your DNS server is dual-stack and hands out AAAA (IPv6) records, but the host has no IPv6 default route. Tools with good "Happy Eyeballs" fallback (curl, browsers) quietly retry over IPv4. .NET's HttpClient and some Docker registry connections don't fall back nearly as gracefully, so every request waits on an IPv6 connection that can never succeed. It often shows up right after a DNS change, e.g. pointing Docker at a new resolver in /etc/docker/daemon.json.

Diagnose.

# Does the resolver return IPv6 addresses?
dig AAAA registry-1.docker.io +short

# Is there an IPv6 default route? Empty output = no IPv6 connectivity
ip -6 route show default

# Compare v4 vs v6 directly
curl -4 -sS -o /dev/null -w '%{time_total}\n' https://registry-1.docker.io/v2/
curl -6 -sS -o /dev/null -w '%{time_total}\n' https://registry-1.docker.io/v2/   # fails/hangs

AAAA answers + no default route = this problem.

Fix. Pick one, from narrowest to broadest:

services:
  sonarr:
    environment:
      - DOTNET_SYSTEM_NET_DISABLEIPV6=true

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

sudo sysctl --system      # apply now, persists across reboots
ip a | grep inet6         # should be empty

Containers can't resolve names after a reboot (allow-hotplug vs auto)¶

Symptom. After a host reboot, some containers (often the ones that start first, e.g. your Git server, deployment agent or reverse proxy) fail with Could not resolve host, lookup … server misbehaving, or NO EXTERNAL NAMESERVERS DEFINED. The host itself resolves fine once it's up. Restarting the container fixes it until the next reboot.

Cause. On Debian/ifupdown, the NIC is declared only as allow-hotplug in /etc/network/interfaces. ifup -a at boot only brings up interfaces marked auto; allow-hotplug interfaces are brought up when udev detects them. So network-online.target can be reached before the NIC has a DHCP lease and DNS servers, and docker.service (ordered after network-online.target) starts in that gap. Docker gives each container a copy of the host's /etc/resolv.conf when the container starts, so early containers keep the empty one.

Diagnose.

grep -E '^(auto|allow-hotplug)' /etc/network/interfaces

# Did the container start before resolv.conf was written?
uptime -s
stat -c '%y' /etc/resolv.conf
docker inspect -f '{{.Name}} {{.State.StartedAt}}' $(docker ps -q)

# What does the container actually have?
docker exec <container> cat /etc/resolv.conf

Container start time earlier than the host's resolv.conf mtime, plus an allow-hotplug-only NIC, is this problem.

Fix. Add an auto line for the interface (keeping allow-hotplug is harmless):

auto <INTERFACE>
allow-hotplug <INTERFACE>
iface <INTERFACE> inet dhcp

Now ifup -a brings it up synchronously and network-online.target waits for it. For containers already running with a bad resolv.conf, docker restart <container>. If you build hosts from a template or Ansible role, fix it there, because every host built from it will have the same race.

DNS rate limiting a whole server subnet (502s from the reverse proxy)¶

Symptom. Intermittent 502 Bad Gateway from Nginx Proxy Manager (or another reverse proxy) with could not be resolved (2: Server failure) in its error log. Uptime monitors flap. Nothing is wrong with the backends, and it's worse when several servers are busy at once.

Cause. AdGuard Home's rate limit. By default it allows a fixed number of requests per second per client subnet (ratelimit + ratelimit_subnet_len_ipv4, a /24 by default), not per client. All your servers on one VLAN share one bucket. Queries over the limit are silently dropped (UDP), so the client just times out. A reverse proxy that resolves upstream hostnames per request (short or 1-second TTLs on local records) burns through that budget fast.

Diagnose. Fire a burst of queries at AdGuard and count failures, then compare with the upstream resolver directly:

# 300 back-to-back queries through AdGuard Home
for i in $(seq 300); do
  dig @<YOUR_DNS_SERVER_IP> example.com +short +tries=1 +time=1 >/dev/null || echo fail
done | wc -l

# Same thing straight at the upstream resolver (e.g. Unbound on its own port)
for i in $(seq 300); do
  dig @<YOUR_DNS_SERVER_IP> -p 5335 example.com +short +tries=1 +time=1 >/dev/null || echo fail
done | wc -l

Failures through AdGuard but none against the upstream = rate limiting.

Fix. In AdGuard Home → Settings → DNS settings → Rate limit, raise the limit (a few hundred is plenty for a homelab) or set it to 0 to disable it. Then re-run the burst test and confirm zero failures.

Also consider longer TTLs on your local DNS records so the proxy isn't resolving on every request.

CrowdSec banning your own server¶

Symptom. One server suddenly can still talk to machines on its own subnet but nothing routed: no gateway, no DNS, no internet, nothing on other VLANs. It comes back on its own a few hours later. It may happen at about the same time on several days, and any scheduled job that runs during the window (backups, updates) fails.

Cause. CrowdSec on the firewall (e.g. the OPNsense/pfSense plugin) decided the server was port-scanning and banned it. A common trigger is the pf-scan-multi_ports scenario firing on legitimate burst traffic: for example a service that gets paused for a nightly job (a SnapRAID sync, a backup) while clients keep polling it, then answers hundreds of queued connections at once on resume. The firewall drops those late replies as out-of-state, and many distinct ports in a few milliseconds look exactly like a scan.

Diagnose.

# On the firewall (or wherever the CrowdSec LAPI runs)
cscli decisions list --ip <YOUR_SERVER_IP>
cscli alerts list --ip <YOUR_SERVER_IP>        # which scenario fired, and when

On OPNsense you can also check Services → CrowdSec → Decisions and the CrowdSec log under the plugin's log view, and Firewall → Diagnostics → Aliases for the blocklist table. Line up the alert time with your cron/timer schedule.

Fix.

1. Lift the ban: cscli decisions delete --ip <YOUR_SERVER_IP> (or delete the decision in the plugin UI).
2. Fix the trigger. Better than allowlisting: stop the burst. For a pause/resume job, stop the service cleanly (ending client sessions) and start it again afterwards instead of freezing it with docker pause, so clients reconnect normally rather than flooding it.

3. Allowlist narrowly if you must, with the server's IP only, not your whole internal range:

   cscli allowlists create homelab-servers -d "internal servers"
   cscli allowlists add homelab-servers <YOUR_SERVER_IP> -d "server that bursts on resume"
   

   A blanket allowlist for all internal subnets also stops CrowdSec from catching a compromised machine inside your network.

Desktop VPN tunnelling your LAN (hung network mounts)¶

Symptom. With a VPN client connected on your desktop, the file manager freezes or gets killed, file dialogs hang, and anything touching a CIFS/NFS mount from your NAS locks up. Machines on your own subnet are fine; servers on other VLANs aren't. Local local hostnames stop resolving too.

Cause. The VPN client installs a policy route that sends everything except your directly connected subnet into the tunnel. Your other VLANs (where the NAS lives) are "not local" to the desktop, so their traffic goes to the VPN provider and dies. The mounts are hard/soft-hung waiting on a server that's now unreachable. The VPN also replaces your DNS servers with the provider's, which don't know your local names.

Diagnose.

ip route get <NAS_IP>        # "dev <vpn-interface>" = it's going into the tunnel
ip rule                      # look for the VPN's fwmark/table rules
resolvectl status            # or: cat /etc/resolv.conf — which DNS is active?

Fix. Add a route for your whole home address range via your LAN gateway, on the LAN connection, so it wins over the VPN's catch-all. With NetworkManager:

nmcli connection modify "<LAN_CONNECTION>" +ipv4.routes "<HOME_RANGE> <YOUR_GATEWAY_IP> 100"
nmcli connection up "<LAN_CONNECTION>"
ip route get <NAS_IP>        # should now show your LAN interface

<HOME_RANGE> is a supernet covering all your VLANs (for example a /16 if your VLANs are all /24s inside it). Most WireGuard-based clients install a rule that ignores the VPN table for more specific routes in the main table (suppress_prefixlength 0), which is why this works. Many VPN apps also have an "allow LAN access" or split-tunnelling setting that does the same.

For local names while the VPN is up you need split DNS: send your local domain to your own resolver and everything else to the VPN's. For example, with NetworkManager's dnsmasq plugin (dns=dnsmasq) add server=/<YOUR_LOCAL_DOMAIN>/<YOUR_DNS_SERVER_IP>, or use resolvectl domain with systemd-resolved.

Mount network shares with soft (or use automount, see Storage) so an unreachable server can't freeze your desktop indefinitely.

Tailscale peers are blocked by the firewall although the rules say "pass"¶

Symptom. With OPNsense (or pfSense) as a Tailscale subnet router / exit node / DNS server: tailscale ping <firewall> works in a millisecond, but DNS queries to the firewall's tailnet address time out, subnet routes go nowhere and the exit node passes no traffic. tailscaled logs dns udp query: … context deadline exceeded and the Tailscale client shows a dns-forward-failing health warning. Hosts using MagicDNS (100.100.100.100) then fail lookups that work fine on the LAN.

Cause. The rules on the Tailscale interface use the interface's net alias (Tailscale net, TS net) as source. On the firewall the tailscale interface often has a single address with a /32 mask, so "net" means only the firewall itself. No peer ever matches, and everything from the tailnet falls through to the default deny. The DNS server's own ACL is fine; pf drops the packets first.

Diagnose.

• Firewall → Diagnostics → Aliases (or the rule's tooltip): what does the source alias resolve to? A single /32 is the problem.
• The live firewall log shows tailnet sources (100.64.0.0/10) blocked by the default deny on the Tailscale interface.
• From a peer: dig @<FIREWALL_TAILNET_IP> example.com times out, while dig @<YOUR_DNS_SERVER_IP> example.com on the LAN answers.

Fix. Create an alias tailnet_v4 = 100.64.0.0/10 and use it as the source of every Tailscale rule, in this order:

1. pass → the firewall's tailnet IP, port 53 (DNS)
2. block → This Firewall (otherwise the exit-node rule below opens the GUI and SSH on every firewall address: PrivateNetworks doesn't include 100.64.0.0/10)
3. block → management and IoT networks
4. pass → the networks the tailnet should reach
5. pass → !PrivateNetworks (exit node: internet only)

If you use the firewall as an exit node, add an outbound NAT rule for 100.64.0.0/10 → WAN address too; FreeBSD's tailscale doesn't masquerade by itself. Test from a phone on mobile data: DNS, a LAN service, and an IP-check site through the exit node. Until the rules work, tailscale up --accept-dns=false on servers keeps them on the LAN resolver.

Wildcard DNS keeps dead names alive (and publishes new ones)¶

Symptom. Hostnames for services you shut down long ago still resolve and return something (a login page, an error page). New proxy hosts become reachable from the internet the moment you create them, before you meant to publish them. The apex of the domain returns a Cloudflare error 1033 (tunnel not found / no connector).

Cause. A wildcard record (*.<YOUR_DOMAIN> → tunnel or proxy) plus a wildcard ingress rule answers for every name. Nothing ever fails to resolve, so you can't tell what's live from DNS. A record left pointing at a deleted tunnel returns 1033.

Diagnose.

dig +short random-$RANDOM.<YOUR_DOMAIN>       # answers = a wildcard exists
curl -sI https://random-$RANDOM.<YOUR_DOMAIN>/ | head -3
dig +short <YOUR_DOMAIN>                      # the apex: does it point at a live target?

Fix.

• Prefer explicit records and explicit tunnel ingress rules per hostname, ending with a catch-all service: http_status:404. Publishing something then takes a deliberate step.
• If you keep a wildcard (it's convenient), make sure the SSO policy in front of it is deny by default, and review which hostnames your proxy actually serves.
• Delete records and tunnels you no longer use, including the apex.
• While you're in the DNS zone of a domain that sends no mail: v=spf1 -all, a DMARC p=reject record and a CAA record cost nothing and stop others spoofing it.

Reverse proxy certificates drift: old CA, missing HTTPS, expired origin cert¶

Symptom. Some internal hostnames show a certificate warning while others are fine. A host you set to "Force SSL" still answers on plain HTTP, and HTTPS on it fails. The proxy's renewal job logs an error every hour. A public certificate has been expired for months and nothing broke.

Cause. In Nginx Proxy Manager (and similar UIs), hosts reference a certificate by ID. Delete or replace a certificate and the hosts that used it can keep pointing at the deleted one: some serve the stale files, some get no 443 server block rendered at all, even though the database says SSL is forced. Behind a CDN or tunnel, the edge terminates TLS, so an expired origin certificate is invisible from outside.

Diagnose.

# Which certificate does a host really serve, and until when?
openssl s_client -connect <YOUR_SERVER_IP>:443 -servername <SERVICE>.<YOUR_LOCAL_DOMAIN> </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -enddate

# Does plain HTTP redirect, or serve the app?
curl -sI http://<SERVICE>.<YOUR_LOCAL_DOMAIN>/ | head -1

Fix. In the proxy's UI (not by editing its database), reassign the current certificate to every host, tick Force SSL, and save each host so its config is re-rendered. Delete certificates you no longer use. Then monitor expiry: most uptime tools can alert N days before a certificate expires, and a private CA's leaf certificates need a renewal reminder of their own.

IP bans and ACLs behind a reverse proxy¶

Symptom. Either: turning on a login ban (Home Assistant ip_ban_enabled, fail2ban, CrowdSec) bans the proxy, locking everyone out. Or: an app's "allow without login from local addresses" setting lets the whole internet in, because every request arrives from the proxy's private IP.

Cause. Behind a proxy or tunnel, the app sees the proxy's address unless it trusts X-Forwarded-For from that proxy. Trust too little and every client looks like one local IP. Trust too much (a whole /12, every Docker network) and anyone who can reach the app directly can forge the header.

Diagnose. Look at what the app records as the client address for a few known sessions (its log, its session list). If you see your proxy's container IP, it isn't getting the real client IP. If you see real public and LAN addresses, it is.

Fix.

• Configure the app's trusted proxies to the exact proxy addresses (give the proxy and tunnel containers fixed IPs on their Docker network), and the proxy's own set_real_ip_from to the exact upstream (the tunnel connector).
• Only enable IP bans after confirming the app sees real client IPs.
• Never rely on "authentication disabled for local addresses" for anything reachable through a proxy. Require a login, and put admin UIs behind SSO as well.

DNS rebind protection breaks *.plex.direct (and similar)¶

Symptom. Local Plex clients connect, but only insecurely over plain HTTP; the dashboard never shows a secure local connection. Other apps that hand out public names for LAN addresses misbehave the same way.

Cause. Plex gives each server a public name like <ip-with-dashes>.<hash>.plex.direct that resolves to its private LAN address. A resolver with DNS rebind protection (Unbound's private-address filtering, a rebind blocklist in AdGuard Home or Pi-hole) strips answers that point public names at private ranges, so the lookup returns no address and the client falls back to HTTP.

Diagnose.

dig +short <ip-with-dashes>.<hash>.plex.direct @<YOUR_DNS_SERVER_IP>   # empty
dig +short <ip-with-dashes>.<hash>.plex.direct @1.1.1.1                # the LAN IP

Fix. Keep rebind protection on and add an exception for that one domain: in OPNsense Services → Unbound DNS → Advanced → Private Domains add plex.direct (Unbound's private-domain:). Don't enable a rebind-protection blocklist in AdGuard Home without the same exception.

Discovery doesn't cross VLANs (and apps call other VLANs "remote")¶

Symptom. After splitting the network into VLANs: Syncthing devices stop finding each other, KDE Connect can't see the phone, Chromecast or DLNA devices disappear, and Plex labels a TV on another VLAN as (WAN) and applies remote streaming limits.

Cause. Local discovery uses broadcast or multicast, which stays inside one broadcast domain (and doesn't cross Tailscale either). Apps that decide "local vs remote" by subnet only know the subnets you've told them about.

Fix.

• Give peers explicit addresses instead of relying on discovery: Syncthing device addresses like tcp://<NAS_HOSTNAME>:22000, KDE Connect "Add device by IP".
• Allow just those ports between the VLANs involved (Syncthing 22000/tcp, KDE Connect 1714-1764), and write them into your segmentation plan so tightening the rules later doesn't break them.
• Add your other VLANs to the app's list of local networks (Plex: Settings → Network → LAN Networks), or move the client onto the same VLAN as the server.
• For mDNS-dependent devices (casting), an mDNS repeater on the firewall is the usual answer, scoped to the VLANs that need it.

Sources & further reading¶

• interfaces(5) — Debian manpage
• .NET environment variables — DOTNET_SYSTEM_NET_DISABLEIPV6
• Servarr Wiki — Prowlarr troubleshooting: DNS/SSL connection issues
• AdGuard Home — Configuration (wiki)
• CrowdSec — Centralized allowlists
• CrowdSec — cscli decisions
• NetworkManager — nmcli
• Tailscale — Install on OPNsense
• Tailscale — Exit nodes
• Cloudflare — Tunnel configuration file (ingress rules, http_status:404)
• Cloudflare — Error 1033
• Home Assistant — HTTP integration (trusted_proxies, ip_ban_enabled)
• nginx — ngx_http_realip_module
• Unbound — unbound.conf (private-address, private-domain)
• Plex — Secure connections and DNS rebinding
• Syncthing — Device addresses