The 3 AM Power Outage
Power outage. Both nodes reboot.
Smart Node (bare metal Debian) boots in ~30 seconds. Infra Node (Proxmox + ZFS + VMs) takes ~3 minutes — ZFS pool import, VM startup, Samba/NFS services. The infrastructure node has the storage. The smart node has the compute. They don’t talk to each other during boot.
Smart Node’s media stack starts via Docker Compose / systemd. Sonarr, Radarr, Plex, qBittorrent all try to reach /tank/media via NFS. The share doesn’t exist yet. Infra Node is still importing the ZFS pool, probably contemplating the meaning of life.
Containers crash, enter restart loops. Sonarr marks series as "unavailable." Plex shows "media not found." Sometimes Sonarr would re-download episodes it thought were missing — because nothing says "I’m helping" like downloading 50GB of content you already have.
After every power outage, I’d SSH into Smart Node, see containers restarting, manually wait for Infra Node, then restart the media stack. At 3 AM. In my pajamas. Wondering why I thought "homelab" sounded fun.
The Fix: Make Smart Node Wait
Lesson: In a multi-node setup, boot order matters. Don’t assume the storage node will be ready. Make the compute node wait explicitly. It’s like teaching a teenager to wait for the bathroom to be free — except the teenager is a container orchestration system and the bathroom is a ZFS pool.
Systemd drop-in units for the media stack containers with After=network-online.target and a custom oneshot service that waits for the NFS mount.
# /etc/systemd/system/wait-for-nfs.service
[Unit]
Description=Wait for Infra Node NFS share
DefaultDependencies=no
After=network-online.target
Before=docker.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/wait-for-nfs.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
#!/bin/bash
# /usr/local/bin/wait-for-nfs.sh
SHARED_HOST="infra-node"
SHARED_PATH="/tank/media"
MOUNT_POINT="/tank/media"
TIMEOUT=180
INTERVAL=5
echo "Waiting for $SHARED_HOST:$SHARED_PATH to be available..."
for ((i=0; i<TIMEOUT; i+=INTERVAL)); do
if showmount -e "$SHARED_HOST" 2>/dev/null | grep -q "$SHARED_PATH"; then
echo "NFS share detected, attempting mount..."
if mount "$MOUNT_POINT" 2>/dev/null; then
echo "Mount successful"
exit 0
fi
fi
sleep "$INTERVAL"
done
echo "ERROR: Timeout waiting for NFS share"
exit 1
The script is essentially "are we there yet?" in a loop with a 3-minute timeout. If systemd had a Patience= directive, I’d have used that instead.
Then add to each media stack container’s systemd unit (or Docker Compose with depends_on + condition: service_healthy):
# docker-compose.yml snippet
services:
sonarr:
depends_on:
wait-for-nfs:
condition: service_completed_successfully
Result: Smart Node boots, media stack waits. Infra Node finishes, exports NFS, wait-for-nfs succeeds, media stack starts clean. Zero manual intervention after power loss. I can now enjoy power outages like a normal person — confused, then back to sleep.
The Honest Summary
Multi-node homelabs have a hidden dependency graph. The storage node must be ready before the compute node starts its workloads. Systemd gives you the tools to express that explicitly — After, Before, WantedBy, oneshot services. It’s basically a directed acyclic graph, but for boot order, and you’re the one drawing the arrows.
The fix is ~30 lines of bash and systemd config. The peace of mind after a power outage is worth infinitely more. Also I no longer have to explain to my partner why the TV is "downloading the entire series again" at 3 AM.
Moral: If your smart node is smarter than your infra node, you have a boot race. Make the smart node wait. It’s not lazy — it’s synchronized.