Upgrade from 1.4.x to 1.5.x breaks all app connectivity (tested on Raspberry Pi 5) EDIT: still happening in 1.7.2

[jleaders]

Upgrade from 1.4.x to 1.5.x breaks all app connectivity (tested on Raspberry Pi 5) -- EDIT: still happening in 1.7.2

Missing iptables FORWARD rules on umbrel_main_network means no apps can get a port.

Environment

• Device: Raspberry Pi 5
• OS: UmbrelOS 1.5
• Upgrade Path: Upgraded from previous version (issue appeared after upgrade)

Problem Description

After upgrading to UmbrelOS 1.5 on Raspberry Pi 5, all Umbrel apps become inaccessible from the local network. Containers are running and healthy, but connections to any app port fail or are rejected.

Symptoms

• SSH to Umbrel and the web dashboard (port 80) work fine
• docker ps shows all containers running
• All app URLs timeout or get "Connection reset by peer"
• Inter-container communication fails (apps can't reach their databases, app_proxy can't reach backends)
• app_proxy containers log either "Waiting for <host>:<port> to open..." or "The address '<container>' cannot be found"

Root Cause Analysis

There are two bugs that compound each other:

Bug 1: Docker doesn't manage iptables for umbrel_main_network

The umbrel_main_network is created as an external network by umbreld rather than by Docker Compose directly. Docker's iptables management creates FORWARD chain rules for networks it manages, but not for this externally-created network. Since the FORWARD chain default policy is DROP, all traffic to/from/within the umbrel_main_network bridge is silently dropped.

Evidence — other bridges have rules, umbrel_main_network does not:

$ sudo iptables -L FORWARD -n -v | grep br-
... ACCEPT all -- * br-eb4acdafcd65 ... ctstate RELATED,ESTABLISHED
... ACCEPT all -- br-eb4acdafcd65 !br-eb4acdafcd65 ...
... ACCEPT all -- br-eb4acdafcd65 br-eb4acdafcd65 ...
# (no rules for the umbrel_main_network bridge)

Bug 2: The network ID changes on every umbrel.service restart

Each time umbrel.service starts, it runs docker compose up which destroys and recreates the umbrel_main_network. This gives the bridge a new ID every time (e.g., br-bfea11e6272f → br-69d73c5945d2 → br-07efc0bf5a6e). Any iptables rules referencing the old bridge name become stale and have no effect.

Why the previous workaround didn't work

The systemd service in the original workaround uses After=docker.service, but umbrel_main_network doesn't exist until umbrel.service creates it. Both services start After=docker.service, so the fix script runs before the network exists. The docker network inspect call fails silently, no rules get applied, and all apps remain broken.

The cascade failure

1. umbrel.service starts → creates umbrel_main_network with a new bridge ID
2. Docker doesn't create FORWARD rules for this externally-managed network
3. FORWARD policy is DROP → all inter-container TCP traffic is dropped
4. Apps can't reach their databases → apps crash → restart: on-failure kicks in
5. app_proxy containers can't reach backends → return "connection reset" to clients
6. The whole stack looks "running" but nothing is actually reachable

Fix

The fix script must:

1. Run after umbrel.service, not just after docker.service
2. Wait for umbrel_main_network to actually exist (handles slow startup)
3. Use the current bridge ID dynamically
4. Clean up stale rules from old (dead) bridges

Installation

# 1. Create the fix script
sudo tee /usr/local/bin/fix-umbrel-network.sh > /dev/null << 'SCRIPT'
#!/bin/bash
# Fix missing iptables FORWARD rules for umbrel_main_network
# https://github.com/getumbrel/umbrel/issues/2101

MAX_WAIT=120
WAITED=0

echo "Waiting for umbrel_main_network to be created..."
while ! docker network inspect umbrel_main_network >/dev/null 2>&1; do
    sleep 2
    WAITED=$((WAITED + 2))
    if [ $WAITED -ge $MAX_WAIT ]; then
        echo "ERROR: umbrel_main_network not found after ${MAX_WAIT}s"
        exit 1
    fi
done

NETWORK_ID=$(docker network inspect umbrel_main_network --format '{{.Id}}' | cut -c1-12)
BRIDGE="br-${NETWORK_ID}"

if ! ip link show "$BRIDGE" >/dev/null 2>&1; then
    echo "ERROR: Bridge $BRIDGE does not exist"
    exit 1
fi

# Clean up stale rules from previous bridge IDs
for old_bridge in $(iptables -L FORWARD -n | grep -oP 'br-[a-f0-9]+' | sort -u); do
    if [ "$old_bridge" != "$BRIDGE" ] && ! ip link show "$old_bridge" >/dev/null 2>&1; then
        echo "Removing stale rules for $old_bridge"
        iptables -D FORWARD -i "$old_bridge" -o "$old_bridge" -j ACCEPT 2>/dev/null || true
        iptables -D FORWARD -i "$old_bridge" ! -o "$old_bridge" -j ACCEPT 2>/dev/null || true
        iptables -D FORWARD -o "$old_bridge" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || true
        iptables -D FORWARD -o "$old_bridge" -j ACCEPT 2>/dev/null || true
    fi
done

# Add FORWARD rules matching Docker's standard pattern
iptables -I FORWARD -o "$BRIDGE" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -I FORWARD -i "$BRIDGE" ! -o "$BRIDGE" -j ACCEPT
iptables -I FORWARD -i "$BRIDGE" -o "$BRIDGE" -j ACCEPT

echo "Applied iptables FORWARD rules for $BRIDGE (waited ${WAITED}s)"
SCRIPT

sudo chmod +x /usr/local/bin/fix-umbrel-network.sh

# 2. Create systemd service (note: After=umbrel.service is critical)
sudo tee /etc/systemd/system/fix-umbrel-network.service > /dev/null << 'EOF'
[Unit]
Description=Fix missing iptables FORWARD rules for umbrel_main_network
After=docker.service umbrel.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/fix-umbrel-network.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

# 3. Enable and start
sudo systemctl daemon-reload
sudo systemctl enable fix-umbrel-network.service
sudo systemctl start fix-umbrel-network.service

Verify

# Check the service ran successfully
sudo systemctl status fix-umbrel-network.service

# Confirm rules exist for the current bridge
BRIDGE="br-$(docker network inspect umbrel_main_network --format '{{.Id}}' | cut -c1-12)"
sudo iptables -L FORWARD -n -v | grep "$BRIDGE"

# Test an app port
curl -I http://umbrel.local:5678  # or any app port

Apps may take 1–2 minutes to fully recover after applying the fix, as the app_proxy containers need to restart and reconnect to their backends.

After a reboot

The service runs automatically. Apps should be accessible within 2–3 minutes of boot. If not, check:

sudo journalctl -u fix-umbrel-network.service --no-pager

Key difference from the original workaround

	Original workaround	This fix
Runs after	docker.service only	docker.service and umbrel.service
Handles missing network	Fails silently	Waits up to 120s with polling
Stale bridge rules	Accumulates dead rules	Cleans up rules for non-existent bridges
Overly broad rules	Includes iptables -I FORWARD -o $BRIDGE -j ACCEPT (accepts ALL inbound)	Uses only the 3 standard Docker rules

Proper fix (for Umbrel maintainers)

The underlying issue is that umbreld creates umbrel_main_network as a Docker network but Docker doesn't wire up iptables FORWARD rules for it. Possible proper fixes:

1. Have umbreld add the iptables rules itself after creating the network, since it already runs as root
2. Create the network with com.docker.network.bridge.enable_icc=true and ensure Docker manages the iptables rules
3. Use docker network create with --opt com.docker.network.bridge.name=umbrel0 to give the bridge a stable name, then add a persistent iptables rule for umbrel0 that survives network recreation

Impact

• Severity: Critical — all apps completely inaccessible
• Affected: Raspberry Pi 5 (confirmed), likely affects other ARM devices running UmbrelOS 1.5
• Trigger: Upgrade to 1.5.x, or any reboot / umbrel.service restart