podman

Overview: Standalone container workloads on ul run under rootless Podman (podman 6.x) managed as native systemd user services via Podman Quadlets under the dedicated unprivileged account docker-data (UID 1100, GID 1100). This eliminates the privileged root daemon attack surface while preserving the per-service ULA IPv6 reverse-proxy architecture on eth77.

1. Architecture & Security Comparison: Rootless Podman vs. Docker

Under traditional Docker, dockerd runs as root (UID 0) inside the host init user namespace (user:[4026531837]). Every container spawned by dockerd without user-namespace remapping shares the host kernel's root UID 0 whenever the container process runs as root, and any vulnerability in the daemon or runtime starts with full host privileges.

Under rootless Podman, there is no central root daemon. Instead, systemd manages a per-user service manager (user@1100.service) for docker-data, which spawns lightweight conmon and crun processes entirely inside an unprivileged Linux user namespace (user:[4026534031]).

1.1 User Namespace UID/GID Translation Model

Rootless Podman maps container UIDs and GIDs into unprivileged host IDs using /etc/subuid and /etc/subgid (docker-data:624288:65536). Under the default rootless mapping (UserNS=host within the rootless user namespace), any container UID $u \in [0, 65536]$ translates to a host kernel UID according to:

$$ \text{UID}\sb{_{\text{host}}(u) = \begin{cases} 1100 & \text{if } u = 0 \quad (\text{container } \texttt{root} \longrightarrow \text{host } \texttt{docker-data}) \cr 624287 + u & \text{if } 1 \le u \le 65536 \quad (\text{subordinate range } 624288\text{--}689823) \end{cases} $$

When a service explicitly sets UserNS=keep-id with User=1100:1100 (such as drawpile, whose binary refuses to start if internal getuid() == 0), the identity mapping preserves UID 1100 across the namespace boundary:

$$ \text{UID}\sb{_{\text{container}keep-id}}(1100) \stackrel{\texttt{keep-id}userns}}{\longleftrightarrow} 1100 \text{UID} (\sb{\text{host}}(1100)texttt{docker-data}) $$

1.2 Side-by-Side Comparison

2. Host Configuration on ul

2.1 Unprivileged Account, Subordinate IDs & Systemd Lingering

1. Service Account (docker-data):
   • Primary UID/GID: 1100:1100 (docker-data)
   • Home Directory: /home/docker-data
   • Supplementary Groups: video (91) and render (991) for AMD ROCm GPU device access (/dev/dri and /dev/kfd).
2. Subordinate UID/GID Ranges (/etc/subuid & /etc/subgid):

   docker-data:624288:65536
   

3. Systemd User Lingering: Enabled via sudo loginctl enable-linger docker-data (creating /var/lib/systemd/linger/docker-data). This ensures user@1100.service starts automatically at system boot and keeps all rootless Podman Quadlet services running without requiring an active login session.

2.2 Global Configuration Files & Storage Layout

• /etc/containers/registries.conf: Configures unqualified image names to resolve against Docker Hub by default:

  unqualified-search-registries = ["docker.io"]
  

• /etc/containers/containers.conf:

  [network]
  default_network_backend = "pasta"
  pasta_options = ["-a", "10.0.2.100", "-a", "fd00::100"]
  

• Storage Paths:
  • Persistent image/layer store (GraphRoot): /home/docker-data/.local/share/containers/storage (native kernel overlay filesystem on NVMe /).
  • Ephemeral runtime state (RunRoot): /run/user/1100/containers.
  • Quadlet unit directory: /home/docker-data/.config/containers/systemd/ (contains symlinks pointing to /data/docker/<service>/<service>.container).

2.3 Networking Architecture on ul: Why Network=bridge Is Used

Every container service on ul binds its published port to a dedicated ULA IPv6 address on eth77 (fd77:7777:5010:0:1337:f1ac:d00d:<hex>/64), and host Nginx (UID 33 http) reverse-proxies requests to http://[fd77:...:<hex>]:<port>.

All rootless Quadlet definitions on ul specify Network=bridge rather than default pasta networking for two architectural reasons:

1. Preventing Local ULA IPv6 Hairpin Collisions: By default, pasta clones the host's eth77 interface and copies all 36 /64 ULA IPv6 addresses from the host into the container namespace. When host Nginx connects from the same machine (src = fd77:...:<hex>, dst = fd77:...:<hex>), a pasta-networked container sees the source IP as a local address on its own interface unless isolated.
2. Seamless IPv6-to-IPv4 Protocol Translation (rootlessport): Many container images (tldraw, sopro, demucs, drawpile, navidrome, vaultwarden) bind their internal server to IPv4 0.0.0.0:<port> only (not IPv6 [::]:<port>). With Network=bridge, Podman spawns /usr/lib/podman/rootlessport on the host side to accept IPv6 connections on [fd77:...:<hex>]:<host_port> and proxy them cleanly over IPv4 into the container's netavark bridge interface (10.88.0.x:<container_port>):

$$ \text{Nginx } (\texttt{http}) \stackrel{\text{IPv6 TCP}}{\longrightarrow} [\texttt{fd77:...:hex}]:P\sb{P_{\text{host}} \ (\texttt{rootlessport}) \stackrel{\text{IPv4 TCP}}{\longrightarrow} \texttt{10.88.0.x}:P\sb{P(\text{container}}) $$

3. Quadlet Service Architecture & Migrated Inventory

3.1 How Quadlets Work

Instead of docker-compose.yml, each service directory /data/docker/<service>/ contains a declarative <service>.container file symlinked into /home/docker-data/.config/containers/systemd/<service>.container.

Whenever sudo systemctl --user -M docker-data@ daemon-reload runs (or on boot), /usr/lib/systemd/user-generators/podman-system-generator compiles each .container file into a transient systemd unit at /run/user/1100/systemd/generator/<service>.service.

3.2 Migrated Rootless Podman Service Inventory

3.3 Reference Quadlet Patterns

[Unit]
Description=Vaultwarden Rootless Podman Container
After=network-online.target
Wants=network-online.target

[Container]
ContainerName=vaultwarden
Image=docker.io/vaultwarden/server:latest
Network=bridge
User=0:0
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:a117]:8222:80
EnvironmentFile=/data/docker/vaultwarden/.env
Environment=TZ=America/New_York
Volume=/data/docker/vaultwarden/vw-data:/data

[Service]
Restart=always

[Install]
WantedBy=default.target

[Unit]
Description=Drawpile Rootless Podman Container
After=network-online.target
Wants=network-online.target

[Container]
ContainerName=drawpile
Image=docker.io/drawpile/drawpile-srv:2
Network=bridge
UserNS=keep-id
User=1100:1100
Exec=--port 27750 --websocket-port 27751 --web-admin-port 27752 --web-admin-access all --web-admin-auth admin:drawpile123 --sessions /data/sessions --database /data/sessions/drawpile.db
Volume=/data/docker/drawpile/sessions:/data/sessions
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:d7a0]:27750:27750
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:d7a0]:27751:27751
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:d7a0]:27752:27752

[Service]
Restart=always

[Install]
WantedBy=default.target

[Unit]
Description=Demucs ROCm Rootless Podman Container
After=network-online.target
Wants=network-online.target

[Container]
ContainerName=demucs
Image=localhost/demucs:latest
Network=bridge
AddDevice=/dev/dri:/dev/dri
AddDevice=/dev/kfd:/dev/kfd
GroupAdd=keep-groups
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:de3]:8000:8000
EnvironmentFile=/data/docker/demucs/.env
Environment=TZ=America/New_York
Environment=TORCH_HOME=/root/.cache/torch
Environment=HF_HOME=/root/.cache/huggingface
Environment=MIOPEN_USER_DB_PATH=/root/.cache/miopen
Environment=MIOPEN_CUSTOM_CACHE_DIR=/root/.cache/miopen
Volume=/data/docker/demucs/app.py:/app/app.py:ro
Volume=/data/docker/demucs/static:/app/static:ro
Volume=/data/docker/demucs/cache/torch:/root/.cache/torch
Volume=/data/docker/demucs/cache/huggingface:/root/.cache/huggingface
Volume=/data/docker/demucs/cache/miopen:/root/.cache/miopen
Volume=/data/docker/demucs/media:/app/media

[Service]
Restart=always

[Install]
WantedBy=default.target

4. Day-to-Day Operations: Status, Logs & Lifecycle

Important Execution Context: Rootless Podman containers belong to docker-data (UID 1100), not root or luke. Running bare sudo podman ps inspects root's empty store, not docker-data's store. Always use sudo systemctl --user -M docker-data@ ... for service management, or set HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 from a directory readable by docker-data (such as cd /home/docker-data) when invoking podman directly.

4.1 Checking Service & Container Status

# 1. Check systemd unit status for a specific Quadlet service
$ sudo systemctl --user -M docker-data@ status vaultwarden.service

# 2. List all active Quadlet container units under docker-data
$ sudo systemctl --user -M docker-data@ list-units --type=service --state=running

# 3. List all running rootless Podman containers, ports, and uptime
$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman ps -a)

# 4. View live CPU, memory, and I/O resource usage across all rootless containers
$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman stats --no-stream)

# 5. Inspect processes running inside a container (showing container UID vs host UID)
$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman top vaultwarden user uid huser huid pid comm)

4.2 Viewing Logs

Because Quadlet services integrate directly with systemd (--sdnotify=conmon), container output is available through both journalctl and podman logs:

# Follow live systemd journal logs for a service (includes systemd start/stop events + container stdout/stderr)
$ sudo journalctl --user -M docker-data@ -u navidrome.service -f

# View the last 100 lines of systemd journal logs
$ sudo journalctl --user -M docker-data@ -u demucs.service -n 100 --no-pager

# View container stdout/stderr directly via podman logs
$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman logs --tail 100 -f sopro)

4.3 Starting, Stopping, Restarting & Editing Quadlets

# Restart a service
$ sudo systemctl --user -M docker-data@ restart vaultwarden.service

# Stop or start a service
$ sudo systemctl --user -M docker-data@ stop drawpile.service
$ sudo systemctl --user -M docker-data@ start drawpile.service

# Apply edits made to /data/docker/<service>/<service>.container
$ sudo systemctl --user -M docker-data@ daemon-reload
$ sudo systemctl --user -M docker-data@ restart <service>.service

4.4 Updating Images & Rebuilding Local Dockerfiles

• Updating a Registry-Backed Image (vaultwarden, navidrome, bentopdf, etc.):

  $ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull docker.io/vaultwarden/server:latest)
  $ sudo systemctl --user -M docker-data@ restart vaultwarden.service
  

• Rebuilding a Local Dockerfile Image (sopro, demucs): Use TMPDIR=/dev/shm so large multi-gigabyte layer builds (demucs ROCm is ~16.4 GB) stage in RAM rather than disk:

  $ (cd /data/docker/demucs && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 TMPDIR=/dev/shm podman build -t localhost/demucs:latest /data/docker/demucs)
  $ sudo systemctl --user -M docker-data@ restart demucs.service
  

• Pruning Unused/Dangling Images:

  $ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman image prune -f)
  

4.5 Onboarding a New Standalone Service

1. Assign a dedicated ULA IPv6 address on eth77 (fd77:7777:5010:0:1337:f1ac:d00d:<hex>/64) in /etc/systemd/network/ and bind it live with sudo ip -6 addr add ... dev eth77.
2. Create /data/docker/<service>/<service>.container with Network=bridge and PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:<hex>]:<host_port>:<container_port>.
3. Set ownership and permissions:

   $ sudo chown -R docker-data:docker-data /data/docker/<service>
   $ sudo chmod 755 /data/docker/<service>
   $ sudo -u docker-data ln -sf /data/docker/<service>/<service>.container /home/docker-data/.config/containers/systemd/<service>.container
   

4. Reload and start:

   $ sudo systemctl --user -M docker-data@ daemon-reload
   $ sudo systemctl --user -M docker-data@ start <service>.service
   

5. Create /home/luke/git/backups/ul/scripts/backup/<service> and /home/luke/git/backups/ul/scripts/restore/<service> archiving <service>.container and any data volumes, and verify with shellcheck.

5. Troubleshooting & Diagnostics

5.1 Symptom: Local curl or Nginx Times Out / Resets on [fd77:...]:PORT

• Cause:
  1. If Network=pasta (the default when Network= is omitted) is used without isolated -a addresses, pasta copies all 36 host ULA IPv6 addresses from eth77 into the container namespace, causing replies to local connections from the host's own ULA IPv6 address to loop back inside the container.
  2. Even with Network=pasta:-a,10.0.2.100,-a,fd00::100, pasta forwards connections arriving on an IPv6 host socket as IPv6 packets inside the container. If the container application only listens on IPv4 0.0.0.0:<port> (such as tldraw, sopro, demucs, or drawpile), the container kernel sends TCP RST (curl: (56) Recv failure: Connection reset by peer).
• Fix: Ensure the .container file specifies Network=bridge, then run sudo systemctl --user -M docker-data@ daemon-reload && sudo systemctl --user -M docker-data@ restart <service>.service. With Network=bridge, rootlessport terminates the IPv6 connection on the host and connects over IPv4 (10.88.0.x) inside the container.

5.2 Symptom: Container Fails to Start with Permission denied on Volumes or Refuses UID 0

• Case 1 — Application refuses to run as root (getuid() == 0) inside the container (e.g., drawpile-srv): Under default rootless Podman, User=0:0 maps to UID 1100 on the host, so the process is unprivileged on the host—but inside the container getuid() returns 0. If a binary explicitly checks getuid() == 0 and exits, add both UserNS=keep-id and User=1100:1100 to [Container].
• Case 2 — Multi-process container (nginx, apache2) starts as root (0 -> 1100) and drops worker privileges to UID 101 (624388) or UID 33 (624320): Because worker processes run as host UID 624388 or 624320 (not 1100), they access bind-mounted directories via the other permission bits. Ensure the parent directory /data/docker/<service> has chmod 755 (drwxr-xr-x) and read-only content directories (html, content) have world-readable traversal (chmod -R a+rX).
• Case 3 — Inspecting host-mapped UIDs inside a container: Run podman unshare as docker-data to view filesystem ownership from inside the container's user namespace:

  $ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman unshare ls -lan /data/docker/vaultwarden/vw-data)
  

5.3 Symptom: Error: chdir("/home/luke"): Permission denied When Running sudo -u docker-data podman

• Cause: When you run sudo -u docker-data ... while your current working directory is /home/luke (0700 luke:luke), podman or its OCI runtime attempts to stat()/chdir() the inherited working directory and fails with Permission denied.
• Fix: Always change directory to /home/docker-data (or /tmp) in a subshell and pass HOME and XDG_RUNTIME_DIR:

  $ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman ps)
  

5.4 Symptom: Quadlet .container File Edits Not Applied or Service Not Found

• Diagnostic Command: Run the Quadlet systemd generator in --dry-run mode as docker-data to inspect any syntax or parsing errors in /home/docker-data/.config/containers/systemd/*.container:

  $ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 /usr/lib/systemd/user-generators/podman-system-generator --user --dry-run)
  

• Checklist:
  1. Confirm /home/docker-data/.config/containers/systemd/<service>.container is a valid symlink to /data/docker/<service>/<service>.container and that /data/docker/<service> is readable by docker-data.
  2. Run sudo systemctl --user -M docker-data@ daemon-reload.
  3. If the service previously hit start-limit-hit, clear the failed state before starting:

     $ sudo systemctl --user -M docker-data@ reset-failed <service>.service
     $ sudo systemctl --user -M docker-data@ restart <service>.service
     

5.5 Symptom: ROCm GPU (torch.cuda.is_available() == False) Inside Rootless Container

• Checklist:
  1. Verify docker-data is a member of video (91) and render (991) on the host (id docker-data).
  2. Verify the .container file includes all three directives under [Container]:

     AddDevice=/dev/dri:/dev/dri
     AddDevice=/dev/kfd:/dev/kfd
     GroupAdd=keep-groups
     

  3. Test GPU detection live inside demucs:

     $ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec demucs python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))")
     True AMD Radeon AI PRO R9700
     

5.6 Full Audit Script: Verify Host UIDs, User Namespaces & HTTP Health

Run the following one-liner on ul to audit every rootless Podman service's host kernel UID, user namespace isolation, and ULA IPv6 HTTP response:

$ (cd /home/docker-data && init_ns=$(sudo readlink /proc/1/ns/user) && echo "Init UserNS: $init_ns" && \
  for c in audiomass bentopdf excalidraw tldraw wick webrcade drawpile navidrome vaultwarden sopro demucs; do \
    pid=$(sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman inspect "$c" --format "{{.State.Pid}}"); \
    uid=$(awk "/^Uid:/ {print \$2}" "/proc/$pid/status"); \
    ns=$(sudo readlink "/proc/$pid/ns/user"); \
    printf "%-12s PID=%-8s HostUID=%-8s UserNS=%s\n" "$c" "$pid" "$uid" "$ns"; \
  done)