podman

Overview: All 30 containerized services (4041 containers total) on ul run exclusively under rootless Podman (podman 6.1.4) managed as native systemd --user services via Podman Quadlets under the dedicated unprivileged account docker-data (UID 1100, GID 1100). Rootful Docker (docker.service, docker.socket, and containerd.service) is completely disabled and decommissioned, eliminating the privileged root container daemon attack surface while preserving per-service ULA IPv6 reverse-proxy isolation on eth77.

1. Architecture & Security Comparison: Rootless Podman vs. Rootful Docker

Under traditional Docker, dockerd and containerd execute permanently as root (UID 0) inside the host init user namespace (user:[4026531837]). Every container spawned by dockerd without user-namespace remapping shares kernel UID 0 whenever the container process runs as root, mutates host iptables/nftables chains directly as root, and requires root socket access (/var/run/docker.sock) for orchestration or maintenance timers.

Under rootless Podman on ul, there is no root container daemon anywhere on the host:

• systemd starts an unprivileged user manager (user@1100.service via lingering) for docker-data.
• Each Quadlet .container unit directly spawns lightweight conmon and crun processes inside an unprivileged Linux user namespace (user:[4026534031]).
• Shared stateful databases (PostgreSQL on 127.0.0.1:5432 and Valkey on 127.0.0.1:6379) run natively on the host locked strictly to loopback, with zero LAN or Docker bridge exposure.

1.1 User Namespace UID/GID Translation Model

Rootless Podman translates container UIDs and GIDs into unprivileged host IDs using /etc/subuid and /etc/subgid (docker-data:624288:65536). Under the default rootless mapping, any container UID $u \in [0, 65536]$ maps deterministically to a host kernel UID:

$$ \text{UID}_{\text{host}}(u) = \begin{cases} 1100 & \text{if } u = 0 \quad (\text{container } \texttt{root} \longrightarrow \text{host } \texttt{docker-data}) \cr 624287 + u & \text{if } 1 \le u \le 65536 \quad (\text{subordinate range } 624288\text{--}689823) \end{cases} $$

Common live worker translations visible in ps / podman-map-tree on ul:

• Container root (u = 0) $\longrightarrow$ Host UID 1100 (docker-data) with zero host capabilities.
• Container www-data (u = 33, e.g., nextcloud, yourls, webrcade) $\longrightarrow$ Host UID 624320.
• Container nginx (u = 101, e.g., audiomass, bentopdf, excalidraw, penpot-frontend, tldraw, wick) $\longrightarrow$ Host UID 624388.
• Container postgres (u = 999, e.g., penpot-postgres) $\longrightarrow$ Host UID 625286.
• Container authentik / degoog (u = 1000) $\longrightarrow$ Host UID 625287.
• Container penpot (u = 1001) $\longrightarrow$ Host UID 625288.

When a binary explicitly refuses to start if internal getuid() == 0 (specifically drawpile-srv), setting UserNS=keep-id with User=1100:1100 preserves UID 1100 across the namespace boundary:

$$ \text{UID}_{\text{keep-id}}(1100) \stackrel{\texttt{userns}}{\longleftrightarrow} 1100 \quad (\texttt{docker-data}) $$

Why Default Rootless UserNS Beats UserNS=keep-id for Large ROCm Images: Avoid setting UserNS=keep-id unless an application explicitly aborts when getuid() == 0. On multi-gigabyte images such as immich-machine-learning:v3.3.1-rocm (23.3 GB) or subocr, UserNS=keep-id triggers a full recursive ownership copy-up across thousands of Python/ROCm files that stalls startup for over 5 minutes. Under the default rootless user namespace, container UID 0 already maps directly to host docker-data (1100:1100) with zero layer copy-up and sub-2-second startup.

1.2 Side-by-Side Comparison

2. Host Configuration & Intra-Service Networking on ul

2.1 Unprivileged Account, Subordinate IDs & Systemd Lingering

1. Service Account (docker-data):
   • Primary UID/GID: 1100:1100 (docker-data), home /home/docker-data.
   • Supplementary Groups: video (91) and render (991) for AMD Radeon AI PRO R9700 (gfx1201) ROCm compute access (/dev/dri and /dev/kfd).
2. Subordinate UID/GID Ranges (/etc/subuid & /etc/subgid):

   docker-data:624288:65536
   

3. Systemd Lingering & Automatic Maintenance:
   • Lingering is enabled at /var/lib/systemd/linger/docker-data (sudo loginctl enable-linger docker-data) so user@1100.service starts automatically at boot without an interactive login.
   • Weekly image pruning runs rootlessly via /etc/systemd/system/podman-prune.{service,timer} (User=docker-data, Sundays at 04:00 EDT/EST), replacing the retired docker-prune.timer.

2.2 Global Configuration Files & Storage Layout

• /etc/containers/containers.conf: Configures pasta (passt) as the default rootless network backend and maps the synthetic guest address 169.254.1.2 (host.containers.internal) cleanly to host 127.0.0.1 loopback:

  [network]
  default_rootless_network_cmd = "pasta"
  pasta_options = ["--map-host-loopback", "169.254.1.2"]
  

• /etc/containers/registries.conf:

  unqualified-search-registries = ["docker.io"]
  

• Storage Paths:
  • Persistent image/layer store (GraphRoot): /home/docker-data/.local/share/containers/storage (overlay driver owned by docker-data).
  • Ephemeral runtime state & DNS tables (RunRoot): /run/user/1100/containers (including /run/user/1100/containers/networks/aardvark-dns/).
  • Quadlet unit directory: /home/docker-data/.config/containers/systemd/ (symlinks to /data/docker/<service>/*.{container,network}).
  • Per-service persistent data & definitions: /data/docker/<service>/ on the ZFS storage pool.

2.3 The Four Intra-Service Networking Patterns on ul

All container traffic on ul follows four clean, deterministic patterns without shared mega-bridges like the retired infra_net:

1. Ingress (Nginx $\longrightarrow$ Container ULA IPv6 Socket): Host Nginx (UID 33 http) terminates public HTTPS (*.drheck.dev) and proxies upstream HTTP/WebSocket traffic to each service's dedicated /64 ULA IPv6 address on eth77:

   $$ \text{Client HTTPS} \longrightarrow \text{Host Nginx } (\texttt{UID 33}) \longrightarrow [\texttt{fd77:7777:5010:0:1337:f1ac:d00d:hex}]:P_{\text{host}} \longrightarrow \text{Rootless Container } (\texttt{UID 1100}) $$

2. Container $\longrightarrow$ Native Host Daemons (PostgreSQL, Valkey, Ollama) via 169.254.1.2: Native postgresql.service (127.0.0.1:5432), valkey.service (127.0.0.1:6379, per-service ACL users), and ollama.service (127.0.0.1:11434) listen exclusively on host loopback. Inside any rootless container, pasta translates 169.254.1.2 directly to host 127.0.0.1:

   $$ \text{Container App} \longrightarrow \texttt{169.254.1.2}:(5432, 6379, 11434) \stackrel{\texttt{pasta}}{\longrightarrow} \text{Host } \texttt{127.0.0.1}:(5432, 6379, 11434) $$

3. Container $\longrightarrow$ Host OIDC / WOPI / LDAPS Hairpin (AddHost=auth.drheck.dev:169.254.1.2): Inside a rootless pasta network namespace, connecting to an IP address owned by the host itself (10.0.0.100, 10.0.0.3, or a host fd77:... ULA address) loops back inside the container namespace (Connection refused) rather than hairpinning out to host Nginx or HAProxy. Every OIDC, WOPI, or LDAPS client container therefore declares:

   AddHost=auth.drheck.dev:169.254.1.2
   

   (plus AddHost=cloud.drheck.dev:169.254.1.2 and AddHost=office.drheck.dev:169.254.1.2 on Nextcloud + Collabora). This preserves full Let's Encrypt TLS SNI and hostname verification while routing cleanly over loopback to host Nginx (127.0.0.1:443) and HAProxy (127.0.0.1:636).

4. Multi-Container Private Stack Bridges (<service>.network + aardvark-dns) & Privileged Port Proxy (HAProxy):

   • Multi-container stacks (authentik, immich, penpot, nextcloud+collabora, bookstack, romm, yourls, wipstudio, webshell+step-ca) each have a dedicated isolated bridge (10.89.X.0/24) with container-name DNS served by /usr/lib/podman/aardvark-dns. Database sidecars (bookstack_db, romm-db, yourls-db, penpot-postgres, penpot-redis) publish zero host ports.
   • Because unprivileged UID 1100 does not bind privileged ports < 1024, authentik-ldap listens on [fd77:...:aaaa]:3389 (LDAP) and :6636 (LDAPS), and host HAProxy (/etc/haproxy/conf.d/ldap.cfg) performs zero-copy Layer-4 TCP passthrough from 127.0.0.1, 10.0.0.3, 10.0.0.100, and [fd77:...:aaaa] ports 389/636 to [fd77:...:aaaa]:3389/6636.

3. Complete Rootless Podman Service Inventory (30 Services / 40 Containers)

Reference Quadlet Examples

[Unit]
Description=Vikunja Task Manager (Rootless Podman)
After=network-online.target
Wants=network-online.target

[Container]
Pull=never
ContainerName=vikunja
Image=docker.io/vikunja/vikunja:latest
Network=vikunja.network
AddHost=auth.drheck.dev:169.254.1.2
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:71c0]:3456:3456
EnvironmentFile=/data/docker/vikunja/.env
Volume=/data/docker/vikunja/config.yml:/etc/vikunja/config.yml:ro
Volume=/data/docker/vikunja/files:/app/vikunja/files:rw
HealthCmd=["/app/vikunja/vikunja", "healthcheck"]
HealthInterval=30s
HealthRetries=3

[Service]
Restart=always
RestartSec=5
TimeoutStartSec=120

[Install]
WantedBy=default.target

[Unit]
Description=Immich ROCm Machine Learning Worker (Rootless Podman)
After=network-online.target
Wants=network-online.target

[Container]
Pull=never
ContainerName=immich_machine_learning
Image=ghcr.io/immich-app/immich-machine-learning:v3.3.1-rocm
Network=immich.network
AddDevice=/dev/dri:/dev/dri
AddDevice=/dev/kfd:/dev/kfd
GroupAdd=keep-groups
EnvironmentFile=/data/docker/immich/.env
Volume=/data/docker/immich/model-cache:/cache:rw

[Service]
Restart=always
RestartSec=5
TimeoutStartSec=180

[Install]
WantedBy=default.target

4. Day-to-Day Operations: Status, Logs & Lifecycle

Important Execution Context on Arch Linux: Rootless Podman containers run under docker-data (UID 1100), not root or luke. Two Arch Linux systemd/podman details matter in CLI commands:
1. Always run sudo -u docker-data from a world-traversable directory such as cd /tmp (otherwise podman tries to stat("/home/luke") on mode 0700 and warns about Permission denied).
2. On systemd 258+, systemctl --user -M docker-data@ ... works directly, whereas journalctl --user -M docker-data@ returns Connecting to a machine as non-root is not supported. Always read user unit journals via sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u <service>.service.

4.1 Checking Service, Container & Namespace Status

# 1. Host-wide health summary across all nginx, native systemd, and rootless Podman services
$ status_all
$ services

# 2. Live container-to-host UID, PID, and UserNS translation audit tree
$ sudo /usr/local/sbin/podman-map-tree

# 3. Check systemd unit status for any Quadlet container
$ sudo systemctl --user -M docker-data@ status authentik-server.service

# 4. List all running rootless Podman containers, networks, ports, and health states
$ (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman ps -a)

# 5. Live CPU, memory, and I/O telemetry across all 4041 containers
$ (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman stats --no-stream)

4.2 Viewing Service & Container Logs

# Follow live systemd user unit logs (includes systemd lifecycle + container stdout/stderr)
$ sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u authentik-server.service -f

# View the last 100 journal lines without paging
$ sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u immich_server.service -n 100 --no-pager

# Follow container stdout/stderr directly via podman logs
$ (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman logs --tail 100 -f bookstack)

4.3 Starting, Stopping, Restarting & Applying Quadlet Edits

# Restart a running Quadlet container
$ sudo systemctl --user -M docker-data@ restart vaultwarden.service

# Reload systemd generator after editing /data/docker/<service>/<service>.container
$ sudo systemctl --user -M docker-data@ daemon-reload
$ sudo systemctl --user -M docker-data@ restart <service>.service

4.4 Updating Images & Rebuilding Local Dockerfiles

Because every .container unit sets Pull=never for offline boot safety, image updates are always explicit:

# Pull updated registry image into docker-data's store and restart
$ (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull docker.io/vaultwarden/server:latest)
$ sudo systemctl --user -M docker-data@ restart vaultwarden.service

# Rebuild a locally built Dockerfile image (use TMPDIR=/dev/shm for large ROCm images)
$ (cd /data/docker/subocr && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 TMPDIR=/dev/shm podman build -t localhost/subocr:latest /data/docker/subocr)
$ sudo systemctl --user -M docker-data@ restart subocr.service

5. Troubleshooting & Diagnostics Runbook

5.1 Symptom: OIDC Login or Container LDAPS Fails with Connection refused to auth.drheck.dev

• Root Cause: Inside a rootless pasta network namespace, auth.drheck.dev resolves via DNS to host 10.0.0.100 or fd77:...:aaaa, which loops back inside the container's own network namespace where nothing listens on :443 or :636.
• Fix: Add AddHost=auth.drheck.dev:169.254.1.2 under [Container] in <service>.container, reload daemon-reload, and restart the service so OIDC and LDAPS connections route cleanly through --map-host-loopback to 127.0.0.1:443 / 127.0.0.1:636.

5.2 Symptom: Multi-Container Service Fails to Connect to Sibling Hostname (penpot-redis / bookstack_db) After Abrupt Kill

• Root Cause: If a container on a custom <service>.network bridge is killed abruptly without running netavark teardown, a stale container ID + dead IP line can remain ahead of the new container's IP in /run/user/1100/containers/networks/aardvark-dns/<net_name>. Clients that connect only to the first A record returned by DNS (such as Java's Lettuce client in penpot-backend) time out against the stale IP.
• Diagnosis & Fix:
  1. Inspect /run/user/1100/containers/networks/aardvark-dns/<net_name> against live container IDs from podman ps --no-trunc.
  2. Remove any line whose 64-character container ID is no longer running, then reload aardvark-dns without restarting healthy containers:

     $ sudo kill -HUP "$(cat /run/user/1100/containers/networks/aardvark-dns/aardvark.pid)"
     

5.3 Symptom: Permission denied Inside Non-Root Worker Container (UID 624320 / 625288) on /data/docker/<svc>/...

• Root Cause: Worker processes running as container UID $u > 0$ execute on the host as subordinate UID $624287 + u$ (outside group 1100). If the parent directory /data/docker/<svc> has mode 2770 (drwxrws---, missing other execute +x), subordinate UIDs cannot traverse /data/docker/<svc> to reach child bind mounts, and tools run on the host (like git pull) can accidentally write root-owned files over 625287-owned files.
• Fix:
  1. Keep parent directories traversable without allowing directory listing (sudo chmod 2771 /data/docker/<svc>, i.e., drwxrws--x).
  2. Execute inside-container file updates inside the container namespace (or via podman unshare), just as /usr/local/bin/degoog-update runs git fetch and bun install inside degoog.

5.4 Symptom: Distroless Container Marks unhealthy Despite Listening Normally (e.g., vikunja)

• Root Cause: Distroless images contain no /bin/sh, so string-form HealthCmd=/app/vikunja/vikunja healthcheck fails because Podman invokes /bin/sh -c unless HealthCmd is formatted as a JSON exec array.
• Fix: Write HealthCmd=["/app/vikunja/vikunja", "healthcheck"] as a JSON array in <service>.container.

5.5 Symptom: 502 Bad Gateway on Dual-Stack Service When Passing IPv6-Only Flag (e.g., ttyd -6 -i ::)

• Root Cause: Rootless port forwarding may connect into the container over IPv4 even when accepting connections on an eth77 ULA IPv6 socket. Passing IPv6-only bind flags inside the container rejects IPv4 loopback/bridge forwards.
• Fix: Let internal container daemons listen on default dual-stack (0.0.0.0 / ::) rather than forcing IPv6-only flags.

5.6 Validating Quadlet Syntax Before Restarting

$ (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 /usr/lib/systemd/user-generators/podman-system-generator --user --dry-run)