podman

Overview: StandaloneAll container30 workloadscontainerized services (40 containers total) on ul run exclusively under rootless Podman (podman 6.x1.4) managed as native systemd --user user services via Podman Quadlets under the dedicated unprivileged account docker-data (UID 1100, GID 1100). ThisRootful eliminatesDocker (docker.service, docker.socket, and containerd.service) is completely disabled and decommissioned, eliminating the privileged root container daemon attack surface while preserving the per-service ULA IPv6 reverse-proxy architectureisolation on eth77.

1. Architecture & Security Comparison: Rootless Podman vs. Rootful Docker

Under traditional Docker, dockerd runsand containerd execute permanently as root (UID 0) inside the host init user namespace (user:[4026531837]). Every container spawned by dockerd without user-namespace remapping shares thekernel host kernel's root UID 0 whenever the container process runs as root, mutates host iptables/nftables chains directly as root, and anyrequires vulnerabilityroot insocket theaccess daemon(/var/run/docker.sock) for orchestration or runtimemaintenance starts with full host privileges.timers.

Under rootless Podman,Podman on ul, there is no central root container daemon anywhere on the host.:

• systemd managesstarts aan per-unprivileged user service manager (user@1100.service) via lingering) for docker-data,.

1.1 User Namespace UID/GID Translation Model

Rootless Podman mapstranslates container UIDs and GIDs into unprivileged host IDs using /etc/subuid and /etc/subgid (docker-data:624288:65536). Under the default rootless mapping (UserNS=host within the rootless user namespace),mapping, any container UID $u \in [0, 65536]$ translatesmaps deterministically to a host kernel UID according to:UID:

$$ \text{UID}_{\text{host}}(u) = \begin{cases} 1100 & \text{if } u = 0 \quad (\text{container } \texttt{root} \longrightarrow \text{host } \texttt{docker-data}) \cr 624287 + u & \text{if } 1 \le u \le 65536 \quad (\text{subordinate range } 624288\text{--}689823) \end{cases} $$

Common live worker translations visible in ps / podman-map-tree on ul:

When a servicebinary explicitly sets UserNS=keep-id with User=1100:1100 (such as drawpile, whose binary refuses to start if internal getuid() == 0 (specifically drawpile-srv), thesetting identityUserNS=keep-id mappingwith User=1100:1100 preserves UID 1100 across the namespace boundary:

$$ \text{UID}_{\text{keep-id}}(1100) \stackrel{\texttt{userns}}{\longleftrightarrow} 1100 \quad (\texttt{docker-data}) $$

Why Default Rootless UserNS Beats UserNS=keep-id for Large ROCm Images: Avoid setting UserNS=keep-id unless an application explicitly aborts when getuid() == 0. On multi-gigabyte images such as immich-machine-learning:v3.3.1-rocm (23.3 GB) or subocr, UserNS=keep-id triggers a full recursive ownership copy-up across thousands of Python/ROCm files that stalls startup for over 5 minutes. Under the default rootless user namespace, container UID 0 already maps directly to host docker-data (1100:1100) with zero layer copy-up and sub-2-second startup.

1.2 Side-by-Side Comparison

2. Host Configuration & Intra-Service Networking on ul

2.1 Unprivileged Account, Subordinate IDs & Systemd Lingering

1. Service Account (docker-data):
   • Primary UID/GID: 1100:1100 (docker-data)
   ,
2. Home Directory:home /home/docker-data.
3. Supplementary Groups: video (91) and render (991) for AMD Radeon AI PRO R9700 (gfx1201) ROCm GPU devicecompute access (/dev/dri and /dev/kfd).
4. Subordinate UID/GID Ranges (/etc/subuid & /etc/subgid):

   docker-data:624288:65536
   

5. Systemd UserLingering Lingering& Automatic Maintenance: Enabled
   via
6. Lingering is enabled at /var/lib/systemd/linger/docker-data (sudo loginctl enable-linger docker-data) (creating /var/lib/systemd/linger/docker-data). This ensuresso user@1100.service starts automatically at system boot and keeps all rootless Podman Quadlet services running without requiring an activeinteractive loginlogin.

2.2 Global Configuration Files & Storage Layout

• /etc/containers/registries.containers.conf: Configures unqualifiedpasta image(passt) namesas the default rootless network backend and maps the synthetic guest address 169.254.1.2 (host.containers.internal) cleanly to resolvehost against127.0.0.1 Dockerloopback: Hub

  [network]
  bydefault_rootless_network_cmd default:= "pasta"
  pasta_options = ["--map-host-loopback", "169.254.1.2"]
  

unqualified-search-registries = ["docker.io"]

[network]
default_network_backend = "pasta"
pasta_options = ["-a", "10.0.2.100", "-a", "fd00::100"]

• Persistent image/layer store (GraphRoot): /home/docker-data/.local/share/containers/storage (native kernel overlay filesystemdriver onowned NVMeby /docker-data).
• Ephemeral runtime state & DNS tables (RunRoot): /run/user/1100/containers (including /run/user/1100/containers/networks/aardvark-dns/).
• Quadlet unit directory: /home/docker-data/.config/containers/systemd/ (contains symlinks pointing to /data/docker/<service>/*.{container,network}).

2.3 The Four Intra-Service Networking ArchitecturePatterns on ul: Why Network=bridge Is Used

EveryAll container servicetraffic on ul bindsfollows itsfour publishedclean, portdeterministic topatterns awithout dedicatedshared mega-bridges like the retired infra_net:

Ingress (Nginx $\longrightarrow$ Container ULA IPv6 addressSocket): on eth77 (fd77:7777:5010:0:1337:f1ac:d00d:<hex>/64), and hostHost Nginx (UID 33 http) reverse-terminates public HTTPS (*.drheck.dev) and proxies requestsupstream HTTP/WebSocket traffic to http://[fd77:...:<hex>]:<port>.

All rootless Quadlet definitions on ul specify Network=bridge rather than default pasta networking for two architectural reasons:

$$ \text{Client HTTPS} \longrightarrow \text{Host Nginx } (\texttt{UID 33}) \longrightarrow [\texttt{fd77:7777:5010:0:1337:f1ac:d00d:hex}]:P_{\text{host}} \longrightarrow \text{Rootless Container } (\texttt{UID 1100}) $$

Container $\longrightarrow$ Native Host Daemons (PostgreSQL, Valkey, Ollama) via 169.254.1.2: Native postgresql.service (127.0.0.1:5432), valkey.service (127.0.0.1:6379, per-service ACL users), and ollama.service (127.0.0.1:11434) listen exclusively on host loopback. Inside any rootless container, pasta translates 169.254.1.2 directly to host 127.0.0.1:

$$ \text{Container App} \longrightarrow \texttt{169.254.1.2}:(5432, 6379, 11434) \stackrel{\texttt{pasta}}{\longrightarrow} \text{Host } \texttt{127.0.0.1}:(5432, 6379, 11434) $$

Container $\longrightarrow$ Host OIDC / WOPI / LDAPS Hairpin (AddHost=auth.drheck.dev:169.254.1.2): Inside a rootless pasta network namespace, connecting to an IP address owned by the host intoitself (10.0.0.100, 10.0.0.3, or a host fd77:... ULA address) loops back inside the container namespace.namespace When(Connection refused) rather than hairpinning out to host Nginx connectsor fromHAProxy. theEvery sameOIDC, machineWOPI, or LDAPS client container therefore declares:

AddHost=auth.drheck.dev:169.254.1.2

(plus AddHost=cloud.drheck.dev:169.254.1.2 and AddHost=office.drheck.dev:169.254.1.2 on Nextcloud + Collabora). This preserves full Let's Encrypt TLS SNI and hostname verification while routing cleanly over loopback to host Nginx (src127.0.0.1:443) =and fd77:HAProxy (127.0.0.1:636).

Multi-Container Private Stack Bridges (<service>..network + aardvark-dns) & Privileged Port Proxy (HAProxy):<hex>

$$zero-copy \text{NginxLayer-4 }TCP (\texttt{http})passthrough \stackrel{\text{IPv6from TCP}}{\longrightarrow}127.0.0.1, 10.0.0.3, 10.0.0.100, and [\texttt{fd77:...:hex}]aaaa] ports 389/636 to [fd77:...:P_{\text{host}}aaaa]:3389/6636.

3. Quadlet Service Architecture & Migrated Inventory

3.1 How Quadlets Work

Instead of docker-compose.yml, each service directory /data/docker/<service>/ contains a declarative <service>.container file symlinked into /home/docker-data/.config/containers/systemd/<service>.container.

Whenever sudo systemctl --user -M docker-data@ daemon-reload runs (or on boot), /usr/lib/systemd/user-generators/podman-system-generator compiles each .container file into a transient systemd unit at /run/user/1100/systemd/generator/<service>.service.

3.2 MigratedComplete Rootless Podman Service Inventory

3.3 Reference Quadlet PatternsExamples

[Unit]
Description=VaultwardenVikunja Task Manager (Rootless Podman ContainerPodman)
After=network-online.target
Wants=network-online.target

[Container]
Pull=never
ContainerName=vaultwardenvikunja
Image=docker.io/vaultwarden/server:vikunja/vikunja:latest
Network=bridgevikunja.network
User=0:0AddHost=auth.drheck.dev:169.254.1.2
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:a117]71c0]:8222:803456:3456
EnvironmentFile=/data/docker/vaultwarden/vikunja/.env
Environment=TZ=America/New_York
Volume=/data/docker/vaultwarden/vw-data:vikunja/config.yml:/dataetc/vikunja/config.yml:ro
Volume=/data/docker/vikunja/files:/app/vikunja/files:rw
HealthCmd=["/app/vikunja/vikunja", "healthcheck"]
HealthInterval=30s
HealthRetries=3

[Service]
Restart=always
RestartSec=5
TimeoutStartSec=120

[Install]
WantedBy=default.target

[Unit]
Description=DrawpileImmich ROCm Machine Learning Worker (Rootless Podman ContainerPodman)
After=network-online.target
Wants=network-online.target

[Container]
Pull=never
ContainerName=drawpileimmich_machine_learning
Image=docker.ghcr.io/drawpile/drawpile-srv:2immich-app/immich-machine-learning:v3.3.1-rocm
Network=bridge
UserNS=keep-id
User=1100:1100
Exec=--port 27750 --websocket-port 27751 --web-admin-port 27752 --web-admin-access all --web-admin-auth admin:drawpile123 --sessions /data/sessions --database /data/sessions/drawpile.db
Volume=/data/docker/drawpile/sessions:/data/sessions
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:d7a0]:27750:27750
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:d7a0]:27751:27751
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:d7a0]:27752:27752

[Service]
Restart=always

[Install]
WantedBy=default.target

[Unit]
Description=Demucs ROCm Rootless Podman Container
After=network-online.target
Wants=network-online.target

[Container]
ContainerName=demucs
Image=localhost/demucs:latest
Network=bridgeimmich.network
AddDevice=/dev/dri:/dev/dri
AddDevice=/dev/kfd:/dev/kfd
GroupAdd=keep-groups
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:de3]:8000:8000
EnvironmentFile=/data/docker/demucs/immich/.env
Environment=TZ=America/New_York
Environment=TORCH_HOME=/root/.cache/torch
Environment=HF_HOME=/root/.cache/huggingface
Environment=MIOPEN_USER_DB_PATH=/root/.cache/miopen
Environment=MIOPEN_CUSTOM_CACHE_DIR=/root/.cache/miopen
Volume=/data/docker/demucs/app.py:immich/model-cache:/app/app.py:ro
Volume=/data/docker/demucs/static:/app/static:ro
Volume=/data/docker/demucs/cache/torch:/root/.cache/torch
Volume=/data/docker/demucs/cache/huggingface:/root/.cache/huggingface
Volume=/data/docker/demucs/cache/miopen:/root/.cache/miopen
Volume=/data/docker/demucs/media:/app/mediacache:rw

[Service]
Restart=always
RestartSec=5
TimeoutStartSec=180

[Install]
WantedBy=default.target

4. Day-to-Day Operations: Status, Logs & Lifecycle

Important Execution Context:Context on Arch Linux: Rootless Podman containers belongrun tounder docker-data (UID 1100), not root or luke. RunningTwo bareArch Linux systemd/podman details matter in CLI commands:
1. Always run sudo podman-u ps inspects root's empty store, not docker-data's store.from Alwaysa useworld-traversable directory such as sudocd /tmp (otherwise podman tries to stat("/home/luke") on mode 0700 and warns about Permission denied).
2. On systemd 258+, systemctl --user -M docker-data@ ... forworks servicedirectly, management, or setwhereas HOME=/home/journalctl --user -M docker-data@ returns Connecting to a machine as non-root is not supported. Always read user unit journals via sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u <service>.service from a directory readable by docker-data (such as cd /home/docker-data) when invoking podman directly..

4.1 Checking ServiceService, Container & ContainerNamespace Status

# 1. Host-wide health summary across all nginx, native systemd, and rootless Podman services
$ status_all
$ services

# 2. Live container-to-host UID, PID, and UserNS translation audit tree
$ sudo /usr/local/sbin/podman-map-tree

# 3. Check systemd unit status for a specificany Quadlet servicecontainer
$ sudo systemctl --user -M docker-data@ status vaultwarden.authentik-server.service

# 2. List all active Quadlet container units under docker-data
$ sudo systemctl --user -M docker-data@ list-units --type=service --state=running

# 3.4. List all running rootless Podman containers, networks, ports, and uptimehealth states
$ (cd /home/docker-datatmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman ps -a)

# 4.5. View liveLive CPU, memory, and I/O resource usagetelemetry across all rootless40 containers
$ (cd /home/docker-datatmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman stats --no-stream)
# 5. Inspect processes running inside a container (showing container UID vs host UID)
$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman top vaultwarden user uid huser huid pid comm)

4.2 Viewing Service & Container Logs

Because Quadlet services integrate directly with systemd (--sdnotify=conmon), container output is available through both journalctl and podman logs:

# Follow live systemd journaluser unit logs for a service (includes systemd start/stop eventslifecycle + container stdout/stderr)
$ sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -M docker-data@ -u navidrome.authentik-server.service -f

# View the last 100 journal lines ofwithout systemd journal logspaging
$ sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -M docker-data@ -u demucs.immich_server.service -n 100 --no-pager

# ViewFollow container stdout/stderr directly via podman logs
$ (cd /home/docker-datatmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman logs --tail 100 -f sopro)bookstack)

4.3 Starting, Stopping, Restarting & EditingApplying QuadletsQuadlet Edits

# Restart a servicerunning Quadlet container
$ sudo systemctl --user -M docker-data@ restart vaultwarden.service

# StopReload orsystemd startgenerator aafter service
$ sudo systemctl --user -M docker-data@ stop drawpile.service
$ sudo systemctl --user -M docker-data@ start drawpile.service

# Apply edits made toediting /data/docker/<service>/<service>.container
$ sudo systemctl --user -M docker-data@ daemon-reload
$ sudo systemctl --user -M docker-data@ restart <service>.service

4.4 Updating Images & Rebuilding Local Dockerfiles

Because

# Pull updated registry image into docker-data's store and restart
$ (cd /home/docker-datatmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull docker.io/vaultwarden/server:latest)
$ sudo systemctl --user -M docker-data@ restart vaultwarden.service

$ (cd /data/docker/demucssubocr && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 TMPDIR=/dev/shm podman build -t localhost/demucs:subocr:latest /data/docker/demucs)subocr)
$ sudo systemctl --user -M docker-data@ restart demucs.subocr.service

$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman image prune -f)

4.5 Onboarding a New Standalone Service

$ sudo chown -R docker-data:docker-data /data/docker/<service>
$ sudo chmod 755 /data/docker/<service>
$ sudo -u docker-data ln -sf /data/docker/<service>/<service>.container /home/docker-data/.config/containers/systemd/<service>.container

$ sudo systemctl --user -M docker-data@ daemon-reload
$ sudo systemctl --user -M docker-data@ start <service>.service

5. Troubleshooting & Diagnostics Runbook

5.1 Symptom: LocalOIDC curlLogin or NginxContainer TimesLDAPS OutFails / Resets onwith [fd77:...]:PORTConnection refused to auth.drheck.dev

• Root Cause:
  Inside
• Ifa Network=pasta (the default when Network= is omitted) is used without isolated -a addresses,rootless pasta copiesnetwork allnamespace, 36auth.drheck.dev resolves via DNS to host ULA10.0.0.100 IPv6 addresses fromor eth77fd77:...:aaaa, intowhich the container namespace, causing replies to local connections from the host's own ULA IPv6 address to looploops back inside the container.

5.2 Symptom: Multi-Container Service Fails to StartConnect withto Sibling Hostname (penpot-redis / bookstack_db) After Abrupt Kill

$ sudo kill -HUP "$(cat /run/user/1100/containers/networks/aardvark-dns/aardvark.pid)"

5.3 Symptom: Permission denied onInside VolumesNon-Root orWorker RefusesContainer (UID 0624320 / 625288) on /data/docker/<svc>/...

• CaseRoot 1Cause: —Worker Applicationprocesses refuses to runrunning as root (getuid() == 0) inside the container (e.g., drawpile-srv): Under default rootless Podman, User=0:0 maps to UID 1100$u > 0$ execute on the host, so the process is unprivileged on the host—but inside the container getuid() returns 0. If a binary explicitly checks getuid() == 0 and exits, add both UserNS=keep-id and User=1100:1100 to [Container].

$host (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman unshare ls -lan /data/docker/vaultwarden/vw-data)

5.3 Symptom:like Error:git chdir("/home/luke"pull): Permissioncan deniedaccidentally When Runningwrite sudoroot-owned -ufiles docker-data podman

inside $degoog.

5.4 Symptom: QuadletDistroless Container Marks .containerunhealthy FileDespite EditsListening NotNormally Applied(e.g., or Service Not Foundvikunja)

• DiagnosticRoot CommandCause: RunDistroless theimages Quadletcontain systemdno generator/bin/sh, so string-form HealthCmd=/app/vikunja/vikunja healthcheck fails because Podman invokes /bin/sh -c unless HealthCmd is formatted as a JSON exec array.

5.5 Symptom: 502 Bad Gateway on Dual-Stack Service When Passing IPv6-Only Flag (e.g., ttyd -6 -i ::)

5.6 Validating Quadlet Syntax Before Restarting

$ (cd /home/docker-datatmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 /usr/lib/systemd/user-generators/podman-system-generator --user --dry-run)

$ sudo systemctl --user -M docker-data@ reset-failed <service>.service
$ sudo systemctl --user -M docker-data@ restart <service>.service

5.5 Symptom: ROCm GPU (torch.cuda.is_available() == False) Inside Rootless Container

AddDevice=/dev/dri:/dev/dri
AddDevice=/dev/kfd:/dev/kfd
GroupAdd=keep-groups

$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec demucs python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))")
True AMD Radeon AI PRO R9700

5.6 Full Audit Script: Verify Host UIDs, User Namespaces & HTTP Health

Run the following one-liner on ul to audit every rootless Podman service's host kernel UID, user namespace isolation, and ULA IPv6 HTTP response:

$ (cd /home/docker-data && init_ns=$(sudo readlink /proc/1/ns/user) && echo "Init UserNS: $init_ns" && \
  for c in audiomass bentopdf excalidraw tldraw wick webrcade drawpile navidrome vaultwarden sopro demucs; do \
    pid=$(sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman inspect "$c" --format "{{.State.Pid}}"); \
    uid=$(awk "/^Uid:/ {print \$2}" "/proc/$pid/status"); \
    ns=$(sudo readlink "/proc/$pid/ns/user"); \
    printf "%-12s PID=%-8s HostUID=%-8s UserNS=%s\n" "$c" "$pid" "$uid" "$ns"; \
  done)