podman

Overview: Standalone container workloads on ul run under rootless Podman (podman 6.x) managed as native systemd user services via Podman Quadlets under the dedicated unprivileged account docker-data (UID 1100, GID 1100). This eliminates the privileged root daemon attack surface while preserving the per-service ULA IPv6 reverse-proxy architecture on eth77.

1. Architecture & Security Comparison: Rootless Podman vs. Docker

Under traditional Docker, dockerd runs as root (UID 0) inside the host init user namespace (user:[4026531837]). Every container spawned by dockerd without user-namespace remapping shares the host kernel's root UID 0 whenever the container process runs as root, and any vulnerability in the daemon or runtime starts with full host privileges.

Under rootless Podman, there is no central root daemon. Instead, systemd manages a per-user service manager (user@1100.service) for docker-data, which spawns lightweight conmon and crun processes entirely inside an unprivileged Linux user namespace (user:[4026534031]).

1.1 User Namespace UID/GID Translation Model

Rootless Podman maps container UIDs and GIDs into unprivileged host IDs using /etc/subuid and /etc/subgid (docker-data:624288:65536). Under the default rootless mapping (UserNS=host within the rootless user namespace), any container UID $u \in [0, 65536]$ translates to a host kernel UID according to:

$$ \text{UID}_{\text{host}}(u) = \begin{cases} 1100 & \text{if } u = 0 \quad (\text{container } \texttt{root} \longrightarrow \text{host } \texttt{docker-data}) \cr 624287 + u & \text{if } 1 \le u \le 65536 \quad (\text{subordinate range } 624288\text{--}689823) \end{cases} $$

When a service explicitly sets UserNS=keep-id with User=1100:1100 (such as drawpile, whose binary refuses to start if internal getuid() == 0), the identity mapping preserves UID 1100 across the namespace boundary:

$$ \text{UID}{\text{container}}(1100) \stackrel{\texttt{keep-id}}{\longleftrightarrow} \text{UID}{\text{host}}(1100) $$

1.2 Side-by-Side Comparison

2. Host Configuration on ul

2.1 Unprivileged Account, Subordinate IDs & Systemd Lingering

docker-data:624288:65536

2.2 Global Configuration Files & Storage Layout

unqualified-search-registries = ["docker.io"]

[network]
default_network_backend = "pasta"
pasta_options = ["-a", "10.0.2.100", "-a", "fd00::100"]

2.3 Networking Architecture on ul: Why Network=bridge Is Used

Every container service on ul binds its published port to a dedicated ULA IPv6 address on eth77 (fd77:7777:5010:0:1337:f1ac:d00d:<hex>/64), and host Nginx (UID 33 http) reverse-proxies requests to http://[fd77:...:<hex>]:<port>.

All rootless Quadlet definitions on ul specify Network=bridge rather than default pasta networking for two architectural reasons:

$$ \text{Nginx } (\texttt{http}) \stackrel{\text{IPv6 TCP}}{\longrightarrow} [\texttt{fd77:...:hex}]:P_{\text{host}} \ (\texttt{rootlessport}) \stackrel{\text{IPv4 TCP}}{\longrightarrow} \texttt{10.88.0.x}:P_{\text{container}} $$

3. Quadlet Service Architecture & Migrated Inventory

3.1 How Quadlets Work

Instead of docker-compose.yml, each service directory /data/docker/<service>/ contains a declarative <service>.container file symlinked into /home/docker-data/.config/containers/systemd/<service>.container.

Whenever sudo systemctl --user -M docker-data@ daemon-reload runs (or on boot), /usr/lib/systemd/user-generators/podman-system-generator compiles each .container file into a transient systemd unit at /run/user/1100/systemd/generator/<service>.service.

3.2 Migrated Rootless Podman Service Inventory

3.3 Reference Quadlet Patterns

[Unit]
Description=Vaultwarden Rootless Podman Container
After=network-online.target
Wants=network-online.target

[Container]
ContainerName=vaultwarden
Image=docker.io/vaultwarden/server:latest
Network=bridge
User=0:0
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:a117]:8222:80
EnvironmentFile=/data/docker/vaultwarden/.env
Environment=TZ=America/New_York
Volume=/data/docker/vaultwarden/vw-data:/data

[Service]
Restart=always

[Install]
WantedBy=default.target

[Unit]
Description=Drawpile Rootless Podman Container
After=network-online.target
Wants=network-online.target

[Container]
ContainerName=drawpile
Image=docker.io/drawpile/drawpile-srv:2
Network=bridge
UserNS=keep-id
User=1100:1100
Exec=--port 27750 --websocket-port 27751 --web-admin-port 27752 --web-admin-access all --web-admin-auth admin:drawpile123 --sessions /data/sessions --database /data/sessions/drawpile.db
Volume=/data/docker/drawpile/sessions:/data/sessions
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:d7a0]:27750:27750
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:d7a0]:27751:27751
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:d7a0]:27752:27752

[Service]
Restart=always

[Install]
WantedBy=default.target

[Unit]
Description=Demucs ROCm Rootless Podman Container
After=network-online.target
Wants=network-online.target

[Container]
ContainerName=demucs
Image=localhost/demucs:latest
Network=bridge
AddDevice=/dev/dri:/dev/dri
AddDevice=/dev/kfd:/dev/kfd
GroupAdd=keep-groups
PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:de3]:8000:8000
EnvironmentFile=/data/docker/demucs/.env
Environment=TZ=America/New_York
Environment=TORCH_HOME=/root/.cache/torch
Environment=HF_HOME=/root/.cache/huggingface
Environment=MIOPEN_USER_DB_PATH=/root/.cache/miopen
Environment=MIOPEN_CUSTOM_CACHE_DIR=/root/.cache/miopen
Volume=/data/docker/demucs/app.py:/app/app.py:ro
Volume=/data/docker/demucs/static:/app/static:ro
Volume=/data/docker/demucs/cache/torch:/root/.cache/torch
Volume=/data/docker/demucs/cache/huggingface:/root/.cache/huggingface
Volume=/data/docker/demucs/cache/miopen:/root/.cache/miopen
Volume=/data/docker/demucs/media:/app/media

[Service]
Restart=always

[Install]
WantedBy=default.target

4. Day-to-Day Operations: Status, Logs & Lifecycle

Important Execution Context: Rootless Podman containers belong to docker-data (UID 1100), not root or luke. Running bare sudo podman ps inspects root's empty store, not docker-data's store. Always use sudo systemctl --user -M docker-data@ ... for service management, or set HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 from a directory readable by docker-data (such as cd /home/docker-data) when invoking podman directly.

4.1 Checking Service & Container Status

# 1. Check systemd unit status for a specific Quadlet service
$ sudo systemctl --user -M docker-data@ status vaultwarden.service

# 2. List all active Quadlet container units under docker-data
$ sudo systemctl --user -M docker-data@ list-units --type=service --state=running

# 3. List all running rootless Podman containers, ports, and uptime
$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman ps -a)

# 4. View live CPU, memory, and I/O resource usage across all rootless containers
$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman stats --no-stream)

# 5. Inspect processes running inside a container (showing container UID vs host UID)
$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman top vaultwarden user uid huser huid pid comm)

4.2 Viewing Logs

Because Quadlet services integrate directly with systemd (--sdnotify=conmon), container output is available through both journalctl and podman logs:

# Follow live systemd journal logs for a service (includes systemd start/stop events + container stdout/stderr)
$ sudo journalctl --user -M docker-data@ -u navidrome.service -f

# View the last 100 lines of systemd journal logs
$ sudo journalctl --user -M docker-data@ -u demucs.service -n 100 --no-pager

# View container stdout/stderr directly via podman logs
$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman logs --tail 100 -f sopro)

4.3 Starting, Stopping, Restarting & Editing Quadlets

# Restart a service
$ sudo systemctl --user -M docker-data@ restart vaultwarden.service

# Stop or start a service
$ sudo systemctl --user -M docker-data@ stop drawpile.service
$ sudo systemctl --user -M docker-data@ start drawpile.service

# Apply edits made to /data/docker/<service>/<service>.container
$ sudo systemctl --user -M docker-data@ daemon-reload
$ sudo systemctl --user -M docker-data@ restart <service>.service

4.4 Updating Images & Rebuilding Local Dockerfiles

$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull docker.io/vaultwarden/server:latest)
$ sudo systemctl --user -M docker-data@ restart vaultwarden.service

$ (cd /data/docker/demucs && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 TMPDIR=/dev/shm podman build -t localhost/demucs:latest /data/docker/demucs)
$ sudo systemctl --user -M docker-data@ restart demucs.service

$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman image prune -f)

4.5 Onboarding a New Standalone Service

$ sudo chown -R docker-data:docker-data /data/docker/<service>
$ sudo chmod 755 /data/docker/<service>
$ sudo -u docker-data ln -sf /data/docker/<service>/<service>.container /home/docker-data/.config/containers/systemd/<service>.container

$ sudo systemctl --user -M docker-data@ daemon-reload
$ sudo systemctl --user -M docker-data@ start <service>.service

5. Troubleshooting & Diagnostics

5.1 Symptom: Local curl or Nginx Times Out / Resets on [fd77:...]:PORT

5.2 Symptom: Container Fails to Start with Permission denied on Volumes or Refuses UID 0

$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman unshare ls -lan /data/docker/vaultwarden/vw-data)

5.3 Symptom: Error: chdir("/home/luke"): Permission denied When Running sudo -u docker-data podman

$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman ps)

5.4 Symptom: Quadlet .container File Edits Not Applied or Service Not Found

$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 /usr/lib/systemd/user-generators/podman-system-generator --user --dry-run)

$ sudo systemctl --user -M docker-data@ reset-failed <service>.service
$ sudo systemctl --user -M docker-data@ restart <service>.service

5.5 Symptom: ROCm GPU (torch.cuda.is_available() == False) Inside Rootless Container

AddDevice=/dev/dri:/dev/dri
AddDevice=/dev/kfd:/dev/kfd
GroupAdd=keep-groups

$ (cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec demucs python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))")
True AMD Radeon AI PRO R9700

5.6 Full Audit Script: Verify Host UIDs, User Namespaces & HTTP Health

Run the following one-liner on ul to audit every rootless Podman service's host kernel UID, user namespace isolation, and ULA IPv6 HTTP response:

$ (cd /home/docker-data && init_ns=$(sudo readlink /proc/1/ns/user) && echo "Init UserNS: $init_ns" && \
  for c in audiomass bentopdf excalidraw tldraw wick webrcade drawpile navidrome vaultwarden sopro demucs; do \
    pid=$(sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman inspect "$c" --format "{{.State.Pid}}"); \
    uid=$(awk "/^Uid:/ {print \$2}" "/proc/$pid/status"); \
    ns=$(sudo readlink "/proc/$pid/ns/user"); \
    printf "%-12s PID=%-8s HostUID=%-8s UserNS=%s\n" "$c" "$pid" "$uid" "$ns"; \
  done)