# podman

Overview: All 30 containerized services (41 containers total) on ul run exclusively under rootless Podman (podman 6.1.4) managed as native systemd --user services via Podman Quadlets under the dedicated unprivileged account docker-data (UID 1100, GID 1100). Rootful Docker (docker.service, docker.socket, and containerd.service) is completely disabled and decommissioned, eliminating the privileged root container daemon attack surface while preserving per-service ULA IPv6 reverse-proxy isolation on eth77.

--- ## 1. Architecture & Security Comparison: Rootless Podman vs. Rootful Docker Under traditional Docker, `dockerd` and `containerd` execute permanently as `root` (`UID 0`) inside the host init user namespace (`user:[4026531837]`). Every container spawned by `dockerd` without user-namespace remapping shares kernel `UID 0` whenever the container process runs as `root`, mutates host `iptables`/`nftables` chains directly as `root`, and requires root socket access (`/var/run/docker.sock`) for orchestration or maintenance timers. Under rootless Podman on `ul`, **there is no root container daemon anywhere on the host**: * `systemd` starts an unprivileged user manager (`user@1100.service` via lingering) for `docker-data`. * Each Quadlet `.container` unit directly spawns lightweight `conmon` and `crun` processes inside an unprivileged Linux user namespace (`user:[4026534031]`). * Shared stateful databases (**PostgreSQL** on `127.0.0.1:5432` and **Valkey** on `127.0.0.1:6379`) run natively on the host locked strictly to loopback, with zero LAN or Docker bridge exposure. ### 1.1 User Namespace UID/GID Translation Model Rootless Podman translates container UIDs and GIDs into unprivileged host IDs using `/etc/subuid` and `/etc/subgid` (`docker-data:624288:65536`). Under the default rootless mapping, any container UID $u \in [0, 65536]$ maps deterministically to a host kernel UID: $$ \text{UID}_{\text{host}}(u) = \begin{cases} 1100 & \text{if } u = 0 \quad (\text{container } \texttt{root} \longrightarrow \text{host } \texttt{docker-data}) \cr 624287 + u & \text{if } 1 \le u \le 65536 \quad (\text{subordinate range } 624288\text{--}689823) \end{cases} $$ Common live worker translations visible in `ps` / `podman-map-tree` on `ul`: * Container `root` (`u = 0`) $\longrightarrow$ Host `UID 1100` (`docker-data`) with zero host capabilities. * Container `www-data` (`u = 33`, e.g., `nextcloud`, `yourls`, `webrcade`) $\longrightarrow$ Host `UID 624320`. * Container `nginx` (`u = 101`, e.g., `audiomass`, `bentopdf`, `excalidraw`, `penpot-frontend`, `tldraw`, `wick`) $\longrightarrow$ Host `UID 624388`. * Container `postgres` (`u = 999`, e.g., `penpot-postgres`) $\longrightarrow$ Host `UID 625286`. * Container `authentik` / `degoog` (`u = 1000`) $\longrightarrow$ Host `UID 625287`. * Container `penpot` (`u = 1001`) $\longrightarrow$ Host `UID 625288`. When a binary explicitly refuses to start if internal `getuid() == 0` (specifically `drawpile-srv`), setting `UserNS=keep-id` with `User=1100:1100` preserves `UID 1100` across the namespace boundary: $$ \text{UID}_{\text{keep-id}}(1100) \stackrel{\texttt{userns}}{\longleftrightarrow} 1100 \quad (\texttt{docker-data}) $$

Why Default Rootless UserNS Beats UserNS=keep-id for Large ROCm Images: Avoid setting UserNS=keep-id unless an application explicitly aborts when getuid() == 0. On multi-gigabyte images such as immich-machine-learning:v3.3.1-rocm (23.3 GB) or subocr, UserNS=keep-id triggers a full recursive ownership copy-up across thousands of Python/ROCm files that stalls startup for over 5 minutes. Under the default rootless user namespace, container UID 0 already maps directly to host docker-data (1100:1100) with zero layer copy-up and sub-2-second startup.

### 1.2 Side-by-Side Comparison | Dimension | Legacy Rootful Docker (`dockerd`) | Current Rootless Podman (`docker-data` Quadlets) | | :--- | :--- | :--- | | **Daemon Model** | Persistent root daemons (`dockerd` + `containerd`) running as `UID 0` | Daemonless fork/exec (`user@1100` $\longrightarrow$ `conmon` $\longrightarrow$ `crun`) running as `UID 1100` | | **Host Kernel UID of Container `root`** | Host `UID 0` (`root` in init `user:[4026531837]`) | Host `UID 1100` (`docker-data` in isolated `user:[4026534031]`) | | **Host Kernel UID of Container Worker (`u > 0`)** | Same numeric host UID (`33` collides with host `http`, `999` collides with system accounts) | Isolated subordinate UID $624287 + u$ (`624288`--`689823`, zero host UID collisions) | | **Service Definition & Lifecycle** | `docker-compose.yml` managed by `dockerd` | Declarative `.container` + `.network` Quadlets in `/data/docker//` | | **Offline Boot Resilience** | Cached locally unless pull policy overridden | Explicit `Pull=never` on all 41 `.container` units; 100% WAN-outage-safe boot | | **Host Firewall Mutation** | Mutates host `iptables`/`nftables` chains as `root` | Zero host firewall mutation; unprivileged user sockets bind directly to `eth77` ULA IPs | | **Privileged Ports (`389` / `636` LDAP/LDAPS)** | Direct root port bind in container + mounted `/etc/nginx/certs` | Rootless `authentik-ldap` on `:3389`/`:6636` fronted by host HAProxy TCP passthrough (`/etc/haproxy/conf.d/ldap.cfg`); zero TLS key mounts in containers | | **Shared Host PostgreSQL & Valkey** | Exposed on Docker bridges (`172.16.0.0/12`) + LAN (`10.0.0.0/24`) | Locked strictly to `127.0.0.1` / `::1`; reached from containers via `pasta` `--map-host-loopback 169.254.1.2` | | **ROCm GPU Access (`/dev/kfd`, `/dev/dri`)** | `devices:` + `group_add` as `root` | `AddDevice=` + `GroupAdd=keep-groups` inheriting `docker-data`'s `video` (`91`) and `render` (`991`) groups | --- ## 2. Host Configuration & Intra-Service Networking on `ul` ### 2.1 Unprivileged Account, Subordinate IDs & Systemd Lingering 1. **Service Account (`docker-data`)**: * Primary UID/GID: `1100:1100` (`docker-data`), home `/home/docker-data`. * Supplementary Groups: `video` (`91`) and `render` (`991`) for AMD Radeon AI PRO R9700 (`gfx1201`) ROCm compute access (`/dev/dri` and `/dev/kfd`). 2. **Subordinate UID/GID Ranges (`/etc/subuid` & `/etc/subgid`)**: ```text docker-data:624288:65536 ``` 3. **Systemd Lingering & Automatic Maintenance**: * Lingering is enabled at `/var/lib/systemd/linger/docker-data` (`sudo loginctl enable-linger docker-data`) so `user@1100.service` starts automatically at boot without an interactive login. * Weekly image pruning runs rootlessly via `/etc/systemd/system/podman-prune.{service,timer}` (`User=docker-data`, Sundays at `04:00` EDT/EST), replacing the retired `docker-prune.timer`. ### 2.2 Global Configuration Files & Storage Layout * **`/etc/containers/containers.conf`**: Configures `pasta` (`passt`) as the default rootless network backend and maps the synthetic guest address `169.254.1.2` (`host.containers.internal`) cleanly to host `127.0.0.1` loopback: ```toml [network] default_rootless_network_cmd = "pasta" pasta_options = ["--map-host-loopback", "169.254.1.2"] ``` * **`/etc/containers/registries.conf`**: ```toml unqualified-search-registries = ["docker.io"] ``` * **Storage Paths**: * Persistent image/layer store (`GraphRoot`): `/home/docker-data/.local/share/containers/storage` (`overlay` driver owned by `docker-data`). * Ephemeral runtime state & DNS tables (`RunRoot`): `/run/user/1100/containers` (including `/run/user/1100/containers/networks/aardvark-dns/`). * Quadlet unit directory: `/home/docker-data/.config/containers/systemd/` (symlinks to `/data/docker//*.{container,network}`). * Per-service persistent data & definitions: `/data/docker//` on the ZFS storage pool. ### 2.3 The Four Intra-Service Networking Patterns on `ul` All container traffic on `ul` follows four clean, deterministic patterns without shared mega-bridges like the retired `infra_net`: 1. **Ingress (`Nginx` $\longrightarrow$ Container ULA IPv6 Socket)**: Host Nginx (`UID 33` `http`) terminates public HTTPS (`*.drheck.dev`) and proxies upstream HTTP/WebSocket traffic to each service's dedicated `/64` ULA IPv6 address on `eth77`: $$ \text{Client HTTPS} \longrightarrow \text{Host Nginx } (\texttt{UID 33}) \longrightarrow [\texttt{fd77:7777:5010:0:1337:f1ac:d00d:hex}]:P_{\text{host}} \longrightarrow \text{Rootless Container } (\texttt{UID 1100}) $$ 2. **Container $\longrightarrow$ Native Host Daemons (`PostgreSQL`, `Valkey`, `Ollama`) via `169.254.1.2`**: Native `postgresql.service` (`127.0.0.1:5432`), `valkey.service` (`127.0.0.1:6379`, per-service ACL users), and `ollama.service` (`127.0.0.1:11434`) listen exclusively on host loopback. Inside any rootless container, `pasta` translates `169.254.1.2` directly to host `127.0.0.1`: $$ \text{Container App} \longrightarrow \texttt{169.254.1.2}:(5432, 6379, 11434) \stackrel{\texttt{pasta}}{\longrightarrow} \text{Host } \texttt{127.0.0.1}:(5432, 6379, 11434) $$ 3. **Container $\longrightarrow$ Host OIDC / WOPI / LDAPS Hairpin (`AddHost=auth.drheck.dev:169.254.1.2`)**: Inside a rootless `pasta` network namespace, connecting to an IP address owned by the host itself (`10.0.0.100`, `10.0.0.3`, or a host `fd77:...` ULA address) loops back inside the container namespace (`Connection refused`) rather than hairpinning out to host Nginx or HAProxy. Every OIDC, WOPI, or LDAPS client container therefore declares: ```ini AddHost=auth.drheck.dev:169.254.1.2 ``` (plus `AddHost=cloud.drheck.dev:169.254.1.2` and `AddHost=office.drheck.dev:169.254.1.2` on Nextcloud + Collabora). This preserves full Let's Encrypt TLS SNI and hostname verification while routing cleanly over loopback to host Nginx (`127.0.0.1:443`) and HAProxy (`127.0.0.1:636`). 4. **Multi-Container Private Stack Bridges (`.network` + `aardvark-dns`) & Privileged Port Proxy (`HAProxy`)**: * Multi-container stacks (`authentik`, `immich`, `penpot`, `nextcloud`+`collabora`, `bookstack`, `romm`, `yourls`, `wipstudio`, `webshell`+`step-ca`) each have a dedicated isolated bridge (`10.89.X.0/24`) with container-name DNS served by `/usr/lib/podman/aardvark-dns`. Database sidecars (`bookstack_db`, `romm-db`, `yourls-db`, `penpot-postgres`, `penpot-redis`) publish **zero** host ports. * Because unprivileged `UID 1100` does not bind privileged ports `< 1024`, `authentik-ldap` listens on `[fd77:...:aaaa]:3389` (LDAP) and `:6636` (LDAPS), and host HAProxy (`/etc/haproxy/conf.d/ldap.cfg`) performs zero-copy Layer-4 TCP passthrough from `127.0.0.1`, `10.0.0.3`, `10.0.0.100`, and `[fd77:...:aaaa]` ports `389`/`636` to `[fd77:...:aaaa]:3389`/`6636`. --- ## 3. Complete Rootless Podman Service Inventory (30 Services / 40 Containers) | Service | Systemd Unit(s) (`docker-data@`) | Network | Published Host Binding(s) | Backing Store / Notes | | :--- | :--- | :--- | :--- | :--- | | **`anchor`** | `anchor.service` | `anchor_net` | `[fd77:...:c4a1]:3000 -> 3000` | Host PG (`169.254.1.2:5432`), OIDC | | **`audiomass`** | `audiomass.service` | `podman` | `[fd77:...:a0d1]:9004 -> 80` | Stateless WebAudio editor | | **`authentik`** | `authentik-server.service`
`authentik-worker.service`
`authentik-ldap.service` | `authentik_net` | `[fd77:...:aaaa]:9000,9443`
*(internal worker)*
`[fd77:...:aaaa]:3389,6636` (via HAProxy `:389,:636`) | Host PG + Host Valkey (`db 0`), zero host TLS key mounts | | **`bentopdf`** | `bentopdf.service` | `podman` | `[fd77:...:bdf]:8080 -> 8080` | Stateless PDF toolkit | | **`bookstack`** | `bookstack.service`
`bookstack_db.service` | `bookstack_net` | `[fd77:...:b00c]:6875 -> 80`
*(internal MariaDB 3306)* | MariaDB sidecar + OIDC | | **`cam`** | `cam.service` | `cam_net` | `[fd77:...:ca30]:8000 -> 8000`, `127.0.0.1:8089 -> 8000` | `/proc/net/arp:ro`, YOLOv8n + Reolink Baichuan/RTSP | | **`collabora`** | `collabora.service` | `nextcloud_net` | `[fd77:...:7a31]:9980 -> 9980` | Strict `cloud\.drheck\.dev` WOPI allowlist | | **`degoog`** | `degoog.service` | `degoog_net` | `[fd77:...:de90]:4444 -> 4444` | Host Valkey (`db 4`), updated via `/usr/local/bin/degoog-update` | | **`demucs`** | `demucs.service` | `podman` | `[fd77:...:de3]:8000 -> 8000` | ROCm GPU (`/dev/dri`, `/dev/kfd`, `GroupAdd=keep-groups`) | | **`drawpile`** | `drawpile.service` | `podman` | `[fd77:...:d7a0]:27750-27752` | `UserNS=keep-id`, `User=1100:1100`, SQLite sessions | | **`excalidraw`** | `excalidraw.service` | `podman` | `[fd77:...:e8ca]:9003 -> 80` | Stateless whiteboard | | **`funkwhale`** | `funkwhale.service` | `funkwhale_net` | `[fd77:...:f7a0]:8085 -> 80` | Embedded PG/Redis + LDAPS (`auth.drheck.dev:636` via `169.254.1.2`) | | **`grill`** | `grill.service` | `grill_net` | `[fd77:...:fb02]:8000`, `10.0.0.3:8000,8088 -> 8000` | SQLite + ESP32-C3 HTTP ingest | | **`immich`** | `immich_server.service`
`immich_machine_learning.service` | `immich_net` | `[fd77:...:2283]:2283 -> 2283`
*(internal ML worker)* | Host PG (`pgvecto.rs`) + Host Valkey (`db 1`) + ROCm GPU | | **`navidrome`** | `navidrome.service` | `podman` | `[fd77:...:5a70]:4533 -> 4533` | `/data/music/flac:ro`, SQLite | | **`nextcloud`** | `nextcloud.service` | `nextcloud_net` | `[fd77:...:7a31]:8081 -> 80` | Host PG (`nextcloud_db`) + Host Valkey (`db 2`) + OIDC + Collabora | | **`penpot`** | `penpot-frontend.service`
`penpot-backend.service`
`penpot-exporter.service`
`penpot-postgres.service`
`penpot-redis.service` | `penpot_net` | `[fd77:...:be49]:9001 -> 8080`
*(4 internal services)* | ZFS `/data/docker/penpot/{postgres_data,assets}` + OIDC | | **`romm`** | `romm.service`
`romm-db.service` | `romm_net` | `[fd77:...:7033]:8080 -> 8080`
*(internal MariaDB 3306)* | MariaDB 11 sidecar + `/data/games/roms` + OIDC | | **`scontrol`** | `scontrol.service` | `scontrol_net` | `[fd77:...:5c01]:8000 -> 8000` | `id_scontrol` SSH orchestration to `studio-kvm-ssh` | | **`sopro`** | `sopro.service` | `podman` | `[fd77:...:509]:7860 -> 7860` | Local TTS image (`localhost/sopro:latest`) | | **`step-ca`** | `step-ca.service` | `webshell_net` | `[fd77:...:ca]:9000 -> 9000` | Smallstep SSH & X.509 CA (used by `webshell` & `ipctl`) | | **`subocr`** | `subocr.service` | `subocr_net` | `[fd77:...:c70]:8000 -> 8000`, `127.0.0.1:8077 -> 8000` | ROCm MIGraphX GPU + Host Ollama (`169.254.1.2:11434`) | | **`tldraw`** | `tldraw.service` | `podman` | `[fd77:...:71d7]:9002 -> 80` | Stateless canvas | | **`vaultwarden`** | `vaultwarden.service` | `podman` | `[fd77:...:a117]:8222 -> 80` | SQLite + OIDC (`AddHost=auth.drheck.dev:169.254.1.2`) | | **`vikunja`** | `vikunja.service` | `vikunja_net` | `[fd77:...:71c0]:3456 -> 3456` | Host PG + Host Valkey (`db 3`) + OIDC; JSON `HealthCmd` | | **`webrcade`** | `webrcade.service` | `podman` | `[fd77:...:a1ca]:8080 -> 80` | `/data/docker/webrcade/content`, `/data/games:ro` | | **`webshell`** | `webshell.service` | `webshell_net` | `[fd77:...:771d]:7681 -> 7681` | Dual-stack `ttyd` + ephemeral 5m `step-ca` SSH certs | | **`wick`** | `wick.service` | `podman` | `[fd77:...:41c8]:9005 -> 80` | Static HTML5 studio on `nginx:alpine` | | **`wipstudio`** | `wipstudio.service`
`wipstudio-minio.service` | `wipstudio_net` | `[fd77:...:7190]:3000 -> 3000`
`[fd77:...:7190]:9000 -> 9000` | Host PG + MinIO S3 sidecar + OIDC (`DISABLE_SIGN_UP=true`) | | **`yourls`** | `yourls.service`
`yourls-db.service` | `yourls_net` | `[fd77:...:77]:8080 -> 8080`
*(internal MariaDB 3306)* | MariaDB 10.11 sidecar (`YOURLS_PRIVATE=true` + ForwardAuth on API) | ### Reference Quadlet Examples
Pattern A: OIDC + Host PostgreSQL/Valkey Service (e.g., /data/docker/vikunja/vikunja.container) ```ini [Unit] Description=Vikunja Task Manager (Rootless Podman) After=network-online.target Wants=network-online.target [Container] Pull=never ContainerName=vikunja Image=docker.io/vikunja/vikunja:latest Network=vikunja.network AddHost=auth.drheck.dev:169.254.1.2 PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:71c0]:3456:3456 EnvironmentFile=/data/docker/vikunja/.env Volume=/data/docker/vikunja/config.yml:/etc/vikunja/config.yml:ro Volume=/data/docker/vikunja/files:/app/vikunja/files:rw HealthCmd=["/app/vikunja/vikunja", "healthcheck"] HealthInterval=30s HealthRetries=3 [Service] Restart=always RestartSec=5 TimeoutStartSec=120 [Install] WantedBy=default.target ```
Pattern B: Multi-Container ROCm GPU Stack (e.g., /data/docker/immich/immich_machine_learning.container) ```ini [Unit] Description=Immich ROCm Machine Learning Worker (Rootless Podman) After=network-online.target Wants=network-online.target [Container] Pull=never ContainerName=immich_machine_learning Image=ghcr.io/immich-app/immich-machine-learning:v3.3.1-rocm Network=immich.network AddDevice=/dev/dri:/dev/dri AddDevice=/dev/kfd:/dev/kfd GroupAdd=keep-groups EnvironmentFile=/data/docker/immich/.env Volume=/data/docker/immich/model-cache:/cache:rw [Service] Restart=always RestartSec=5 TimeoutStartSec=180 [Install] WantedBy=default.target ```
--- ## 4. Day-to-Day Operations: Status, Logs & Lifecycle

Important Execution Context on Arch Linux: Rootless Podman containers run under docker-data (UID 1100), not root or luke. Two Arch Linux systemd/podman details matter in CLI commands:
1. Always run sudo -u docker-data from a world-traversable directory such as cd /tmp (otherwise podman tries to stat("/home/luke") on mode 0700 and warns about Permission denied).
2. On systemd 258+, systemctl --user -M docker-data@ ... works directly, whereas journalctl --user -M docker-data@ returns Connecting to a machine as non-root is not supported. Always read user unit journals via sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u <service>.service.

### 4.1 Checking Service, Container & Namespace Status ```shell # 1. Host-wide health summary across all nginx, native systemd, and rootless Podman services $ status_all $ services # 2. Live container-to-host UID, PID, and UserNS translation audit tree $ sudo /usr/local/sbin/podman-map-tree # 3. Check systemd unit status for any Quadlet container $ sudo systemctl --user -M docker-data@ status authentik-server.service # 4. List all running rootless Podman containers, networks, ports, and health states $ (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman ps -a) # 5. Live CPU, memory, and I/O telemetry across all 41 containers $ (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman stats --no-stream) ``` ### 4.2 Viewing Service & Container Logs ```shell # Follow live systemd user unit logs (includes systemd lifecycle + container stdout/stderr) $ sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u authentik-server.service -f # View the last 100 journal lines without paging $ sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u immich_server.service -n 100 --no-pager # Follow container stdout/stderr directly via podman logs $ (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman logs --tail 100 -f bookstack) ``` ### 4.3 Starting, Stopping, Restarting & Applying Quadlet Edits ```shell # Restart a running Quadlet container $ sudo systemctl --user -M docker-data@ restart vaultwarden.service # Reload systemd generator after editing /data/docker//.container $ sudo systemctl --user -M docker-data@ daemon-reload $ sudo systemctl --user -M docker-data@ restart .service ``` ### 4.4 Updating Images & Rebuilding Local Dockerfiles Because every `.container` unit sets `Pull=never` for offline boot safety, image updates are always explicit: ```shell # Pull updated registry image into docker-data's store and restart $ (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull docker.io/vaultwarden/server:latest) $ sudo systemctl --user -M docker-data@ restart vaultwarden.service # Rebuild a locally built Dockerfile image (use TMPDIR=/dev/shm for large ROCm images) $ (cd /data/docker/subocr && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 TMPDIR=/dev/shm podman build -t localhost/subocr:latest /data/docker/subocr) $ sudo systemctl --user -M docker-data@ restart subocr.service ``` --- ## 5. Troubleshooting & Diagnostics Runbook ### 5.1 Symptom: OIDC Login or Container LDAPS Fails with `Connection refused` to `auth.drheck.dev` * **Root Cause**: Inside a rootless `pasta` network namespace, `auth.drheck.dev` resolves via DNS to host `10.0.0.100` or `fd77:...:aaaa`, which loops back inside the container's own network namespace where nothing listens on `:443` or `:636`. * **Fix**: Add `AddHost=auth.drheck.dev:169.254.1.2` under `[Container]` in `.container`, reload `daemon-reload`, and restart the service so OIDC and LDAPS connections route cleanly through `--map-host-loopback` to `127.0.0.1:443` / `127.0.0.1:636`. ### 5.2 Symptom: Multi-Container Service Fails to Connect to Sibling Hostname (`penpot-redis` / `bookstack_db`) After Abrupt Kill * **Root Cause**: If a container on a custom `.network` bridge is killed abruptly without running `netavark teardown`, a stale container ID + dead IP line can remain ahead of the new container's IP in `/run/user/1100/containers/networks/aardvark-dns/`. Clients that connect only to the first `A` record returned by DNS (such as Java's Lettuce client in `penpot-backend`) time out against the stale IP. * **Diagnosis & Fix**: 1. Inspect `/run/user/1100/containers/networks/aardvark-dns/` against live container IDs from `podman ps --no-trunc`. 2. Remove any line whose 64-character container ID is no longer running, then reload `aardvark-dns` without restarting healthy containers: ```shell $ sudo kill -HUP "$(cat /run/user/1100/containers/networks/aardvark-dns/aardvark.pid)" ``` ### 5.3 Symptom: `Permission denied` Inside Non-Root Worker Container (`UID 624320` / `625288`) on `/data/docker//...` * **Root Cause**: Worker processes running as container UID $u > 0$ execute on the host as subordinate UID $624287 + u$ (outside group `1100`). If the parent directory `/data/docker/` has mode `2770` (`drwxrws---`, missing `other` execute `+x`), subordinate UIDs cannot traverse `/data/docker/` to reach child bind mounts, and tools run on the host (like `git pull`) can accidentally write `root`-owned files over `625287`-owned files. * **Fix**: 1. Keep parent directories traversable without allowing directory listing (`sudo chmod 2771 /data/docker/`, i.e., `drwxrws--x`). 2. Execute inside-container file updates inside the container namespace (or via `podman unshare`), just as `/usr/local/bin/degoog-update` runs `git fetch` and `bun install` inside `degoog`. ### 5.4 Symptom: Distroless Container Marks `unhealthy` Despite Listening Normally (e.g., `vikunja`) * **Root Cause**: Distroless images contain no `/bin/sh`, so string-form `HealthCmd=/app/vikunja/vikunja healthcheck` fails because Podman invokes `/bin/sh -c` unless `HealthCmd` is formatted as a JSON exec array. * **Fix**: Write `HealthCmd=["/app/vikunja/vikunja", "healthcheck"]` as a JSON array in `.container`. ### 5.5 Symptom: `502 Bad Gateway` on Dual-Stack Service When Passing IPv6-Only Flag (e.g., `ttyd -6 -i ::`) * **Root Cause**: Rootless port forwarding may connect into the container over IPv4 even when accepting connections on an `eth77` ULA IPv6 socket. Passing IPv6-only bind flags inside the container rejects IPv4 loopback/bridge forwards. * **Fix**: Let internal container daemons listen on default dual-stack (`0.0.0.0` / `::`) rather than forcing IPv6-only flags. ### 5.6 Validating Quadlet Syntax Before Restarting ```shell $ (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 /usr/lib/systemd/user-generators/podman-system-generator --user --dry-run) ```