Service Update Process

This document outlines the standard operational procedures for updating all containerized and native services hosted on ul. All 30 containerized services (41 containers) run under rootless Podman Quadlets (docker-data, UID 1100); dockerd and containerd are retired. See podman for full architecture and troubleshooting details.

Authentik Update Process

Authentik runs as three rootless Podman Quadlet units (authentik-server.service, authentik-worker.service, and authentik-ldap.service) on authentik_net in /data/docker/authentik/, backed by native loopback PostgreSQL (169.254.1.2:5432) and native ACL-isolated Valkey (169.254.1.2:6379, database 0), with host HAProxy (/etc/haproxy/conf.d/ldap.cfg) forwarding ports 389/636 to :3389/:6636.

1. Determining When an Update is Available

Authentik follows Calendar Versioning (YYYY.M.P, e.g., 2026.8.3).

• Admin Web UI: Log in to auth.drheck.dev as an administrator and navigate to Admin Interface → System → Overview.

• GitHub Releases: Monitor the official Authentik Releases page for release tags, changelogs, and release notes.

• Current Installed Version: Check the running image tags on ul:

  # Check configured Quadlet image tags
  grep '^Image=' /data/docker/authentik/authentik-*.container
  
  # Check live running rootless containers
  (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman ps --filter name=authentik --format "table {{.Names}}\t{{.Image}}\t{{.Status}}")
  

• Read the Release Notes for every release between your current version and the target version.

Do not skip minor versions if you are multiple releases behind (e.g., upgrade one minor YYYY.M series at a time and allow database migrations in authentik-worker and authentik-server to finish before proceeding).

2. Update Procedure

Step 1: Perform Pre-Update Backup

Always create an atomic, PAR2-verified snapshot of the PostgreSQL database, configuration files, and .container Quadlet units first:

sudo /home/luke/git/backups/ul/scripts/backup/authentik

Creates /data/backups/services/current/authentik/<YYYY-MM-DD_HHMM.SS>/ containing authentik_db_backup.dump and authentik_files_backup.tar.xz with 10% PAR2 recovery parity.

Step 2: Pull New Container Images

Because all .container units set Pull=never for offline boot resilience, pull the new target images into docker-data's store before editing units:

NEW_VER="2026.8.3"
(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull "ghcr.io/goauthentik/server:${NEW_VER}")
(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull "ghcr.io/goauthentik/ldap:${NEW_VER}")

Step 3: Update Quadlet .container Image Tags

Update Image= in /data/docker/authentik/authentik-{server,worker,ldap}.container:

sudo sed -i "s|^Image=ghcr.io/goauthentik/server:.*|Image=ghcr.io/goauthentik/server:${NEW_VER}|" \
  /data/docker/authentik/authentik-server.container \
  /data/docker/authentik/authentik-worker.container
sudo sed -i "s|^Image=ghcr.io/goauthentik/ldap:.*|Image=ghcr.io/goauthentik/ldap:${NEW_VER}|" \
  /data/docker/authentik/authentik-ldap.container

Step 4: Reload Systemd User Manager & Restart Services

sudo systemctl --user -M docker-data@ daemon-reload
sudo systemctl --user -M docker-data@ restart authentik-worker.service authentik-server.service authentik-ldap.service

Step 5: Monitor Startup & Database Migrations

sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u authentik-worker.service -u authentik-server.service -f

3. After-Update Verification Checks

1. Container & Unit Health:

   (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman ps --filter name=authentik)
   curl -sk -o /dev/null -w "%{http_code}\n" https://auth.drheck.dev/-/health/live/
   

2. LDAPS Certificate & SSSD Online Check:

   openssl s_client -connect 10.0.0.100:636 -servername auth.drheck.dev </dev/null 2>&1 | grep "Verify return code"
   sudo sssctl domain-status goauthentik.io --online
   

   (If TLS certificate sync is needed at any time, run sudo /usr/local/bin/update-authentik-cert.sh.)
3. SSO Flow Check: Open https://auth.drheck.dev/ and test an OIDC or ForwardAuth app (e.g., https://scontrol.drheck.dev/ or https://pw.drheck.dev/).

4. Rollback Procedure (If Upgrade Fails)

# Restore latest backup (or pass an explicit YYYY-MM-DD_HHMM.SS timestamp)
sudo /home/luke/git/backups/ul/scripts/restore/authentik

The restore script verifies PAR2 parity archives, stops authentik-{ldap,server,worker}.service under docker-data, snapshots current files to /data/docker/authentik_rollback_<timestamp>/, restores .env, Quadlets, templates, and PostgreSQL schema/data, and restarts all three units.

Standard Rootless Podman Quadlet Update Workflow

Every containerized service in /data/docker/<service>/ follows a unified backup-first workflow under docker-data:

1. Pre-Update Backup

sudo /home/luke/git/backups/ul/scripts/backup/<service>

2. Updating Registry-Backed Services

• Rolling-Tag Services (audiomass, bentopdf, bookstack, collabora, degoog, drawpile, excalidraw, funkwhale, navidrome, penpot, romm, step-ca, tldraw, vaultwarden, vikunja, webrcade, wick, yourls):

  # Pull the new image(s) into docker-data's storage
  (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull <image_name>)
  
  # Restart the Quadlet service unit(s)
  sudo systemctl --user -M docker-data@ restart <service>.service
  

• Pinned-Tag Services (anchor, authentik, immich, nextcloud):
  1. Pull the new pinned image tag as docker-data.
  2. Update Image= in /data/docker/<service>/<unit>.container (and .env if mirrored there).
  3. Run sudo systemctl --user -M docker-data@ daemon-reload && sudo systemctl --user -M docker-data@ restart <unit>.service.

3. Rebuilding Locally Built Dockerfile Services

Use TMPDIR=/dev/shm so multi-gigabyte builds (demucs and subocr ROCm images) stage layers in RAM:

(cd /data/docker/<service> && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 TMPDIR=/dev/shm podman build -t localhost/<service>:latest /data/docker/<service>)
sudo systemctl --user -M docker-data@ restart <service>.service

Applies to: cam, demucs, grill, scontrol, sopro, subocr, webshell, and wipstudio (localhost/wipstudio:latest).

4. Service-Specific Update Notes

• Degoog (/data/docker/degoog, search.drheck.dev): Automated source/dependency updates run cleanly inside the container namespace via sudo systemctl start degoog-update.service (/usr/local/bin/degoog-update), preserving subordinate container UID 625287 ownership.
• Immich (/data/docker/immich, photos.drheck.dev): Pull both ghcr.io/immich-app/immich-server:vX.Y.Z and ghcr.io/immich-app/immich-machine-learning:vX.Y.Z-rocm, update immich_server.container and immich_machine_learning.container, reload, and restart both units. Verify via curl -sk https://photos.drheck.dev/api/server/ping.
• Nextcloud & Collabora (/data/docker/nextcloud, cloud.drheck.dev, office.drheck.dev): Upgrade nextcloud:<version>-apache one major release at a time, then run post-upgrade index maintenance:

  (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -u www-data nextcloud php occ status)
  (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -u www-data nextcloud php occ db:add-missing-indices)
  

• RomM (/data/docker/romm, romm.drheck.dev): Keep filesystem.structure (default: "roms/{platform}/{game}", firmware: "bios/{platform}") in /data/docker/romm/config/config.yml for RomM 5.3.0+.
• WIPStudio MinIO (/data/docker/wipstudio, wipstudio.drheck.dev): Upstream removed prebuilt Docker Hub tags; a local xz image archive is preserved at /data/docker/wipstudio/minio_image_backup.tar.xz (podman load -i /data/docker/wipstudio/minio_image_backup.tar.xz), and source updates build via /data/docker/wipstudio/Dockerfile.minio.

Native Systemd Services (postgresql, valkey, nginx, haproxy, forgejo, ollama, podman, openssh)

1. Run configuration backups first:

   sudo /home/luke/git/backups/ul/scripts/backup/systemd
   sudo /home/luke/git/backups/ul/scripts/backup/nginx
   sudo /home/luke/git/backups/ul/scripts/backup/forgejo
   sudo /home/luke/git/backups/ul/scripts/backup/valkey
   

2. Upgrade packages via pacman:

   sudo pacman -Sy --noconfirm podman netavark aardvark-dns passt nginx haproxy valkey forgejo ollama ollama-vulkan openssh
   

3. Restart updated host services and verify health:

   sudo systemctl restart nginx haproxy valkey forgejo ollama sshd
   systemctl is-active postgresql valkey nginx haproxy forgejo ollama sshd