Service Update Process

This document outlines the standard operational procedures for updating all containerized and native services hosted on ul. FollowAll these30 stepscontainerized carefullyservices to(40 ensurecontainers) cleanrun migrations,under rootless Podman Quadlets (docker-data, integrity,UID 1100); dockerd and verifiedcontainerd serviceare continuity.retired. See podman for full architecture and troubleshooting details.

---

Authentik Update Process

Authentik isruns deployedas viathree Dockerrootless ComposePodman Quadlet units (authentik-server.service, authentik-worker.service, and authentik-ldap.service) on authentik_net in /data/docker/authentik/, backed by anative hostloopback PostgreSQL database(169.254.1.2:5432) and native ACL-isolated Valkey (Redis)169.254.1.2:6379, caching container on thedatabase infra_net0), with host HAProxy (/etc/haproxy/conf.d/ldap.cfg) forwarding ports 389/636 bridgeto network.:3389/:6636.

1. Determining When an Update is Available

Authentik follows Calendar Versioning (YYYY.M.P, e.g., 2026.5.68.3).

• Admin Web UI: Log in to auth.drheck.dev as an administrator and navigate to Admin Interface → System → Overview. A notification banner appears when a newer version is available.

• GitHub Releases: Monitor the official Authentik Releases page for release tags, changelogs, and release notes.

• Current Installed Version: Check the running versionimage tags on the server:ul:

  # Check configured Quadlet image tags
  in compose file
  grep 'image:^Image=' /data/docker/authentik/docker-compose.ymlauthentik-*.container
  
  # OrCheck checklive activerunning containerrootless imagescontainers
  docker(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman ps --filter "name=authentik"authentik --format "table {{.Names}}\t{{.Image}}\t{{.Status}}")
  

• Read the Release Notes for every release between your current version and the target version.

Do not skip minor versions if you are multiple releases behind (e.g., to upgrade fromone minor 2024.8.xYYYY.M series at a time and allow database migrations in authentik-worker and authentik-server to 2024.12.x,finish firstbefore upgrade to 2024.10.x, verify migrations complete, then upgrade to 2024.12.x)proceeding).

---

2. Update Procedure

Step 1: Perform Pre-Update Backup

Always create an atomic, PAR2-verified snapshot of the PostgreSQL databasedatabase, configuration files, and configuration.container filesQuadlet beforeunits modifying configuration or pulling new images:first:

sudo /home/luke/git/backups/ul/scripts/backup/authentik

This creates a timestamped archive inCreates /data/backups/services/current/authentik/<YYYY-MM-DD_HHMM.SS>/ containing authentik_db_backup.dump and authentik_files_backup.tar.xz with 10% PAR2 recovery parity.

Step 2: Edit docker-compose.yml

Navigate to the Authentik directory and update the image tags for server, worker, and ldap to the new version:

cd /data/docker/authentik
sudo $EDITOR docker-compose.yml

Update the version tag across all three services:

services:
  server:
    image: ghcr.io/goauthentik/server:<NEW_VERSION>
    ...
  worker:
    image: ghcr.io/goauthentik/server:<NEW_VERSION>
    ...
  ldap:
    image: ghcr.io/goauthentik/ldap:<NEW_VERSION>
    ...

Step 3: Pull New Container Images

DownloadBecause all .container units set Pull=never for offline boot resilience, pull the updatednew Dockertarget images:images into docker-data's store before editing units:

NEW_VER="2026.8.3"
(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull "ghcr.io/goauthentik/server:${NEW_VER}")
(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull "ghcr.io/goauthentik/ldap:${NEW_VER}")

Step 3: Update Quadlet .container Image Tags

Update Image= in /data/docker/authentik/authentik-{server,worker,ldap}.container:

sudo sed -i "s|^Image=ghcr.io/goauthentik/server:.*|Image=ghcr.io/goauthentik/server:${NEW_VER}|" \
  /data/docker/authentikauthentik/authentik-server.container \
  /data/docker/authentik/authentik-worker.container
sudo dockersed compose-i pull"s|^Image=ghcr.io/goauthentik/ldap:.*|Image=ghcr.io/goauthentik/ldap:${NEW_VER}|" \
  /data/docker/authentik/authentik-ldap.container

Step 4: RecreateReload ContainersSystemd User Manager & ApplyRestart MigrationsServices

Recreate the containers with the new images in detached mode:

sudo docker compose upsystemctl -d-user -M docker-data@ daemon-reload
sudo systemctl --user -M docker-data@ restart authentik-worker.service authentik-server.service authentik-ldap.service

Step 5: Monitor Startup & Database Migrations

Follow container logs to monitor automatic database migrations and ensure all workers initialize without errors:

sudo docker-u composedocker-data logsXDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u authentik-worker.service -u authentik-server.service -f

(Press Ctrl+C once the server and worker report healthy startup status).

---

3. After-Update Verification Checks

Perform

the following verification checks to ensure Authentik and dependent services are operating normally:

1.

Container Health& StatusUnit

Verify that authentik-server, authentik-worker, and authentik-ldap are running and reporting healthy:

Health:

(cd /tmp && sudo docker-u composedocker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman ps 

--filter

2.name=authentik) Authentikcurl Admin-sk Interface-o &/dev/null Outpost-w Health

"%{http_code}\n"

Open https://auth.drheck.dev/-/health/live/ in your browser and sign in. Go to Admin Interface → System → Overview:

Verify the reported version matches the newly installed version. Verify Workers are active and task queues are processing. Check for any outstanding warnings, failed tasks, or blueprint sync errors. CheckLDAPS ApplicationsCertificate & SSSD Online Check: →

openssl Outposts:s_client -connect 10.0.0.100:636 -servername auth.drheck.dev </dev/null 2>&1 | grep "Verify return code"
sudo sssctl domain-status goauthentik.io --online

(If TLS certificate sync is needed at any time, run sudo /usr/local/bin/update-authentik-cert.sh.) VerifySSO bothFlow theCheck: EmbeddedOpen Outposthttps://auth.drheck.dev/ and LDAP Outpost (authentik-ldap) show green/healthy connectivity status.

3. Authentication & Proxy Flow Verification

Forward Auth / Nginx Proxy Verification: Test authentication againsttest an Authentik-protectedOIDC endpointor ForwardAuth app (e.g., https://scontrol.drheck.dev/, https://bt.drheck.dev/, or https://drawpile.pw.drheck.dev/) in a private/incognito window to verify authentication redirects and session tokens function properly.. LDAP Outpost Verification: Ensure LDAP-dependent services (if any) can authenticate over port 389/636.

---

4. Rollback Procedure (If Upgrade Fails)

If the upgrade encounters irrecoverable errors or breaking schema issues, roll back immediately using the automated restore script:

# Restore from the latest pre-upgradebackup backup(or pass an explicit YYYY-MM-DD_HHMM.SS timestamp)
sudo /home/luke/git/backups/ul/scripts/restore/authentik
# Or restore an explicit backup timestamp
sudo /home/luke/git/backups/ul/scripts/restore/authentik YYYY-MM-DD_HHMM.SS

The restore script automatically:

Verifiesverifies PAR2 integrityparity of database dumps and configuration archives (auto-repairing if needed). Gracefullyarchives, stops Authentikauthentik-{ldap,server,worker}.service containersunder (dockerdocker-data, compose down). Saves a rollback snapshot ofsnapshots current files to /data/docker/authentik_rollback_<timestamp>/., Extractsrestores original configuration files (.env, docker-compose.yml,Quadlets, templates, data). Restoresand PostgreSQL database schemaschema/data, and datarestarts fromall thethree dump. Restarts Authentik services with the previous configuration. units.

---

Standard DockerRootless ComposePodman ServiceQuadlet Update Workflow

AllEvery containerized servicesservice underin /data/docker/<service>/ followfollows a unified backup-first update workflow:

1. Pre-Update Backup

Run the service backup script to create a timestamped snapshot in /data/backups/services/current/<service>/ with 10% PAR2 redundancy:

sudo /home/luke/git/backups/ul/scripts/backup/<service>

2. Image Update & Container Recreation

Rolling-Tag Docker Compose Services (bookstack, collabora, degoog, penpot, romm, valkey in /data/docker/infra, yourls):

cd /data/docker/<service>
sudo docker compose pull
sudo docker compose up -d

Pinned-Tag Services (anchor, authentik, immich, nextcloud): Update the version tag in docker-compose.yml or .env, then pull and recreate:

cd /data/docker/<service>
sudo docker compose pull
sudo docker compose up -d

3. Service-Specific Update Notes

Immich (/data/docker/immich, photos.drheck.dev)

Update IMMICH_VERSION=vX.Y.Z in /data/docker/immich/.env (immich_server uses ${IMMICH_VERSION} and immich_machine_learning uses ${IMMICH_VERSION}-rocm). Starting in v3.3.1, immich-machine-learning natively implements ROCm ShapePolicy (ocr_canvases) and migraphx_model_cache_dir, so legacy v3.1.0 ml-overrides bind mounts must not be used. Verify health and version via curl -sk https://photos.drheck.dev/api/server/version.

Nextcloud & Collabora Online (/data/docker/nextcloud, cloud.drheck.dev, office.drheck.dev)

Upgrade nextcloud:<version>-apache one major release at a time. After container recreation, run database index and maintenance checks:

sudo docker exec -u www-data nextcloud php occ status
sudo docker exec -u www-data nextcloud php occ db:add-missing-indices

Verify https://cloud.drheck.dev/status.php ("maintenance":false) and https://office.drheck.dev/hosting/discovery.

RomM (/data/docker/romm, romm.drheck.dev)

Starting in RomM 5.3.0+, legacy filesystem.roms_folder and filesystem.firmware_folder keys in /data/docker/romm/config/config.yml are retired and must use filesystem.structure:

filesystem:
  structure:
    default: "roms/{platform}/{game}"
    firmware: "bios/{platform}"
  skip_hash_calculation: false

WIPStudio MinIO (/data/docker/wipstudio, wipstudio.drheck.dev)

Starting with RELEASE.2025-10-15T17-29-55Z, upstream MinIO moved to source-only distribution. Build the updated image using /data/docker/wipstudio/Dockerfile.minio:

cd /data/docker/wipstudio
sudo docker compose build minio
sudo docker compose up -d minio

Locally Built Docker Compose Services (subocr, webshell)

SubOCR (/data/docker/subocr, subocr.drheck.dev): Update the base image tag FROM ghcr.io/immich-app/immich-machine-learning:vX.Y.Z-rocm in /data/docker/subocr/Dockerfile, then rebuild (yt-dlp and uv are updated during build):

cd /data/docker/subocr
sudo docker compose build
sudo docker compose up -d

Webshell (/data/docker/webshell, webshell.drheck.dev): Update the ttyd and step-cli release versions in /data/docker/webshell/Dockerfile, then rebuild:

cd /data/docker/webshell
sudo docker compose build webshell
sudo docker compose up -d webshell

---

Rootless Podman Quadlet Service Update Workflow

The 11 standalone single-container services (audiomass, bentopdf, excalidraw, tldraw, wick, webrcade, drawpile, navidrome, vaultwarden, sopro, and demucs) run as rootless Podman Quadletsworkflow under user docker-data (UID 1100). See the podman reference page for full architecture details.:

1. Pre-Update Backup

sudo /home/luke/git/backups/ul/scripts/backup/<service>

2. Updating Registry-Backed QuadletServices

Rolling-Tag Services (audiomass, bentopdf, bookstack, collabora, degoog, drawpile, excalidraw, funkwhale, navidrome, penpot, romm, step-ca, tldraw, vaultwarden, vikunja, webrcade, wick, yourls):

# Pull the new image(s) into docker-data's storage
(cd /home/docker-datatmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull <image_name>)

# Restart the Quadlet service unit(s)
sudo systemctl --user -M docker-data@ restart <service>.service

3.

Rebuilding Locally Built QuadletPinned-Tag Services (soproanchor, demucsauthentik, immich, nextcloud):

• 1. SoproPull (the new pinned image tag as docker-data.

Update Image= in /data/docker/sopro<service>/<unit>.container, (and sopro.drheck.dev.env):

if mirrored there).

Run (cdsudo /data/docker/soprosystemctl --user -M docker-data@ daemon-reload && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 TMPDIR=/dev/shm podman build --no-cache -t localhost/sopro:latest /data/docker/sopro) sudo systemctl --user -M docker-data@ restart sopro.<unit>.service. Demucs

3. Rebuilding Locally Built Dockerfile Services

Use TMPDIR=/dev/shm so multi-gigabyte builds (demucs and subocr ROCm (/data/docker/demucs,images) demucs.drheck.dev):stage layers in RAM:

(cd /data/docker/demucs<service> && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 TMPDIR=/dev/shm podman build -t localhost/demucs:<service>:latest /data/docker/demucs)<service>)
sudo systemctl --user -M docker-data@ restart demucs.<service>.service

Applies to: cam, demucs, grill, scontrol, sopro, subocr, webshell, and wipstudio (localhost/wipstudio:latest).

4. Service-Specific Update Notes

Degoog (/data/docker/degoog, search.drheck.dev): Automated source/dependency updates run cleanly inside the container namespace via sudo systemctl start degoog-update.service (/usr/local/bin/degoog-update), preserving subordinate container UID 625287 ownership. Immich (/data/docker/immich, photos.drheck.dev): Pull both ghcr.io/immich-app/immich-server:vX.Y.Z and ghcr.io/immich-app/immich-machine-learning:vX.Y.Z-rocm, update immich_server.container and immich_machine_learning.container, reload, and restart both units. Verify via curl -sk https://photos.drheck.dev/api/server/ping. Nextcloud & Collabora (/data/docker/nextcloud, cloud.drheck.dev, office.drheck.dev): Upgrade nextcloud:<version>-apache one major release at a time, then run post-upgrade index maintenance:

(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -u www-data nextcloud php occ status)
(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -u www-data nextcloud php occ db:add-missing-indices)

RomM (/data/docker/romm, romm.drheck.dev): Keep filesystem.structure (default: "roms/{platform}/{game}", firmware: "bios/{platform}") in /data/docker/romm/config/config.yml for RomM 5.3.0+. WIPStudio MinIO (/data/docker/wipstudio, wipstudio.drheck.dev): Upstream removed prebuilt Docker Hub tags; a local xz image archive is preserved at /data/docker/wipstudio/minio_image_backup.tar.xz (podman load -i /data/docker/wipstudio/minio_image_backup.tar.xz), and source updates build via /data/docker/wipstudio/Dockerfile.minio.

---

Native Systemd Services (postgresql, valkey, nginx, haproxy, forgejo, ollama, docker, podman, openssh)

1. Run the systemd configuration backup:backups first:

   sudo /home/luke/git/backups/ul/scripts/backup/systemd
   sudo /home/luke/git/backups/ul/scripts/backup/nginx
   sudo /home/luke/git/backups/ul/scripts/backup/forgejo
   sudo /home/luke/git/backups/ul/scripts/backup/valkey
   

2. Sync and upgradeUpgrade packages via pacman:

   sudo pacman -Sy --noconfirm podman netavark aardvark-dns passt nginx haproxy valkey forgejo ollama ollama-vulkan docker podman passt openssh
   

3. Restart updated host services and verify the service units (note: /etc/tmpfiles.d/forgejo.conf automatically preserves git:git ownership on /var/lib/forgejo across pacman hooks):health:

   sudo systemctl restart nginx haproxy valkey forgejo ollama docker sshd
   systemctl is-active postgresql valkey nginx haproxy forgejo ollama docker sshd