Service Update Process

This document outlines the standard operational procedures for updating services hosted on ul. Follow these steps carefully to ensure clean migrations, data integrity, and verified service continuity.

Authentik Update Process

Authentik is deployed via Docker Compose in /data/docker/authentik/ backed by a host PostgreSQL database and Valkey (Redis) caching container on the infra_net bridge network.

1. Determining When an Update is Available

Authentik follows Calendar Versioning (YYYY.M.P, e.g., 2026.5.6).

Admin Web UI: Log in to auth.drheck.dev as an administrator and navigate to Admin Interface → System → Overview. A notification banner appears when a newer version is available.
GitHub Releases: Monitor the official Authentik Releases page for release tags, changelogs, and release notes.

Current Installed Version: Check the running version on the server:

# Check image tags in compose file
grep 'image:' /data/docker/authentik/docker-compose.yml

# Or check active container images
docker ps --filter "name=authentik" --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

Read the Release Notes for every release between your current version and the target version.

Do not skip minor versions if you are multiple releases behind (e.g., to upgrade from 2024.8.x to 2024.12.x, first upgrade to 2024.10.x, verify migrations complete, then upgrade to 2024.12.x).

2. Update Procedure

Step 1: Perform Pre-Update Backup

Always create an atomic, PAR2-verified snapshot of the PostgreSQL database and configuration files before modifying configuration or pulling new images:

sudo /home/luke/git/backups/ul/scripts/backup/authentik

This creates a timestamped archive in /data/backups/services/current/authentik/<YYYY-MM-DD_HHMM.SS>/ containing authentik_db_backup.dump and authentik_files_backup.tar.xz with 10% PAR2 recovery parity.

Step 2: Edit `docker-compose.yml`

Navigate to the Authentik directory and update the image tags for server, worker, and ldap to the new version:

cd /data/docker/authentik
sudo $EDITOR docker-compose.yml

Update the version tag across all three services:

services:
  server:
    image: ghcr.io/goauthentik/server:<NEW_VERSION>
    ...
  worker:
    image: ghcr.io/goauthentik/server:<NEW_VERSION>
    ...
  ldap:
    image: ghcr.io/goauthentik/ldap:<NEW_VERSION>
    ...

Step 3: Pull New Container Images

Download the updated Docker images:

cd /data/docker/authentik
sudo docker compose pull

Step 4: Recreate Containers & Apply Migrations

Recreate the containers with the new images in detached mode:

sudo docker compose up -d

Step 5: Monitor Startup & Database Migrations

Follow container logs to monitor automatic database migrations and ensure all workers initialize without errors:

sudo docker compose logs -f

(Press Ctrl+C once the server and worker report healthy startup status).

3. After-Update Verification Checks

Perform the following verification checks to ensure Authentik and dependent services are operating normally:

1. Container Health Status

Verify that authentik-server, authentik-worker, and authentik-ldap are running and reporting healthy:

sudo docker compose ps

2. Authentik Admin Interface & Outpost Health

Open https://auth.drheck.dev/ in your browser and sign in.
Go to Admin Interface → System → Overview:
- Verify the reported version matches the newly installed version.
- Verify Workers are active and task queues are processing.
- Check for any outstanding warnings, failed tasks, or blueprint sync errors.
Check Applications → Outposts:
- Verify both the Embedded Outpost and LDAP Outpost (authentik-ldap) show green/healthy connectivity status.

3. Authentication & Proxy Flow Verification

Forward Auth / Nginx Proxy Verification: Test authentication against an Authentik-protected endpoint (e.g., https://scontrol.drheck.dev/, https://bt.drheck.dev/, or https://drawpile.drheck.dev/) in a private/incognito window to verify authentication redirects and session tokens function properly.
LDAP Outpost Verification: Ensure LDAP-dependent services (if any) can authenticate over port 389/636.

4. Rollback Procedure (If Upgrade Fails)

If the upgrade encounters irrecoverable errors or breaking schema issues, roll back immediately using the automated restore script:

# Restore from the latest pre-upgrade backup
sudo /home/luke/git/backups/ul/scripts/restore/authentik

# Or restore an explicit backup timestamp
sudo /home/luke/git/backups/ul/scripts/restore/authentik YYYY-MM-DD_HHMM.SS

The restore script automatically:

Verifies PAR2 integrity of database dumps and configuration archives (auto-repairing if needed).
Gracefully stops Authentik containers (docker compose down).
Saves a rollback snapshot of current files to /data/docker/authentik_rollback_<timestamp>/.
Extracts original configuration files (.env, docker-compose.yml, templates, data).
Restores PostgreSQL database schema and data from the dump.
Restarts Authentik services with the previous configuration.

Standard Docker Compose Service Update Workflow

All containerized services under /data/docker/<service>/ follow a unified backup-first update workflow:

1. Pre-Update Backup

Run the service backup script to create a timestamped snapshot in /data/backups/services/current/<service>/ with 10% PAR2 redundancy:

sudo /home/luke/git/backups/ul/scripts/backup/<service>

2. Image Update & Container Recreation

Rolling-Tag Docker Compose Services (bookstack, collabora, degoog, penpot, romm, valkey in /data/docker/infra, yourls):
```
cd /data/docker/<service>
sudo docker compose pull
sudo docker compose up -d
```
Pinned-Tag Services (anchor, authentik, immich, nextcloud): Update the version tag in docker-compose.yml or .env, then pull and recreate:
```
cd /data/docker/<service>
sudo docker compose pull
sudo docker compose up -d
```

3. Service-Specific Update Notes

Immich (`/data/docker/immich`, `photos.drheck.dev`)

Update IMMICH_VERSION=vX.Y.Z in /data/docker/immich/.env (immich_server uses ${IMMICH_VERSION} and immich_machine_learning uses ${IMMICH_VERSION}-rocm).
Starting in v3.3.1, immich-machine-learning natively implements ROCm ShapePolicy (ocr_canvases) and migraphx_model_cache_dir, so legacy v3.1.0 ml-overrides bind mounts must not be used.
Verify health and version via curl -sk https://photos.drheck.dev/api/server/version.

Nextcloud & Collabora Online (`/data/docker/nextcloud`, `cloud.drheck.dev`, `office.drheck.dev`)

Upgrade nextcloud:<version>-apache one major release at a time.

After container recreation, run database index and maintenance checks:

sudo docker exec -u www-data nextcloud php occ status
sudo docker exec -u www-data nextcloud php occ db:add-missing-indices

Verify https://cloud.drheck.dev/status.php ("maintenance":false) and https://office.drheck.dev/hosting/discovery.

RomM (`/data/docker/romm`, `romm.drheck.dev`)

Starting in RomM 5.3.0+, legacy filesystem.roms_folder and filesystem.firmware_folder keys in /data/docker/romm/config/config.yml are retired and must use filesystem.structure:
```
filesystem:
  structure:
    default: "roms/{platform}/{game}"
    firmware: "bios/{platform}"
  skip_hash_calculation: false
```

WIPStudio MinIO (`/data/docker/wipstudio`, `wipstudio.drheck.dev`)

Starting with RELEASE.2025-10-15T17-29-55Z, upstream MinIO moved to source-only distribution. Build the updated image using /data/docker/wipstudio/Dockerfile.minio:
```
cd /data/docker/wipstudio
sudo docker compose build minio
sudo docker compose up -d minio
```

Locally Built Docker Compose Services (`subocr`, `webshell`)

SubOCR (/data/docker/subocr, subocr.drheck.dev): Update the base image tag FROM ghcr.io/immich-app/immich-machine-learning:vX.Y.Z-rocm in /data/docker/subocr/Dockerfile, then rebuild (yt-dlp and uv are updated during build):
```
cd /data/docker/subocr
sudo docker compose build
sudo docker compose up -d
```
Webshell (/data/docker/webshell, webshell.drheck.dev): Update the ttyd and step-cli release versions in /data/docker/webshell/Dockerfile, then rebuild:
```
cd /data/docker/webshell
sudo docker compose build webshell
sudo docker compose up -d webshell
```

Rootless Podman Quadlet Service Update Workflow

The 11 standalone single-container services (audiomass, bentopdf, excalidraw, tldraw, wick, webrcade, drawpile, navidrome, vaultwarden, sopro, and demucs) run as rootless Podman Quadlets under user docker-data (UID 1100). See the podman reference page for full architecture details.

1. Pre-Update Backup

sudo /home/luke/git/backups/ul/scripts/backup/<service>

2. Updating Registry-Backed Quadlet Services (`audiomass`, `bentopdf`, `drawpile`, `excalidraw`, `navidrome`, `tldraw`, `vaultwarden`, `webrcade`, `wick`)

(cd /home/docker-data && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman pull <image_name>)
sudo systemctl --user -M docker-data@ restart <service>.service

3. Rebuilding Locally Built Quadlet Services (`sopro`, `demucs`)

Sopro (/data/docker/sopro, sopro.drheck.dev):

(cd /data/docker/sopro && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 TMPDIR=/dev/shm podman build --no-cache -t localhost/sopro:latest /data/docker/sopro)
sudo systemctl --user -M docker-data@ restart sopro.service

Demucs ROCm (/data/docker/demucs, demucs.drheck.dev):

(cd /data/docker/demucs && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 TMPDIR=/dev/shm podman build -t localhost/demucs:latest /data/docker/demucs)
sudo systemctl --user -M docker-data@ restart demucs.service

Native Systemd Services (`forgejo`, `ollama`, `docker`, `podman`, `openssh`)

Run the systemd configuration backup:

sudo /home/luke/git/backups/ul/scripts/backup/systemd

Sync and upgrade packages via pacman:

sudo pacman -Sy --noconfirm forgejo ollama ollama-vulkan docker podman passt openssh

Restart and verify the service units (note: /etc/tmpfiles.d/forgejo.conf automatically preserves git:git ownership on /var/lib/forgejo across pacman hooks):
```
sudo systemctl restart forgejo ollama docker sshd
systemctl is-active forgejo ollama docker sshd
```

Service Update Process

Authentik Update Process

1. Determining When an Update is Available

2. Update Procedure

Step 1: Perform Pre-Update Backup

Step 2: Edit docker-compose.yml

Step 3: Pull New Container Images

Step 4: Recreate Containers & Apply Migrations

Step 5: Monitor Startup & Database Migrations

3. After-Update Verification Checks

1. Container Health Status

2. Authentik Admin Interface & Outpost Health

3. Authentication & Proxy Flow Verification

4. Rollback Procedure (If Upgrade Fails)

Standard Docker Compose Service Update Workflow

1. Pre-Update Backup

2. Image Update & Container Recreation

3. Service-Specific Update Notes

Immich (/data/docker/immich, photos.drheck.dev)

Nextcloud & Collabora Online (/data/docker/nextcloud, cloud.drheck.dev, office.drheck.dev)

RomM (/data/docker/romm, romm.drheck.dev)

WIPStudio MinIO (/data/docker/wipstudio, wipstudio.drheck.dev)

Locally Built Docker Compose Services (subocr, webshell)

Rootless Podman Quadlet Service Update Workflow

1. Pre-Update Backup

2. Updating Registry-Backed Quadlet Services (audiomass, bentopdf, drawpile, excalidraw, navidrome, tldraw, vaultwarden, webrcade, wick)

3. Rebuilding Locally Built Quadlet Services (sopro, demucs)

Native Systemd Services (forgejo, ollama, docker, podman, openssh)

Step 2: Edit `docker-compose.yml`

Immich (`/data/docker/immich`, `photos.drheck.dev`)

Nextcloud & Collabora Online (`/data/docker/nextcloud`, `cloud.drheck.dev`, `office.drheck.dev`)

RomM (`/data/docker/romm`, `romm.drheck.dev`)

WIPStudio MinIO (`/data/docker/wipstudio`, `wipstudio.drheck.dev`)

Locally Built Docker Compose Services (`subocr`, `webshell`)

2. Updating Registry-Backed Quadlet Services (`audiomass`, `bentopdf`, `drawpile`, `excalidraw`, `navidrome`, `tldraw`, `vaultwarden`, `webrcade`, `wick`)

3. Rebuilding Locally Built Quadlet Services (`sopro`, `demucs`)

Native Systemd Services (`forgejo`, `ollama`, `docker`, `podman`, `openssh`)