Skip to main content

Service Update Process

This document outlines the standard operational procedures for updating services hosted on ul. Follow these steps carefully to ensure clean migrations, data integrity, and verified service continuity.

Authentik Update Process

Authentik is deployed via Docker Compose in /data/docker/authentik/ backed by a host PostgreSQL database and Valkey (Redis) caching container on the infra_net bridge network.

1. Determining When an Update is Available

Authentik follows Calendar Versioning (YYYY.M.P, e.g., 2026.5.6).

  • Admin Web UI: Log in to auth.drheck.dev as an administrator and navigate to Admin Interface → System → Overview. A notification banner appears when a newer version is available.

  • GitHub Releases: Monitor the official Authentik Releases page for release tags, changelogs, and release notes.

  • Current Installed Version: Check the running version on the server:

    # Check image tags in compose file
grep 'image:' /data/docker/authentik/docker-compose.yml

# Or check active container images
docker ps --filter "name=authentik" --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

  • Read the Release Notes for every release between your current version and the target version.

Do not skip minor versions if you are multiple releases behind (e.g., to upgrade from 2024.8.x to 2024.12.x, first upgrade to 2024.10.x, verify migrations complete, then upgrade to 2024.12.x).

2. Update Procedure

Step 1: Perform Pre-Update Backup

Always create an atomic, PAR2-verified snapshot of the PostgreSQL database and configuration files before modifying configuration or pulling new images:

sudo /home/luke/git/backups/ul/scripts/backup/authentik

This creates a timestamped archive in /data/backups/services/current/authentik/<YYYY-MM-DD_HHMM.SS>/ containing authentik_db_backup.dump and authentik_files_backup.tar.xz with 10% PAR2 recovery parity.

Step 2: Edit docker-compose.yml

Navigate to the Authentik directory and update the image tags for server, worker, and ldap to the new version:

cd /data/docker/authentik
sudo $EDITOR docker-compose.yml

Update the version tag across all three services:

services:
  server:
    image: ghcr.io/goauthentik/server:<NEW_VERSION>
    ...
  worker:
    image: ghcr.io/goauthentik/server:<NEW_VERSION>
    ...
  ldap:
    image: ghcr.io/goauthentik/ldap:<NEW_VERSION>
    ...

Step 3: Pull New Container Images

Download the updated Docker images:

cd /data/docker/authentik
sudo docker compose pull

Step 4: Recreate Containers & Apply Migrations

Recreate the containers with the new images in detached mode:

sudo docker compose up -d

Step 5: Monitor Startup & Database Migrations

Follow container logs to monitor automatic database migrations and ensure all workers initialize without errors:

sudo docker compose logs -f

(Press Ctrl+C once the server and worker report healthy startup status).

3. After-Update Verification Checks

Perform the following verification checks to ensure Authentik and dependent services are operating normally:

1. Container Health Status

Verify that authentik-server, authentik-worker, and authentik-ldap are running and reporting healthy:

sudo docker compose ps

2. Authentik Admin Interface & Outpost Health

  1. Open https://auth.drheck.dev/ in your browser and sign in.
  2. Go to Admin Interface → System → Overview:
    • Verify the reported version matches the newly installed version.
    • Verify Workers are active and task queues are processing.
    • Check for any outstanding warnings, failed tasks, or blueprint sync errors.
  3. Check Applications → Outposts:
    • Verify both the Embedded Outpost and LDAP Outpost (authentik-ldap) show green/healthy connectivity status.

3. Authentication & Proxy Flow Verification

  • Forward Auth / Nginx Proxy Verification: Test authentication against an Authentik-protected endpoint (e.g., https://scontrol.drheck.dev/, https://bt.drheck.dev/, or https://drawpile.drheck.dev/) in a private/incognito window to verify authentication redirects and session tokens function properly.
  • LDAP Outpost Verification: Ensure LDAP-dependent services (if any) can authenticate over port 389/636.

4. Rollback Procedure (If Upgrade Fails)

If the upgrade encounters irrecoverable errors or breaking schema issues, roll back immediately using the automated restore script:

# Restore from the latest pre-upgrade backup
sudo /home/luke/git/backups/ul/scripts/restore/authentik

# Or restore an explicit backup timestamp
sudo /home/luke/git/backups/ul/scripts/restore/authentik YYYY-MM-DD_HHMM.SS

The restore script automatically:

  1. Verifies PAR2 integrity of database dumps and configuration archives (auto-repairing if needed).
  2. Gracefully stops Authentik containers (docker compose down).
  3. Saves a rollback snapshot of current files to /data/docker/authentik_rollback_<timestamp>/.
  4. Extracts original configuration files (.env, docker-compose.yml, templates, data).
  5. Restores PostgreSQL database schema and data from the dump.
  6. Restarts Authentik services with the previous configuration.

Standard Docker Compose Service Update Workflow

All containerized services under /data/docker/<service>/ follow a unified backup-first update workflow:

1. Pre-Update Backup

Run the service backup script to create a timestamped snapshot in /data/backups/services/current/<service>/ with 10% PAR2 redundancy:

sudo /home/luke/git/backups/ul/scripts/backup/<service>

2. Image Update & Container Recreation

  • Rolling-Tag Services (bookstack, collabora, degoog, kasm, navidrome, penpot, romm, valkey in /data/docker/infra, vaultwarden, wick, yourls): 
    cd /data/docker/<service>
sudo docker compose pull
sudo docker compose up -d
  • Pinned-Tag Services (anchor, authentik, immich, nextcloud): Update the version tag in docker-compose.yml or .env, then pull and recreate: 
    cd /data/docker/<service>
sudo docker compose pull
sudo docker compose up -d

3. Service-Specific Update Notes

Immich (/data/docker/immich, photos.drheck.dev)

  • Update IMMICH_VERSION=vX.Y.Z in /data/docker/immich/.env (immich_server uses ${IMMICH_VERSION} and immich_machine_learning uses ${IMMICH_VERSION}-rocm).
  • Starting in v3.3.1, immich-machine-learning natively implements ROCm ShapePolicy (ocr_canvases) and migraphx_model_cache_dir, so legacy v3.1.0 ml-overrides bind mounts must not be used.
  • Verify health and version via curl -sk https://photos.drheck.dev/api/server/version.

Nextcloud & Collabora Online (/data/docker/nextcloud, cloud.drheck.dev, office.drheck.dev)

  • Upgrade nextcloud:<version>-apache one major release at a time.
  • After container recreation, run database index and maintenance checks: 
    sudo docker exec -u www-data nextcloud php occ status
sudo docker exec -u www-data nextcloud php occ db:add-missing-indices
  • Verify https://cloud.drheck.dev/status.php ("maintenance":false) and https://office.drheck.dev/hosting/discovery.

RomM (/data/docker/romm, romm.drheck.dev)

  • Starting in RomM 5.3.0+, legacy filesystem.roms_folder and filesystem.firmware_folder keys in /data/docker/romm/config/config.yml are retired and must use filesystem.structure: 
    filesystem:
  structure:
    default: "roms/{platform}/{game}"
    firmware: "bios/{platform}"
  skip_hash_calculation: false

WIPStudio MinIO (/data/docker/wipstudio, wipstudio.drheck.dev)

  • Starting with RELEASE.2025-10-15T17-29-55Z, upstream MinIO moved to source-only distribution. Build the updated image using /data/docker/wipstudio/Dockerfile.minio: 
    cd /data/docker/wipstudio
sudo docker compose build minio
sudo docker compose up -d minio

Locally Built Container Services (sopro, subocr, webshell)

  • Sopro (/data/docker/sopro, sopro.drheck.dev): Rebuild without cache to pull the latest PyPI sopro release and apply patch.py: 
    cd /data/docker/sopro
sudo docker compose build --no-cache
sudo docker compose up -d
  • SubOCR (/data/docker/subocr, subocr.drheck.dev): Update the base image tag FROM ghcr.io/immich-app/immich-machine-learning:vX.Y.Z-rocm in /data/docker/subocr/Dockerfile, then rebuild (yt-dlp and uv are updated during build): 
    cd /data/docker/subocr
sudo docker compose build
sudo docker compose up -d
  • Webshell (/data/docker/webshell, webshell.drheck.dev): Update the ttyd and step-cli release versions in /data/docker/webshell/Dockerfile, then rebuild: 
    cd /data/docker/webshell
sudo docker compose build webshell
sudo docker compose up -d webshell

Native Systemd Services (ollama, docker, openssh)

  1. Run the systemd configuration backup: 
    sudo /home/luke/git/backups/ul/scripts/backup/systemd
  2. Sync and upgrade packages via pacman: 
    sudo pacman -Sy --noconfirm ollama ollama-vulkan docker openssh
  3. Restart and verify the service units: 
    sudo systemctl restart ollama docker sshd
systemctl is-active ollama docker sshd