Skip to main content

Service Update Process

This document outlines the standard operational procedures for updating services hosted on ul. Follow these steps carefully to ensure clean migrations, data integrity, and verified service continuity.

Authentik Update Process

Authentik is deployed via Docker Compose in /data/docker/authentik/ backed by a host PostgreSQL database and Valkey (Redis) caching container on the infra_net bridge network.

1. Determining When an Update is Available

Authentik follows Calendar Versioning (YYYY.M.P, e.g., 2026.5.6).

  • Admin Web UI: Log in to auth.drheck.dev as an administrator and navigate to Admin Interface → System → Overview. A notification banner appears when a newer version is available.

  • GitHub Releases: Monitor the official Authentik Releases page for release tags, changelogs, and release notes.

  • Current Installed Version: Check the running version on the server:

    # Check image tags in compose file
grep 'image:' /data/docker/authentik/docker-compose.yml

# Or check active container images
docker ps --filter "name=authentik" --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

  • Read the Release Notes for every release between your current version and the target version.

Do not skip minor versions if you are multiple releases behind (e.g., to upgrade from 2024.8.x to 2024.12.x, first upgrade to 2024.10.x, verify migrations complete, then upgrade to 2024.12.x).

2. Update Procedure

Step 1: Perform Pre-Update Backup

Always create an atomic, PAR2-verified snapshot of the PostgreSQL database and configuration files before modifying configuration or pulling new images:

sudo /home/luke/git/backups/ul/scripts/backup/authentik

This creates a timestamped archive in /data/backups/services/current/authentik/<YYYY-MM-DD_HHMM.SS>/ containing authentik_db_backup.dump and authentik_files_backup.tar.xz with 10% PAR2 recovery parity.

Step 2: Edit docker-compose.yml

Navigate to the Authentik directory and update the image tags for server, worker, and ldap to the new version:

cd /data/docker/authentik
sudo $EDITOR docker-compose.yml

Update the version tag across all three services:

services:
  server:
    image: ghcr.io/goauthentik/server:<NEW_VERSION>
    ...
  worker:
    image: ghcr.io/goauthentik/server:<NEW_VERSION>
    ...
  ldap:
    image: ghcr.io/goauthentik/ldap:<NEW_VERSION>
    ...

Step 3: Pull New Container Images

Download the updated Docker images:

cd /data/docker/authentik
sudo docker compose pull

Step 4: Recreate Containers & Apply Migrations

Recreate the containers with the new images in detached mode:

sudo docker compose up -d

Step 5: Monitor Startup & Database Migrations

Follow container logs to monitor automatic database migrations and ensure all workers initialize without errors:

sudo docker compose logs -f

(Press Ctrl+C once the server and worker report healthy startup status).

3. After-Update Verification Checks

Perform the following verification checks to ensure Authentik and dependent services are operating normally:

1. Container Health Status

Verify that authentik-server, authentik-worker, and authentik-ldap are running and reporting healthy:

sudo docker compose ps

2. Authentik Admin Interface & Outpost Health

  1. Open https://auth.drheck.dev/ in your browser and sign in.
  2. Go to Admin Interface → System → Overview:
    • Verify the reported version matches the newly installed version.
    • Verify Workers are active and task queues are processing.
    • Check for any outstanding warnings, failed tasks, or blueprint sync errors.
  3. Check Applications → Outposts:
    • Verify both the Embedded Outpost and LDAP Outpost (authentik-ldap) show green/healthy connectivity status.

3. Authentication & Proxy Flow Verification

  • Forward Auth / Nginx Proxy Verification: Test authentication against an Authentik-protected endpoint (e.g., https://scontrol.drheck.dev/, https://bt.drheck.dev/, or https://drawpile.drheck.dev/) in a private/incognito window to verify authentication redirects and session tokens function properly.
  • LDAP Outpost Verification: Ensure LDAP-dependent services (if any) can authenticate over port 389/636.

4. Rollback Procedure (If Upgrade Fails)

If the upgrade encounters irrecoverable errors or breaking schema issues, roll back immediately using the automated restore script:

# Restore from the latest pre-upgrade backup
sudo /home/luke/git/backups/ul/scripts/restore/authentik

# Or restore an explicit backup timestamp
sudo /home/luke/git/backups/ul/scripts/restore/authentik YYYY-MM-DD_HHMM.SS

The restore script automatically:

  1. Verifies PAR2 integrity of database dumps and configuration archives (auto-repairing if needed).
  2. Gracefully stops Authentik containers (docker compose down).
  3. Saves a rollback snapshot of current files to /data/docker/authentik_rollback_<timestamp>/.
  4. Extracts original configuration files (.env, docker-compose.yml, templates, data).
  5. Restores PostgreSQL database schema and data from the dump.
  6. Restarts Authentik services with the previous configuration.