Skip to main content

Backup Process

Backup Concepts and Practice

1. Overview & Architecture

This server (ul) implements an automated, multi-tiered backup system designed around high reliability, cryptographic security, redundancy, and verified recoverability. The backup infrastructure covers 40+ containerized and host-level services, maintaining local rollbacks and multi-cloud offsite copies.

Key Philosophy

  • Self-Contained Service Scripts: Each service has dedicated backup and restore scripts encapsulating its specific data dependencies (databases, configurations, ACLs, and persistent volumes).
  • Parity & Error Correction: Local backups utilize PAR2 (Parity Volume Set 2) with 10% redundancy to protect against bit-rot and silent disk corruption.
  • Zero-Trust Offsite Storage: Cloud backups are client-side encrypted using GPG (AES-256 / SHA-512) before leaving local disk, preventing cloud storage providers from inspecting sensitive server state or database contents.
  • Strict Verification Before Retention Pruning: Old backups (local and remote) are pruned only after a new backup run completes and passes 100% integrity verification checks (rclone check / par2 verify).

2. Directory & Data Structure

Path Purpose Access Control
/home/luke/git/backups/ul/scripts/ Shell and Python backup/restore execution scripts Executable
/data/backups/services/current/<service>/ Most recent local backup run for each service 700 luke:luke
/data/backups/services/archive/<service>/ Previous local backup run(s) for point-in-time recovery 700 root:root
/data/backups/cloud/ Staged GPG-encrypted net archive ready for cloud upload 700 luke:luke
/data/backups/isos/ Staged optical media ISO images (when generating disk media) 700 luke:luke

3. End-to-End Backup Workflow

Local service archive creation, cloud staging, cloud upload, and optical ISO creation are separated into distinct commands under /home/luke/git/backups/ul/scripts/:

+-----------------------------------------------------------------------------------+
| 1. Local Service Backups (`sudo ./run_all_backups`)                               |
| Loop over /ul/scripts/backup/* -> Dump DBs, tar files, generate PAR2 -> /current |
| Self-archive backup scripts themselves into /current/backup                       |
+-----------------------------------------------------------------------------------+
                     |                                           |
                     v                                           v
+-----------------------------------------+   +-------------------------------------+
| 2a. Cloud Staging                       |   | 2b. Optical ISO Staging             |
| (`./stage_cloud_archive`)               |   | (`./stage_iso_archive`)             |
| Pack /current -> GPG Encrypt -> /cloud  |   | Pack /current -> GPG ISOs -> /isos  |
+-----------------------------------------+   +-------------------------------------+
                     |                                           |
                     v                                           v
+-----------------------------------------+   +-------------------------------------+
| 3a. Cloud Upload & Remote Rotation      |   | 3b. Optical Media Burn & Verify     |
| (`./upload_cloud_archive [remotes]`)    |   | (`./burn_and_verify_iso <iso>`)     |
| Upload /cloud -> rclone check -> Rotate |   | Burn DVD/M-DISC -> Verify SHA256    |
+-----------------------------------------+   +-------------------------------------+

Commands & Flags

  • sudo ./run_all_backups: Creates local archives for all services in /data/backups/services/current/<service>/ and archives the backup scripts themselves into /data/backups/services/current/backup/ (default: local archives only, no cloud staging or upload).
  • sudo ./run_all_backups --upload [remotes]: Runs all local service backups and then executes stage_cloud_archive and upload_cloud_archive [remotes].
  • ./stage_cloud_archive [--upload [remotes]]: Stages the GPG-encrypted cloud archive in /data/backups/cloud/ (and optionally uploads it immediately if --upload is passed).
  • ./upload_cloud_archive [remotes]: Uploads the staged /data/backups/cloud/ directory to all configured cloud remotes, or to specific comma-separated remotes (e.g., ./upload_cloud_archive gdrive,protondrive).
  • ./stage_iso_archive: Packs and GPG-encrypts /data/backups/services/current/ into 4.7GB optical DVD ISO images in /data/backups/isos/.

4. Service-Level Backups (/ul/scripts/backup/)

Every service backup script performs atomic snapshots using the following standard pattern:

  1. Root Verification & Timestamping: Requires root permissions (EUID == 0) to preserve POSIX file ownership, ACLs (--acls), and extended attributes (--xattrs). Generates timestamp formatted as YYYY-MM-DD_HHMM.SS.
  2. Local Archive Rotation: Moves existing /current/<service>/* directories to /archive/<service>/.
  3. Database Dump:
    • MariaDB / MySQL: docker exec <db_container> mariadb-dump --single-transaction --routines --triggers
    • PostgreSQL: sudo -u postgres pg_dump -F c -b -v
  4. Configuration & File Compression:
    • Compressed using multi-threaded XZ compression (tar -cJf ... XZ_OPT="-T0").
    • Archiving preserves environment variables (.env), Compose files (docker-compose.yml), and configuration directories.
  5. PAR2 Recovery Generation:
    • Generates 10% parity redundancy (par2 create -r10) directly alongside the database dump and tar archives.
  6. Execution Logging: Capture output to /tmp/<service>_backup_<pid>.log and move to the final backup path (backup_execution.log).

Backed-Up Services

acme, amux, anchor, audiomass, authentik, bentopdf, bookstack, cam, craft, crowdsec, degoog, demucs, drawpile, excalidraw, funkwhale, grill, immich, jellyfin, kasm, mc_manager, minecraft_router, navidrome, nextcloud, nginx, penpot, rclone, romm, scontrol, sopro, subocr, sunshine, systemd, tldraw, transmission, valkey, vaultwarden, vikunja, wavacity, webrcade, webshell, wick, wipstudio, wireguard, yourls.

5. Cloud Archiving & Encryption (archive_pack.py)

Local service backups are consolidated into a secure net package by archive_pack.py invoked via stage_cloud_archive:

  • Input Directory: /data/backups/services/current
  • Output Directory: /data/backups/cloud/
  • GPG Recipient: Key fingerprint 9F9AC4567EDC8E2B86E5CB9475D6600BEF4E8E8F
  • Cryptographic Specifications: AES-256 cipher, SHA-512 digest.
  • Manifests & Hashes: Generates manifest.json, manifest.txt, SHA256SUMS, and DECRYPT_INSTRUCTIONS.txt mapping original file sizes, staged sizes, SHA-256 hashes, and GPG metadata.
  • Optical Media Support: archive_pack.py also supports --mode disk (for CD/DVD/Blu-ray ISO image staging with optional splitting and PAR2 protection via stage_iso_archive).

6. Cloud Orchestration & Provider Synchronization

Cloud synchronization is orchestrated by upload_cloud_archive calling provider scripts in upload/:

Configured Providers

  1. Google Drive (upload/gdrive)
  2. Proton Drive (upload/protondrive)

Upload Reliability Features

  • Exponential Backoff & Retries: Up to 3 main attempt loops with exponential delay scaling (10s/15s initial delays). Low-level HTTP retries configured up to 20 attempts.
  • Self-Healing Draft Cleanup: If rclone copy encounters mid-transfer errors or failed file copies, the upload script parses rclone logs and explicitly deletes failed/corrupted draft files from the remote before retrying.
  • Strict Integrity Verification: Post-upload, rclone check --one-way --size-only verifies that remote file sizes match local files exactly.
  • Atomic Failure Safety Guard: If upload or verification fails across all attempts, the script deletes the partial remote upload directory (rclone purge) and skips retention rotation to preserve existing valid cloud backups.

Remote Retention Policy

  • Max Remote Backups: Retains the newest 3 successful cloud backups (MAX_BACKUPS=3).
  • Automated Proton Trash Cleaning: Executes custom Go binary (proton-trash-cleaner) to automatically purge deleted backup files from Proton Drive's cloud trash bin.

7. Restoration Practice & Recovery Procedure

Restoration scripts reside in /home/luke/git/backups/ul/scripts/restore/. Each service script handles full state recovery:

Standard Restore Procedure

  1. Target Selection: Accepts an explicit backup directory or timestamp as argument. If omitted, defaults to the latest backup in /data/backups/services/current/<service>/.
  2. PAR2 Integrity Check & Self-Repair: Runs par2 verify on database dumps and file tarballs prior to performing any system modifications. Automatically executes par2 repair if corruption is detected.
  3. Interactive Confirmation: Prompts user for explicit confirmation (yes/no) before stopping services or modifying state.
  4. Service Suspension: Gracefully stops active service containers (docker compose stop <service>).
  5. Rollback Snapshot Creation: Preserves current configurations in a timestamped rollback folder (<service_dir>_rollback_<timestamp>).
  6. File & Database Restoration:
    • Extracts compressed tarball over service data directory with POSIX ACL/xattr preservation.
    • Re-creates target database schema (DROP DATABASE IF EXISTS ... CREATE DATABASE ...) and imports the SQL dump or PostgreSQL binary dump.
  7. Service Resumption: Brings containers back online (docker compose up -d).

8. Retention Pruning Practice (prune_old_local_backups)

To prevent local storage exhaustion on the server, local service backup folders are managed by /home/luke/git/backups/ul/scripts/prune_old_local_backups:

  • Local Retention Limit: Retains the 2 most recent backups per service (KEEP_COUNT=2) across /data/backups/services/current and /data/backups/services/archive.
  • Safe Dry-Run Flag: Supports --dry-run (-n) to inspect directories eligible for pruning without performing deletions.