Skip to main content

Backup Process

Backup Concepts and Practice

1. Overview & Architecture

This server (ul) implements an automated, multi-tiered backup system designed around high reliability, cryptographic security, redundancy, and verified recoverability. The backup infrastructure covers 40+ containerized and host-level services, maintaining local rollbacks and multi-cloud offsite copies.

Key Philosophy

    Self-Contained Service Scripts: Each service has dedicated backup and restore scripts encapsulating its specific data dependencies (databases, configurations, ACLs, and persistent volumes). Parity & Error Correction: Local backups utilize PAR2 (Parity Volume Set 2) with 10% redundancy to protect against bit-rot and silent disk corruption. Zero-Trust Offsite Storage: Cloud backups are client-side encrypted using GPG (AES-256 / SHA-512) before leaving local disk, preventing cloud storage providers from inspecting sensitive server state or database contents. Strict Verification Before Retention Pruning: Old backups (local and remote) are pruned only after a new backup run completes and passes 100% integrity verification checks (rclone check / par2 verify).

    2. Directory & Data Structure

    Path Purpose Access Control /home/luke/git/backups/ul/scripts/ Shell and Python backup/restore execution scripts Executable /data/backups/services/current/<service>/ Most recent local backup run for each service 700 luke:luke /data/backups/services/archive/<service>/ Previous local backup run(s) for point-in-time recovery 700 root:root /data/backups/cloud/ Staged GPG-encrypted net archive ready for cloud upload 700 luke:luke /data/backups/isos/ Staged optical media ISO images (when generating disk media) 700 luke:luke

    3. End-to-End Backup Workflow

    Local service archive creation, cloud staging, cloud upload, and optical ISO creation are separated into distinct commands under /home/luke/git/backups/ul/scripts/:

    +-----------------------------------------------------------------------------------+
| 1. Local Service Backups (`sudo ./run_all_backups`)                               |
| Loop over /ul/scripts/backup/* -> Dump DBs, tar files, generate PAR2 -> /current |
| Self-archive backup scripts themselves into /current/backup                       |
+-----------------------------------------------------------------------------------+
                     |                                           |
                     v                                           v
+-----------------------------------------+   +-------------------------------------+
| 2a. Cloud Staging                       |   | 2b. Optical ISO Staging             |
| (`./stage_cloud_archive`)               |   | (`./stage_iso_archive`)             |
| Pack /current -> GPG Encrypt -> /cloud  |   | Pack /current -> GPG ISOs -> /isos  |
+-----------------------------------------+   +-------------------------------------+
                     |                                           |
                     v                                           v
+-----------------------------------------+   +-------------------------------------+
| 3a. Cloud Upload & Remote Rotation      |   | 3b. Optical Media Burn & Verify     |
| (`./upload_cloud_archive [remotes]`)    |   | (`./burn_and_verify_iso <iso>`)     |
| Upload /cloud -> rclone check -> Rotate |   | Burn DVD/M-DISC -> Verify SHA256    |
+-----------------------------------------+   +-------------------------------------+

    Commands & Flags

      sudo ./run_all_backups: Creates local archives for all services in /data/backups/services/current/<service>/ and archives the backup scripts themselves into /data/backups/services/current/backup/ (default: local archives only, no cloud staging or upload). sudo ./run_all_backups --upload [remotes]: Runs all local service backups and then executes stage_cloud_archive and upload_cloud_archive [remotes]. ./stage_cloud_archive [--upload [remotes]]: Stages the GPG-encrypted cloud archive in /data/backups/cloud/ (and optionally uploads it immediately if --upload is passed). ./upload_cloud_archive [remotes]: Uploads the staged /data/backups/cloud/ directory to all configured cloud remotes, or to specific comma-separated remotes (e.g., ./upload_cloud_archive gdrive,protondrive). ./stage_iso_archive: Packs and GPG-encrypts /data/backups/services/current/ into 4.7GB optical DVD ISO images in /data/backups/isos/.

      4. Service-Level Backups (/ul/scripts/backup/)

      Every service backup script performs atomic snapshots using the following standard pattern:

        Root Verification & Timestamping: Requires root permissions (EUID == 0) to preserve POSIX file ownership, ACLs (--acls), and extended attributes (--xattrs). Generates timestamp formatted as YYYY-MM-DD_HHMM.SS. Local Archive Rotation: Moves existing /current/<service>/* directories to /archive/<service>/. Database Dump:
          MariaDB / MySQL: docker exec <db_container> mariadb-dump --single-transaction --routines --triggers PostgreSQL: sudo -u postgres pg_dump -F c -b -v Configuration & File Compression:
            Compressed using multi-threaded XZ compression (tar -cJf ... XZ_OPT="-T0"). Archiving preserves environment variables (.env), Compose files (docker-compose.yml), and configuration directories. PAR2 Recovery Generation:
              Generates 10% parity redundancy (par2 create -r10) directly alongside the database dump and tar archives. Execution Logging: Capture output to /tmp/<service>_backup_<pid>.log and move to the final backup path (backup_execution.log).

              Backed-Up Services

              acme, amux, anchor, audiomass, authentik, bentopdf, bookstack, cam, craft, crowdsec, degoog, demucs, drawpile, excalidraw, funkwhale, grill, immich, jellyfin, kasm, mc_manager, minecraft_router, navidrome, nextcloud, nginx, penpot, rclone, romm, scontrol, sopro, subocr, sunshine, systemd, tldraw, transmission, valkey, vaultwarden, vikunja, wavacity, webrcade, webshell, wick, wipstudio, wireguard, yourls.

              5. Cloud Archiving & Encryption (archive_pack.py)

              Local service backups are consolidated into a secure net package by archive_pack.py invoked via stage_cloud_archive:

                Input Directory: /data/backups/services/current Output Directory: /data/backups/cloud/ GPG Recipient: Key fingerprint 9F9AC4567EDC8E2B86E5CB9475D6600BEF4E8E8F Cryptographic Specifications: AES-256 cipher, SHA-512 digest. Manifests & Hashes: Generates manifest.json, manifest.txt, SHA256SUMS, and DECRYPT_INSTRUCTIONS.txt mapping original file sizes, staged sizes, SHA-256 hashes, and GPG metadata. Optical Media Support: archive_pack.py also supports --mode disk (for CD/DVD/Blu-ray ISO image staging with optional splitting and PAR2 protection via stage_iso_archive).

                6. Cloud Orchestration & Provider Synchronization

                Cloud synchronization is orchestrated by upload_cloud_archive calling provider scripts in upload/:

                Configured Providers

                  Google Drive (upload/gdrive) Proton Drive (upload/protondrive)

                  Upload Reliability Features

                    Exponential Backoff & Retries: Up to 3 main attempt loops with exponential delay scaling (10s/15s initial delays). Low-level HTTP retries configured up to 20 attempts. Self-Healing Draft Cleanup: If rclone copy encounters mid-transfer errors or failed file copies, the upload script parses rclone logs and explicitly deletes failed/corrupted draft files from the remote before retrying. Strict Integrity Verification: Post-upload, rclone check --one-way --size-only verifies that remote file sizes match local files exactly. Atomic Failure Safety Guard: If upload or verification fails across all attempts, the script deletes the partial remote upload directory (rclone purge) and skips retention rotation to preserve existing valid cloud backups.

                    Remote Retention Policy

                      Max Remote Backups: Retains the newest 3 successful cloud backups (MAX_BACKUPS=3). Automated Proton Trash Cleaning: Executes custom Go binary (proton-trash-cleaner) to automatically purge deleted backup files from Proton Drive's cloud trash bin.

                      7. Restoration Practice & Recovery Procedure

                      Restoration scripts reside in /home/luke/git/backups/ul/scripts/restore/. Each service script handles full state recovery:

                      Standard Restore Procedure

                        Target Selection: Accepts an explicit backup directory or timestamp as argument. If omitted, defaults to the latest backup in /data/backups/services/current/<service>/. PAR2 Integrity Check & Self-Repair: Runs par2 verify on database dumps and file tarballs prior to performing any system modifications. Automatically executes par2 repair if corruption is detected. Interactive Confirmation: Prompts user for explicit confirmation (yes/no) before stopping services or modifying state. Service Suspension: Gracefully stops active service containers (docker compose stop <service>). Rollback Snapshot Creation: Preserves current configurations in a timestamped rollback folder (<service_dir>_rollback_<timestamp>). File & Database Restoration:
                          Extracts compressed tarball over service data directory with POSIX ACL/xattr preservation. Re-creates target database schema (DROP DATABASE IF EXISTS ... CREATE DATABASE ...) and imports the SQL dump or PostgreSQL binary dump. Service Resumption: Brings containers back online (docker compose up -d).

                          8. Retention Pruning Practice (prune_old_local_backups)

                          To prevent local storage exhaustion on the server, local service backup folders are managed by /home/luke/git/backups/ul/scripts/prune_old_local_backups:

                            Local Retention Limit: Retains the 2 most recent backups per service (KEEP_COUNT=2) across /data/backups/services/current and /data/backups/services/archive. Safe Dry-Run Flag: Supports --dry-run (-n) to inspect directories eligible for pruning without performing deletions.