Skip to main content

Backup Process

1. Overview & Architecture

This server (ul) implements an automated, multi-tiered backup system designed around high reliability, cryptographic security, redundancy, and verified recoverability. The backup infrastructure covers 40+43 containerized and host-levelnative host services, maintaining local PAR2-protected rollbacks and GPG-encrypted multi-cloud offsite copies.

Key Philosophy

  • Self-Contained Service Scripts: Each service has dedicated backup and restore scripts encapsulating its specificexact data dependencies (databases, configurations,ACLs, ACLs,Quadlet definitions, and persistent volumes).
  • Parity & Error Correction: Local backups utilize PAR2 (Parity Volume Set 2) with 10% redundancy to protect against bit-rot and silent disk corruption.
  • Zero-Trust Offsite Storage: Cloud backups are client-side encrypted using GPG (AES-256 / SHA-512) before leaving local disk, preventing cloud storage providers from inspecting sensitive server state or database contents.
  • Strict Verification Before Retention Pruning: Old backups (local and remote) are pruned only after a new backup run completes and passes 100% integrity verification checks (rclone check / par2 verify).

2. Directory & Data Structure

Path Purpose Access Control
/home/luke/git/backups/ul/scripts/ Shell and Python backup/restore execution scripts (GPG-signed git repository) Executable
/data/backups/services/current/<service>/ Most recent local backup run for each service 700 luke:luke
/data/backups/services/archive/<service>/ Previous local backup run(s) for point-in-time recovery 700 root:root
/data/backups/cloud/ Staged GPG-encrypted net archive ready for cloud upload 700 luke:luke
/data/backups/isos/ Staged optical media ISO images (when generating disk media) 700 luke:luke

3. End-to-End Backup Workflow

Local service archive creation, cloud staging, cloud upload, and optical ISO creation are separated into distinct commands under /home/luke/git/backups/ul/scripts/:

+-----------------------------------------------------------------------------------+
| 1. Local Service Backups (`sudo ./run_all_backups`)                               |
| Loop over /ul/scripts/backup/* -> Dump DBs, tar files, generate PAR2 -> /current |
| Self-archive backup scripts themselves into /current/backup                       |
+-----------------------------------------------------------------------------------+
                     |                                           |
                     v                                           v
+-----------------------------------------+   +-------------------------------------+
| 2a. Cloud Staging                       |   | 2b. Optical ISO Staging             |
| (`./stage_cloud_archive`)               |   | (`./stage_iso_archive`)             |
| Pack /current -> GPG Encrypt -> /cloud  |   | Pack /current -> GPG ISOs -> /isos  |
+-----------------------------------------+   +-------------------------------------+
                     |                                           |
                     v                                           v
+-----------------------------------------+   +-------------------------------------+
| 3a. Cloud Upload & Remote Rotation      |   | 3b. Optical Media Burn & Verify     |
| (`./upload_cloud_archive [remotes]`)    |   | (`./burn_and_verify_iso <iso>`)     |
| Upload /cloud -> rclone check -> Rotate |   | Burn DVD/M-DISC -> Verify SHA256    |
+-----------------------------------------+   +-------------------------------------+

Commands & Flags

  • sudo ./run_all_backups: Creates local archives for all services in /data/backups/services/current/<service>/ and archives the backup scripts themselves into /data/backups/services/current/backup/ (default: local archives only, no cloud staging or upload).
  • sudo ./run_all_backups --upload [remotes]: Runs all local service backups and then executes stage_cloud_archive and upload_cloud_archive [remotes].
  • ./stage_cloud_archive [--upload [remotes]]: Stages the GPG-encrypted cloud archive in /data/backups/cloud/ (and optionally uploads it immediately if --upload is passed).
  • ./upload_cloud_archive [remotes]: Uploads the staged /data/backups/cloud/ directory to all configured cloud remotes, or to specific comma-separated remotes (e.g., ./upload_cloud_archive gdrive,protondrive).
  • ./stage_iso_archive: Packs and GPG-encrypts /data/backups/services/current/ into 4.7GB optical DVD ISO images in /data/backups/isos/.

4. Service-Level Backups (/ul/scripts/backup/)

Every service backup script performs atomic snapshots using the following standard pattern:

  1. Root Verification & Timestamping: Requires root permissions (EUID == 0) to preserve subordinate container UID/GID ownership (1100, 624288--689823), POSIX file ownership, ACLs (--acls), and extended attributes (--xattrs). GeneratesTimestamp timestamp formatted asformat: YYYY-MM-DD_HHMM.SS.
  2. Local Archive Rotation: Moves existing /current/<service>/* directories to /archive/<service>/.
  3. Consistent Database Dump:& Cache Snapshots:
    • Native Host PostgreSQL (authentik, anchor, forgejo, immich, nextcloud_db, vikunja, wipstudio): Staged to /tmp/<fn>.dump first via sudo -u postgres pg_dump -F c -b -v before moving into /data/backups/services/current/<service>/.
  4. Containerized PostgreSQL (penpot-postgres, funkwhale): Executed rootlessly via (cd /tmp && sudo -u docker-data podman exec penpot-postgres pg_dump -U penpot -F c -b penpot) and podman exec -u funkwhale funkwhale pg_dumpall -U funkwhale.
Containerized MariaDB /(bookstack_db, MySQL:romm-db, yourls-db): docker(cd /tmp && sudo -u docker-data podman exec <db_container> mariadb-dump --single-transaction --routines --triggers ...). PostgreSQL:SQLite WAL Checkpoints (navidrome, subocr, vaultwarden): Consistent online sudo.backup -uor postgresPRAGMA pg_dumpwal_checkpoint(TRUNCATE). -FNative cValkey -b(valkey.service): -vTriggers BGSAVE with dedicated read/save-only ACL user backup and archives /var/lib/valkey/dump.rdb + /etc/valkey/. ConfigurationConfiguration, Quadlet & FileVolume Compression:
  • Compressed using multi-threaded XZ compression (XZ_OPT="-T0" tar --acls --xattrs -cJf ... XZ_OPT="-T0").
  • ArchivingPreserves preserves environment variables (.env), Docker Compose files (docker-compose.yml), Podman Quadlet definitions (<service>.container), <service>.network, application data/assets, and host configs (systemd archives /etc/systemd, /etc/containers, /etc/haproxy, /etc/nftables.conf, and configuration directories./home/docker-data/.config/containers/systemd).
PAR2 Recovery Generation:
  • Generates 10% parity redundancy (par2 create -r10) directly alongside theevery database dump and tar archives.tarball.
Execution Logging: CaptureCaptures output to /tmp/<service>_backup_<pid>.backup_execution.log andinside moveeach totimestamped the finalservice backup path (backup_execution.log).folder.

Backed-Up Services (43 Targets)

acme, amux, anchor, audiomass, authentik, bentopdf, bookstack, cam, craft, crowdsec, degoog, demucs, drawpile, excalidraw, forgejo, funkwhale, grill, immich, jellyfin, mc_manager, minecraft_router, navidrome, nextcloud, nginx, penpot, rclone, romm, scontrol, sopro, subocr, sunshine, systemd, tldraw, transmission, valkey, vaultwarden, vikunja, wavacity, webrcade, webshell, wick, wipstudio, wireguard, yourls.

5. Cloud Archiving & Encryption (archive_pack.py)

Local service backups are consolidated into a secure net package by archive_pack.py invoked via stage_cloud_archive:

  • Input Directory: /data/backups/services/current
  • Output Directory: /data/backups/cloud/
  • GPG Recipient: Key fingerprint 9F9AC4567EDC8E2B86E5CB9475D6600BEF4E8E8F
  • Cryptographic Specifications: AES-256 cipher, SHA-512 digest.
  • Manifests & Hashes: Generates manifest.json, manifest.txt, SHA256SUMS, and DECRYPT_INSTRUCTIONS.txt mapping original file sizes, staged sizes, SHA-256 hashes, and GPG metadata.
  • Optical Media Support: archive_pack.py also supports --mode disk (for CD/DVD/Blu-ray ISO image staging with optional splitting and PAR2 protection via stage_iso_archive).

6. Cloud Orchestration & Provider Synchronization

Cloud synchronization is orchestrated by upload_cloud_archive calling provider scripts in upload/:

Configured Providers

  1. Google Drive (upload/gdrive)
  2. Proton Drive (upload/protondrive)

Upload Reliability Features

  • Exponential Backoff & Retries: Up to 3 main attempt loops with exponential delay scaling (10s/15s initial delays). Low-level HTTP retries configured up to 20 attempts.
  • Self-Healing Draft Cleanup: If rclone copy encounters mid-transfer errors or failed file copies, the upload script parses rclone logs and explicitly deletes failed/corrupted draft files from the remote before retrying.
  • Strict Integrity Verification: Post-upload, rclone check --one-way --size-only verifies that remote file sizes match local files exactly.
  • Atomic Failure Safety Guard: If upload or verification fails across all attempts, the script deletes the partial remote upload directory (rclone purge) and skips retention rotation to preserve existing valid cloud backups.

Remote Retention Policy

  • Max Remote Backups: Retains the newest 3 successful cloud backups (MAX_BACKUPS=3).
  • Automated Proton Trash Cleaning: Executes custom Go binary (proton-trash-cleaner) to automatically purge deleted backup files from Proton Drive's cloud trash bin.

7. Restoration Practice & Recovery Procedure

Restoration scripts reside in /home/luke/git/backups/ul/scripts/restore/. Each service script handles full state recovery:

Standard Restore Procedure

  1. Target Selection: Accepts an explicit backup directory or timestamp as argument. If omitted, defaults to the latest backup in /data/backups/services/current/<service>/.
  2. PAR2 Integrity Check & Self-Repair: Runs par2 verify on database dumps and file tarballs prior to performing any system modifications. Automatically executes par2 repair if corruption is detected.
  3. Interactive Confirmation: Prompts user for explicit confirmation (yes/no) before stopping services or modifying state.
  4. Service Suspension: Gracefully stops active servicerootless containersPodman Quadlet units (dockersudo compose-u stopdocker-data <service>XDG_RUNTIME_DIR=/run/user/1100 or systemctl --user -M docker-data@ stop <service>.service) foror rootlessnative Podmansystemd Quadlets).units.
  5. Rollback Snapshot Creation: Preserves current configurations in a timestamped rollback folder (<service_dir>_rollback_<timestamp>).
  6. File & Database Restoration:
    • Extracts compressed tarball over service data directory with POSIX ACL/xattr and subordinate UID/GID preservation.
    • Re-creates target database schema (DROP DATABASE IF EXISTS ... CREATE DATABASE ...) and imports the SQL dump or PostgreSQL custom binary dump.
  7. Service Resumption: BringsReloads containersand backrestarts onlinerootless Quadlets (docker compose upsudo -du ordocker-data XDG_RUNTIME_DIR=/run/user/1100 systemctl --user -M docker-data@ daemon-reload && sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 systemctl --user -M docker-data@ start <service>.service).

8. Retention Pruning Practice (prune_old_local_backups)

To prevent local storage exhaustion on the server, local service backup folders are managed by /home/luke/git/backups/ul/scripts/prune_old_local_backups:

  • Local Retention Limit: Retains the 2 most recent backups per service (KEEP_COUNT=2) across /data/backups/services/current and /data/backups/services/archive.
  • Safe Dry-Run Flag: Supports --dry-run (-n) to inspect directories eligible for pruning without performing deletions.