System Change log

2026-08-08 Saturday

setting up navidrome
had AI write a script to pull down missing artwork for Jellyfin
noticed collation version mismatch when poking around in posgresql.

Check Which Databases Have Collation Version Mismatches

sudo -u postgres bash -c 'for db in $(psql -At -c "SELECT datname FROM pg_database WHERE datistemplate = false;"); do
  echo "=== DATABASE: $db ==="
  psql -d "$db" -c "SELECT 1;" 2>&1
done'

Run amcheck to Detect Corrupted B-Tree Indexes

sudo -u postgres bash -c '
for db in spacebar immich nextcloud_db anchor authentik postgres gitea nextcloud; do
  echo "=== AMCHECK: $db ==="
  psql -d "$db" -c "
      CREATE EXTENSION IF NOT EXISTS amcheck;
      SELECT c.relname, bt_index_check(c.oid, true)
      FROM pg_index i
      JOIN pg_class c ON c.oid = i.indexrelid
      JOIN pg_namespace n ON n.oid = c.relnamespace
      WHERE c.relkind = '\''i'\'' AND i.indisready AND                i.indisvalid AND n.nspname = '\''public'\'';" 2>&1
done'

Reindex Databases and Refresh Collation Versions

sudo -u postgres bash -c '
for db in spacebar immich nextcloud_db anchor authentik postgres gitea nextcloud; do
  echo "=== REINDEXING & REFRESHING: $db ==="
  psql -d "$db" -c "REINDEX DATABASE \"$db\";"
  psql -d "$db" -c "ALTER DATABASE \"$db\" REFRESH COLLATION VERSION;"
done'

Verify All Warnings Are Cleared

sudo -u postgres bash -c '
for db in $(psql -At -c "SELECT datname FROM pg_database WHERE datistemplate = false;"); do
  echo "=== VERIFY: $db ==="
  psql -d "$db" -c "SELECT 1;" 2>&1
done'

2026-08-05 Wednesday
- updates to crowdsec
  - parsing more SSH errors
  - generated a table of banned IPs but it's not very interesting. Live world view would be cooler
2026-08-04 Tuesday
- docker updates
  - anchor: 0.13.0 → 0.14.0
  - authentik: 2026.5.5 → 2026.5.6
  - bookstack: Updated to latest image
  - immich: v3.0.3 → v3.1.0
  - infra (valkey): Updated to latest image build
  - kasm: Updated to latest image
  - nextcloud: 34.0.1-apache → 34.0.2-apache
  - penpot: Frontend, backend, exporter, postgres, and redis updated
  - romm: Updated to latest image (5.1.0)
  - vaultwarden: Updated to latest image
  - webrcade: Updated to latest image
  - wick: Updated to latest image
- installed crowdsec
  - review of publicly accesible service logs (eg: wiki) shows no bot traffic (yet)!
  - review of nginx logs straight from the public IPs shows bot traffic looking for common vulnerabilities, exploits, and staged malware.
  - documentation
2026-08-03 Monday
- pull all new docker images and prepare for update
- in preparation for more users, ensure unencrypted backups aren't world readable
- gdrive backup credentials failed again
  - finally cleaned up google cloud project, renamed properly and created two separate subprojects, google sso, and rclone for ul
  - redid the certs for rclone, tested good
- ran a backup to repare for docker update and all the online backups had no files except the instructions!
- setting all the backups to root rendered the rclone script useless.
- Trying all backup files owned by my user with a plan
  - create a dedicated backup user
  - create a new gpg key for backups
  - backup gpg key
  - move rclone credentials to backup user
  - chown all backups to 700 backup user
  - Profit!
2026-08-02 Sunday
- Removed old unused "web shells" like the old ttyd, whatever else. Also warpgate.
- Made webshell awesome. Picked a single font, created a service to show ANSI art on login
- made big improvements to ansview
  - first render is completely in memory, remove all jumps and relative and exact positions. This allows the final product to be rendered correctly when not placed at the first screen column.
  - due to this success, added an auto wide mode, if artwork can fit on the screen multiple times horizontally, do it
  - also truncate the artwork if viewing on a smaller screen rather than an awful wrap. This is great for the phone shell.
2026-08-01 Saturday
- Created webshell, and spent a lot time trying to make it customizable per user. This was a failure.
2026-07-31 Friday
- Set up moebius in KASM, generated a KASM setup guide.
2026-07-30 Thursday
- giving up on warpgate for now, trying ttyd again with step-ca to tie it into OIDC. I really want an authentik OIDC enabled direct web -> shell.
2026-07-28 Tuesday
- got warpgate to stop crashing but I'm not sure how I can use it for a web based terminal.. we'll see
2026-07-26 Sunday
- accidentally blocked the git users with the new firewall rules, allowed it to access the gitea service and we are able to upload again
- started to set up warpgate as it looks like it can provide a shell that can login with OIDC
2026-07-25 Saturday
- replaced gemini-cli with opencode, it's way better
- went through each docker service moving off of the default 1000 UID
- removed broken web "shells"
  - Made a process to generate short lived ssh certs for web users to get a shell without typing a password but it felt really insecure and I think you could impersonate other users, so I killed it.
2026-07-23 Thursday
- Set nginx to cache large uploads on the ZFS array as / was filling up with video uploaded from immich
- Set permanent redirects on short names in the browser for convenience, eg: http://photos -> https://photos.drheck.dev/
2026-07-22 Wednesday
- Configured rclone for Google drive and Proton drive. Added this to the backup scripts. Now we have automated on-line backups.
- created rclone client config in Google's cloud console
2026-07-21 Tuesday
- transmission would timeout quickly since exposed behind authentik. After some research I modifed the nginx config to start forwarding $host, $remote_addr, $proxy_add_x_forwarded_for, and $scheme.
2026-07-20 Monday
- IPv6 netblock was wrong again. This time an IPv6 netblock change was detected then the network went out. This caused the API call to desec.io to fail. We were writing the new IPv6 netblock unconditionally though so no more updates were going to happen. Script is updated 1 2
- also added logging to the desec.io call in change 2 ^
2026-07-19 Sunday
- Tested LDAP users
  - could log in but did not have the ability to choose the window environment
  - LDAP guide updated to fix this problem
2026-07-17 Friday
- LDAP guide updated to add systemd to shadow for NSS. This was breaking the graphical boot environment on Mitsuki's laptop.
- write-up
2026-07-16 Thursday
- router updates
  - Remote IPv6 was not updated in public DNS. The router update script was failing to execute in cron
    - shebang was /usr/local/bin/bash but
    - cron executed /bin/ksh $script... (just openbsd things!)
    - fixed
  - also router IPv6 update script updated to use persistent storage for IPv6 address as a reboot would remove the older address
  - bad firewall rule forwarded all IPv6 traffic arriving on 22 to ul
    - fixed
2026-07-13 Monday
- created ldap enable doc and test on desktop
  - works!
- completed SSH cert setup for deskop
  - works!
- drafted a process to auto generate ssh keys and sign them with authentik
2026-07-12 Sunday
- created "admin" specific firefox profile on desktop
- cleared all old disparate passwords for the old individual service passwords
- fixed www server for mitsuki
- checked in kanji service in gitea
  - added bounding boxes
  - changed the stroke order color to be more legible and not run into the background
2026-07-11 Saturday
- created this server change log
- added "backup_all" script to run each service backup
- finally checked in backup/restore scripts
- separated backup output into current and all previous to simplify disk backup
- modified nginx config for immich, upload cap 30GB -> 100GB