System Setup System Change log 2026-10-20 Tuesday Forgejo Two-Factor Authentication ( git.drheck.dev) Cipher Migration Fix ( ul): Resolved HTTP 500 Internal Server Error on POST /user/two_factor ( UserSignIn: chacha20poly1305: message authentication failed) caused by legacy Gitea AES-CFB ( md5(SECRET_KEY)) + double-Base64 ciphertext remaining in two_factor.secret after the Gitea $\longrightarrow$ Forgejo 16.0.6 migration. Decrypted the existing 64-character Base32 TOTP seed without requiring authenticator re-enrollment, re-sealed two_factor.secret using Forgejo 16.0.6's modules/keying HKDF-SHA256 ( totp subkey) + XChaCha20-Poly1305 AEAD with ColumnAndID("secret", id) big-endian associated data, verified all 27 forgejo doctor check --all checks ( OK), and refreshed PAR2 backups. Docker Package Removal, Root NVMe Reclamation & ZFS data/podman Dataset Migration ( ul): Uninstalled docker ( 1:29.9.1-1), docker-compose ( 5.6.0-1), and containerd ( 2.4.2-1) via pacman -Rns and purged /var/lib/docker, /var/lib/containerd, and /etc/docker ( 43 GB freed on /). Provisioned dedicated ZFS dataset data/podman ( compression=lz4, xattr=sa, acltype=posix, dnodesize=auto, atime=off, mountpoint=/home/docker-data/.local/share/containers) and migrated Podman's 67 GB image/layer store from the root NVMe onto the 160 TB ZFS pool via two-pass rsync -aHAX. Cleaned 39 GB of obsolete package tarballs from /var/cache/pacman/pkg ( paccache -rk2 + paccache -ruk0) and enabled weekly paccache.timer. 2026-10-19 Monday Complete 30-Service / 40-Container Rootless Podman Cutover & Docker Decommissioning ( ul): Decommissioned docker.service, docker.socket, and containerd.service completely ( inactive / disabled), removed all 10 leftover Docker kernel bridges ( docker0 + br-*), archived legacy artifacts into /data/backups/legacy_docker_archive/, and replaced docker-prune.timer with rootless /etc/systemd/system/podman-prune.{service,timer} ( User=docker-data). Migrated all remaining 19 services (29 containers) alongside the initial 11 standalone services—totaling all 30 containerized services (41 containers)—to rootless Podman Quadlets ( docker-data, UID 1100, subordinate range docker-data:624288:65536, Pull=never offline boot resilience, /home/docker-data/.config/containers/systemd/*.{container,network}). Migrated shared Valkey to native host valkey.service bound strictly to 127.0.0.1:6379 with per-service ACL users ( authentik, immich, nextcloud, vikunja, degoog), default disabled, and dangerous admin commands disabled; locked host postgresql.service strictly to 127.0.0.1:5432 and upgraded the legacy MD5 role password to SCRAM-SHA-256. Configured host HAProxy Layer-4 TCP passthrough ( /etc/haproxy/conf.d/ldap.cfg) relaying ports 389/ 636 to rootless authentik-ldap ( :3389/ :6636) and removed /etc/nginx/certs:/certs:ro from all Authentik containers so zero host TLS/mTLS keys ( pikvm_client.key, studio_secret.conf) are mounted inside containers. Remediated security & operational findings uncovered during migration: Enabled YOURLS_PRIVATE=true, rotated weak yourls credentials ( admin_pass_9213 / yourls_db_pass_9213), and gated https://drheck.dev/yourls-api.php behind Authentik ForwardAuth while keeping short-link 301 redirects public. Removed /home/luke:/host_home:ro mount from subocr, restricted Collabora WOPI wildcard https://.* strictly to cloud\.drheck\.dev, set DISABLE_SIGN_UP=true and 0770 permissions on wipstudio, and archived the deleted upstream minio/minio image at /data/docker/wipstudio/minio_image_backup.tar.xz. Fixed silent data-loss gaps in /home/luke/git/backups/ul/scripts/: pointed backup/nextcloud + restore/nextcloud at live nextcloud_db (instead of stale nextcloud) + config.php; migrated penpot Postgres & assets out of /var/lib/docker/volumes onto ZFS /data/docker/penpot/{postgres_data,assets} and added assets to backup/penpot; added pg_dumpall -U funkwhale to backup/funkwhale; added authentik-*.container to backup/authentik; and expanded backup/systemd to archive /etc/containers, /etc/haproxy, /etc/nftables.conf, and /home/docker-data/.config/containers/systemd. Fixed HTTP 500 ForwardAuth errors on secondary hostname aliases ( music.drheck.dev $\longrightarrow$ navidrome.drheck.dev, scontroll.drheck.dev $\longrightarrow$ scontrol.drheck.dev) and OIDC callback host mismatches ( vault.drheck.dev $\longrightarrow$ pw.drheck.dev, bookstack.drheck.dev $\longrightarrow$ wiki.drheck.dev) with dedicated TLS 301 redirects across /etc/nginx/sites-available/. Pruned a stale aardvark-dns entry in /run/user/1100/containers/networks/aardvark-dns/penpot_net ( 10.89.13.2 ahead of 10.89.13.5) that caused penpot-backend Lettuce Redis timeouts, fixed root-owned cache generation in /home/luke/bin/read-bookstack-page ( -u abc -e HOME=/config), and updated /usr/local/bin/{update-authentik-cert.sh,mc-bluemap-provision,degoog-update}, /data/minecraft/ipctl, /etc/nftables.conf, and all 43 PAR2-verified service backups. Backup Architecture Decoupling ( ul): Refactored /home/luke/git/backups/ul/scripts/run_all_backups so default invocations execute fast local snapshots and par2 verification only ( --all runs the full local + cloud + ISO pipeline). Extracted standalone /home/luke/git/backups/ul/scripts/stage_cloud_archive, /home/luke/git/backups/ul/scripts/upload_cloud_archive, and /home/luke/git/backups/ul/scripts/stage_iso_archive scripts and updated Backup Process. amux Synchronous Command Waiting ( ul): Implemented native event-driven command completion waiting in amux ( /home/luke/git/amux, installed at /usr/local/bin/tmux-custom) via read-pane -w ( -S , -e , -E , -O , -t ), eliminating polling loops. Added atx-exec helper to /home/luke/.agent_profile and updated amux - Agent modified tmux and SSH & Agent Command Tracking Guide. Gitea $\longrightarrow$ Forgejo Migration ( ul): Migrated git.drheck.dev from Gitea ( 28.1.0-1) to Forgejo ( 16.0.6-1), renaming ZFS dataset data/gitea to data/forgejo ( /data/forgejo), PostgreSQL database/role to forgejo, configuration to /etc/forgejo/app.ini, and working state to /var/lib/forgejo ( User=git). Added /etc/tmpfiles.d/forgejo.conf ( git:git override) to prevent pacman 30-systemd-tmpfiles.hook from resetting /var/lib/forgejo and /etc/forgejo/app.ini ownership to forgejo:forgejo during package upgrades. Updated Nginx ( /etc/nginx/sites-available/forgejo), Authentik OIDC provider/application ( Forgejo / forgejo), pass entries ( forgejo/*), repo_tools ( forgejo-create, forgejo-delete, forgejo-Visibility, gpush), and backup/restore scripts ( /home/luke/git/backups/ul/scripts/{backup,restore}/forgejo). Kasm Decommissioning & Rootless Podman Quadlet Migration ( ul): Decommissioned kasm ( kasm.drheck.dev, /data/docker/kasm, PostgreSQL kasm_db, ULA IPv6 :ca53, Authentik OIDC app, Nginx config, Unbound DNS, and /data/kasm_profiles) and removed obsolete /data/docker/sunshine container stack. Configured rootless Podman ( 6.1.4-1) with passt user-mode networking, subordinate UID/GID ranges ( docker-data:100000:65536), ZFS-backed graphroot ( /data/podman/docker-data/storage), and systemd user lingering ( loginctl enable-linger docker-data). Migrated 11 standalone single-container services ( audiomass, bentopdf, excalidraw, tldraw, wick, webrcade, drawpile, navidrome, vaultwarden, sopro, demucs) from rootful Docker Compose to rootless Podman systemd Quadlets ( /home/docker-data/.config/containers/systemd/*.container). Updated /home/luke/bin/status_all, backup/restore scripts, Service List, Service Update Process, and authored podman. 2026-10-10 Saturday Minecraft Server Control Plane, Live Console, Seed Studio & Text/Icon Studio ( ul, danknasty): Built and deployed mc-manager ( /usr/local/bin/mc-manager, mc-manager.service), mc-runner ( /usr/local/bin/mc-runner, minecraft@.service PTY wrapper with /run/mc-manager/.sock), and mc-seed-render ( /usr/local/bin/mc-seed-render, native cubiomes biome & structure map generator) at mc.drheck.dev behind Nginx and Authentik ForwardAuth. Added Minecraft Text, MOTD, Color Gradient & 8-Bit Pixel Icon Studio to both the + New World wizard and existing server Configuration panel (Unicode font styles, Java-safe symbols, § RGB gradient interpolator, live server-list preview, and 64×64 8-bit pixel badge/icon generator). Updated /etc/haproxy/minecraft.lua and HAProxy backend rules so all Multiplayer Server List pings ( state == 1) query mc-manager ( BuildLiveStatusJSON), overlaying live motd ( server.properties) and favicon ( server-icon.png) onto running server status responses without requiring a server restart. Updated /data/minecraft/ipctl to authenticate programmatically to danknasty using short-lived (5-minute) SSH user certificates issued by step-ca ( webshell-svc provisioner) and trusted via /etc/sshd_config.d/10-drheck-certs.conf on danknasty. Externalized all dynamic ipctl state out of code-reviewed base configs into /etc/systemd/network/20-wired.network.d/minecraft.conf, /etc/haproxy/conf.d/minecraft_backends.cfg, /etc/haproxy/minecraft_hosts.map, and /var/unbound/etc/unbound.minecraft.conf (included from unbound.conf and parsed recursively by update_dns.py, dns_danger_sync.py, and verify_dns.sh), with tracked mirrors in /data/minecraft/.network/. Provisioned desertworld.drheck.dev ( fd77:7777:5010::1337:3c:d00d:2/64) and updated Minecraft Worlds and Service List. 2026-09-27 Sunday amux (Agent-Modified tmux) & Persistent Session Infrastructure ( ul): Created custom tmux fork ( amux) built from /home/luke/git/tmux and installed at /usr/local/bin/tmux-custom. Implemented native write-pane ( writep) command supporting -b and -f with -e (auto-enter) to inject scripts directly into PTY buffers with zero quote escaping or keystroke lag. Built structured audit logging directly into cmd-write-pane.c using sd_journal_send ( SYSLOG_IDENTIFIER=amux, AMUX_PANE_ID, AMUX_WINDOW_NAME, AMUX_ARCHIVE_PATH). Configured automatic pre-execution script archival to persistent storage in /data/amux/scripts/ ( YYYY-MM-DD_HHMM.SS_.sh). Configured dedicated atx agent session in /home/luke/bin/tmux_control on socket /tmp/tmux-1000/tmux_luke_atx, styled with orange status bar ( #ff8800), C-b prefix, and persistent agent environment in ~/.agent_profile. Added backup and restore scripts for amux ( /home/luke/git/backups/ul/scripts/backup/amux and /restore/amux). Documented architecture and audit procedures on amux - Agent modified tmux and SSH & Agent Command Tracking Guide. Minecraft Datapack Enhancements ( pokelovers) ( ul): Installed 5 verified Minecraft 26.3 datapacks in /data/minecraft/poke lovers/world/datapacks/: fast_leaf_decay (rapid leaf decay upon log breakage) graves (death chest/grave saving inventory and XP) invisframes (invisible item frames via shears while sneaking) player-drops-head (player head drops on death) shulker-drops-two-shells (guaranteed 2 shulker shells) Broadcasted in-game countdown notices to online players prior to executing live /reload. BlueMap 3D Web Map & Authentik SSO Integration ( pokelovers) ( ul): Deployed BlueMap 5.28 standalone CLI renderer running as bluemap-pokelovers.service in watch mode ( -w -u), serving live world updates and Server-Sent Events (SSE) on port 8100. Configured NGINX reverse proxy on port 443 with wildcard SSL ( *.drheck.dev) in /etc/nginx/sites-available/pokelovers. Gated https://pokelovers.drheck.dev behind Authentik ForwardAuth proxy provider and application on embedded outpost, protecting map coordinates and player builds behind authentication. 2026-09-26 Saturday Zero-Config Minecraft HAProxy SNI Stream Routing ( ul, danknasty): Decommissioned application-level proxy (Velocity) to preserve native vanilla Diffie-Hellman encryption and Microsoft online-mode=true authentication. Deployed HAProxy 3.4.5 on dual-stack endpoints ( 10.0.0.113:25565 and [fd77:7777:5010::1337:f1ac:d00d:3c]:25565). Implemented /etc/haproxy/minecraft.lua using non-draining buffer peeking ( txn.req:dup()) to inspect the raw unencrypted Minecraft handshake packet ( 0x00), dynamically extracting the SNI virtual host ( pokelovers.drheck.dev). Routed traffic via /etc/haproxy/minecraft_hosts.map directly to dedicated internal ULA addresses on standard port 25565 ( [fd77:7777:5010::1337:3c:d00d:1]:25565). Implemented native fallback responder ( mc_offline) returning custom protocol packets when backends are stopped: Multiplayer Server List MOTD: "[Server] is currently offline. Ping Luke" Multiplayer Join Screen: Immediate disconnect with red bold error: "[Server] is currently offline. Ping Luke" Network & DNS Automation ( ipctl) ( ul, danknasty): Developed /data/minecraft/ipctl (symlinked in /home/luke/bin/ipctl). Automates ULA IPv6 allocation ( fd77:7777:5010::1337:3c:d00d:/64) on eth77 with systemd-networkd reload. Automates HAProxy backend and map configuration reload. Integrated SSH agent validation checking for key ultra-arikui ( SHA256:7Xr0vCxgfpPhbKE3hN9fE2IC9y4x3Kj+kYmXwzgJY3s). Implemented 7-step automated DNS provisioning over SSH to danknasty: updates ~/git/router_config/unbound.conf, validates via unbound-checkconf, deploys to /var/unbound/etc/, restarts unbound, verifies local resolution, and triggers dns_danger_sync.py --force to push public A and AAAA records to deSEC.io. Created dedicated backup and restore scripts in /home/luke/git/backups/ul/scripts/backup/minecraft_router and /restore/minecraft_router. Documented entire architecture and quickstart HOWTO on HeckWiki page 136 (Minecraft Worlds). Firewall Configuration for Minecraft Port 25565 ( danknasty): Configured mc_ipv4 ( 10.0.0.113), mc_gu ( $gu_prefix:1337:f1ac:d00d:3c), and mc_ula ( $ula_prefix:1337:f1ac:d00d:3c) in ~/git/router_config/pf.conf. Added inbound port 25565 forwarding rules for IPv4 NAT and IPv6 ULA redirection. Validated with pfctl -nf, deployed to /etc/pf.conf, loaded into the kernel, and verified reachability externally from rng. 2026-09-21 Monday Native Sunshine & Gamescope Migration on ul: Decommissioned legacy steam-headless container; migrated to bare-metal Gamescope DRM compositor + Sunshine host on AMD Radeon AI PRO R9700 ( 1002:7551, Navi 48). Injected AOC 1080p EDID on DP-1 via mkinitcpio & systemd-boot loader options. Configured Steam library paths directly to 151TB ZFS pool ( /data/games/steam/steamapps). Resolved PS5 DualSense controller disconnection: disabled Steam proprietary HIDAPI polling ( SDL_JOYSTICK_HIDAPI_PS5=0, SDL_JOYSTICK_HIDAPI=0) to prevent hid_read failure timeout on virtual uhid devices. Added -e ( --steam) integration flag to Gamescope launch script to ensure proper Wayland focus relaying to Steam, enabling Steam Input in-game instead of dropping to Desktop mouse layout. Documented complete architecture, launch scripts, and controller gotchas on HeckWiki page 130 (Sunshine Remote Setup). 2026-09-21 Monday Steam Storage Migration to 151TB ZFS Pool ( /data/games/steam) ( ul): Discovered root NVMe filesystem at 85% utilization (71GB free). Consolidated all Steam games, runtimes, manifests, and shader caches from temporary docker volumes to /data/games/steam/steamapps/ on the 151TB ZFS pool. Replaced ~/.local/share/Steam/steamapps and ~/.steam/steam/steamapps with permanent atomic symlinks pointing to /data/games/steam/steamapps/. Cleaned libraryfolders.vdf ensuring 100% of game downloads route directly to /data/games/steam/ with zero bytes hitting the root SSD. TTY1 Getty Deactivation & Display Console Fix ( ul): Disabled getty@tty1.service ( systemctl disable --now getty@tty1.service) to prevent Linux console login prompts from broadcasting over KMS on DP-1. 2026-09-20 Sunday PS5 DualSense Controller Hardware Support & uhid Activation ( ul): Identified PlayStation 5 DualSense controller emulation failure in Sunshine due to missing /dev/uhid (User HID) device node. Loaded uhid kernel module and made persistent across reboots in /etc/modules-load.d/uhid.conf. Configured udev rule in /etc/udev/rules.d/85-sunshine-uinput.rules granting input group permissions to both /dev/uinput and /dev/uhid ( MODE="0660", GROUP="input"). Native Gamescope + Headless DRM Stack Migration ( ul): Decommissioned legacy steam_headless Docker container to eliminate Xorg CPU dummy-driver bottleneck ( xf86-video-dummy) and software rasterization (15 FPS). Installed native Arch Linux gaming stack: gamescope, steam, vulkan-radeon, lib32-vulkan-radeon, pipewire-pulse, wireplumber, seatd, socat, and sunshine-bin (AUR). Granted real-time scheduling capabilities to Gamescope ( setcap cap_sys_nice=eip /usr/bin/gamescope) and rootless DRM frame capture to Sunshine ( setcap cap_sys_admin,cap_sys_nice=eip /usr/bin/sunshine). Configured seatd daemon with group video override in /etc/systemd/system/seatd.service.d/group.conf to allow Gamescope DRM session management without an active logind seat. Configured Gamescope launcher script /home/luke/bin/launch-gamescope-steam.sh driving connector DP-1 via --backend drm with RADV Vulkan acceleration on the AMD Radeon AI PRO R9700. Headless Virtual Display via Boot-Time EDID Injection ( ul): Resolved *ERROR* No EDID found on connector: DP-1 kernel spam on headless AMD GPU. Sourced verified 1080p 144Hz CEA-861 DisplayPort EDID ( aoc_1080p.bin) in /usr/lib/firmware/edid/. Packaged EDID into early initramfs via FILES=(/usr/lib/firmware/edid/aoc_1080p.bin) in /etc/mkinitcpio.conf and regenerated boot image via mkinitcpio -P. Configured bootloader kernel parameters in /boot/loader/entries/arch.conf: drm.edid_firmware=DP-1:edid/aoc_1080p.bin video=DP-1:1920x1080@60e. Pinned Sunshine KMS capture explicitly to adapter_name = /dev/dri/card1 and output_name = DP-1, preventing BMC graphics capture ( VGA-1). Dual-Stack IPv4/IPv6 socat Bridge ( ul): Bound native Sunshine directly to dedicated IPv6 ULA address [fd77:7777:5010:0:1337:f1ac:d00d:4799]. Deployed /usr/local/bin/sunshine-ipv4-bridge.sh and systemd unit /etc/systemd/system/sunshine-ipv4-bridge.service listening on 10.0.0.112 and forwarding all GameStream TCP and UDP ports bidirectionally to IPv6 to prevent asymmetric routing drops on dual-stack LANs. Client Authentication & Stale Certificate Remediation ( ul): Diagnosed Warning: SSL Verification error :: Client certificate identity is not enabled (LizardByte issue #5696) caused by duplicate client certificates with conflicting UUIDs in sunshine_state.json. Cleared stale device records in sunshine_state.json allowing clean one-time client pairing across desktop and TV clients. NetworkManager D-Bus Mock for Steam ( ul): Diagnosed Steam Gamepad UI offline status ( An internet connection is required to load this content / slashed Wi-Fi icon) caused by absence of NetworkManager on systemd-networkd host. Implemented lightweight Python D-Bus daemon /usr/local/bin/fake-networkmanager.py and service /etc/systemd/system/fake-networkmanager.service reporting NM_STATE_CONNECTED_GLOBAL (70) and NM_CONNECTIVITY_FULL (4) to satisfy Steam network checks. Documentation: Authored and published comprehensive architecture and setup runbook on HeckWiki: sunshine remote setup (Page 130). 2026-09-19 Saturday Sunshine Network Tuning & Steam Container Bootstrap ( ul): Diagnosed container web assets and API communication issues on josh5/steam-headless. Configured Sunshine CORS origins ( csrf_allowed_origins = https://sunshine.drheck.dev) and dual-stack network settings in sunshine.conf. Bootstrapped Steam runtime inside container and configured Steam Big Picture startup commands. Authored and published initial HeckWiki guide: Steam Streaming (Page 129). Updated LDAP group enrollment documentation ( ldap-sssd-client-enrollment-guide, Page 25) with corrected repository clone instructions. 2026-09-18 Friday Immich Machine Learning AMD ROCm Migration ( ul): Migrated immich_machine_learning from NVIDIA CUDA to native AMD ROCm: pulled ghcr.io/immich-app/immich-machine-learning:v3.1.0-rocm. Updated /data/docker/immich/.env ( IMMICH_MACHINE_LEARNING_GPU_TYPE=rocm). Recreated Immich stack and verified PyTorch ROCm device detection ( torch.cuda.is_available() = True on AMD Radeon AI PRO R9700 gfx1201). Verified VA-API hardware video acceleration on /dev/dri/renderD128 inside immich_server. Sunshine & Steam-Headless Initial Deployment ( ul & danknasty): Provisioned dedicated internal IPv4 ( 10.0.0.112) and IPv6 ULA ( fd77:7777:5010:0:1337:f1ac:d00d:4799). Updated unbound.conf in ~/git/router_config on danknasty with sunshine.drheck.dev A and AAAA records; deployed via ./sync_unbound. Synced public DNS records via dns_danger_sync.py on deSEC.io. Configured /etc/nginx/sites-available/sunshine with Authentik ForwardAuth and deployed reverse proxy. Provisioned sunshine application and ProxyProvider in Authentik embedded outpost. Deployed josh5/steam-headless container in /data/docker/sunshine/docker-compose.yml. 2026-09-17 Thursday Tmux & SSH Auth Environment Maintenance ( ul): Re-initialized Tmux Plugin Manager ( tpm) and checked plugin compatibility ( check_tmux_version.sh). Maintained persistent SSH agent socket symlinks. 2026-09-15 Tuesday System Package Maintenance & Initramfs Rebuild ( ul): Upgraded system packages via paru -Syu. Rebuilt initramfs-linux-lts.img via ALPM hooks. Verified systemd service health. 2026-09-12 Saturday System Package Maintenance & LTS Kernel Update ( ul): Upgraded system packages via paru -Syu / pacman -Syu. Kernel update to 6.18.52-1-lts and regenerated initramfs-linux-lts.img. Re-verified NVIDIA kernel module conftest compilation checks for 580.178.04. Media library maintenance: sorted and updated Adventure Time subtitle filenames. 2026-09-08 Tuesday - 2026-09-11 Friday System Maintenance & Driver Toolchain Compilation ( ul): Executed routine system maintenance and package updates across ul. Kernel module conftest compilation and pacman keyring verification. 2026-09-05 Saturday OpenBSD Router Reinstallation & CrowdSec Perimeter Restoration ( danknasty): Reinstalled and configured crowdsec-firewall-bouncer connecting to ul's LAPI over IPv6 ULA ( http://[fd77:7777:5010:0:1337:f1ac:d00d:c70d]:8080/). Restored OpenBSD PF firewall tables ( , , ) with rate limiting ( max-src-conn-rate 3/30), loading ~23,880 blacklisted attacker IPs directly into kernel tables. Verified remote syslog forwarding of auth,authpriv.info over IPv6 UDP to ul's CrowdSec syslog parser. Router Network Mapping & Dynamic IPv6 DNS Cron Jobs ( danknasty): Aligned network interface roles after hardware reinstallation ( em0 WAN, em1 LAN with delegated /64 prefix). Updated interface configurations across /etc/hostname.em0, /etc/hostname.em1, /etc/dhcp6leased.conf, /etc/rad.conf, and /etc/dhcpd.conf. Deployed dynamic deSEC.io IPv6 DNS updater ( /etc/update_dns.py) and keepalive monitor ( /etc/ipv6_keepalive.sh). Restored root crontab on danknasty to run /etc/update_dns.py every minute and /etc/ipv6_keepalive.sh every 5 minutes. Triggered manual sync: verified all 43 domain AAAA records updated on deSEC and verified public IPv6 reachability via rng to https://auth.drheck.dev/. Router Disaster Recovery & Reinstall Survival Automation ( danknasty): Created single-command idempotent provisioning script ~/git/router_config/bootstrap.sh ( doas ./bootstrap.sh): installs package dependencies, copies system configs, sets file permissions and ownership, restores root crontab, applies sysctl, and starts all services. Created zero-touch OpenBSD post-install hook ~/git/router_config/install.site to automatically pull repository and execute bootstrap.sh inside chroot /mnt at OS installation completion. Added /etc/sysctl.conf ( net.inet.ip.forwarding=1, net.inet6.ip6.forwarding=1) to repo with automated activation via sysctl -f. Added package manifests to repo: pkglist (generated via pkg_info -mz for native OpenBSD pkg_add -I -l restoration) and packages.txt (human-readable catalog with descriptions). Added /etc/dhcpd.conf (DHCPv4 server) and /var/unbound/etc/unbound.conf (DNS resolver) deployment and service management into bootstrap.sh. Backed up all router configuration templates into ~/git/router_config ( boot.conf, ttys, doas.conf, crontab.root, dhcp6leased.conf, rad.conf, pf.conf, syslog.conf, rc.conf.local, mygate, crowdsec-firewall-bouncer.yaml). Nextcloud Landing Page & occ Runbook Guide ( ul): Configured post-authentication redirect on cloud.drheck.dev to default directly to /apps/files/files ( occ config:system:set defaultapp files, occ config:app:set core defaultpage /apps/files/files). Authored and published comprehensive Nextcloud occ Guide on HeckWiki documenting command-line administration, alias setups, user management, and maintenance mode operations. 2026-09-04 Friday Danknasty Router Hardware & Platform Diagnostics ( danknasty): Identified router hardware platform: Broachlink NOAH4C V10 (Intel Atom E3845 quad-core @ 1.91GHz, 4GB RAM, 4x Intel I210 Gigabit Ethernet ports em0- em3). Enabled RJ45 serial console access on com0 at 115200 baud via /etc/boot.conf ( stty com0 115200, set tty com0) and /etc/ttys ( tty00 std.115200 vt220 on secure). Diagnosed periodic SSH responsiveness latency: traced kernel system CPU lockups and interrupt activity using vmstat -i, sensorsd, and bsd.re-config. Split-Horizon Unbound Authoritative LAN Resolution ( danknasty): Analyzed Unbound configuration on danknasty acting as authoritative LAN resolver ( local-zone / local-data with aa flag set in LAN responses). Evaluated DNS high-availability options: Git-based Unbound configuration replication vs authoritative secondary migration to NSD. 2026-09-03 Thursday SSH CA Documentation Modularization (HeckWiki): Cleanly partitioned SSH Certificate Authority documentation into two focused reference runbooks under the LDAP+PAM+SSH CA book: SSH CA setup (Page 26): Core architecture, Ed25519 CA keys, Vaultwarden storage, signing principles, and Certificate Revocation Lists (KRL). SSH Client and Host Onboarding Guide (Page 112): Step-by-step out-of-band bootstrap and verification workflows for onboarding new clients and servers without circular SSH dependencies. 2026-09-02 Wednesday OpenBSD Router IPv6 Dynamic DNS Rewrite ( update_dns.py) ( danknasty): Diagnosed HTTP 400 bulk update failures on deSEC.io caused by regex string parsing errors and duplicate records in unbound.conf. Replaced legacy bash/awk script ( update_dns.sh) with robust Python 3 implementation ( update_dns.py) using Python's native ipaddress library and typed dictionary deduplication. Added unit test suite test_update_dns.py with 9 unit tests verifying zone record parsing, address synthesis, and deduplication. Removed duplicate libresprite.drheck.dev entry and cleaned up interleaved records in unbound.conf. Deployed /etc/update_dns.py (0755 root:wheel), removed /etc/update_dns.sh, and signed/committed all changes to ~/git/router_config. CrowdSec Log Synchronization & False-Positive Ban Remediation ( ul & danknasty): Investigated repetitive /var/log/authlog session spam on danknasty. Identified redundant SSH polling in crowdsec-sync-authlog on ul running via crowdsec-export-bans.timer every 60s. Disabled unnecessary SSH scraping, relying on native real-time UDP IPv6 syslog forwarding ( auth,authpriv.info) from danknasty to CrowdSec on ul. Resolved false-positive ban on VPS rng ( 66.186.37.205) in PF table caused by unprivileged TCP connect scans. SSH CA Inventory Tracking & Streamlined Key Signing ( ul): Implemented key archival and inventory management in /usr/local/bin/sign-host-key and /usr/local/bin/sign-user-key archiving signed public keys in /etc/ssh/ca_inventory/{hosts,users} with mandatory labeling for revocation tracking. Built revocation utility /usr/local/bin/revoke-ssh-key. Documented out-of-band onboarding workflow on HeckWiki. 2026-08-31 Monday System Package Maintenance & Toolchain Updates ( ul): Executed system upgrades via paru / pacman. Updated Haskell libraries ( haskell-uuid-types, haskell-vector, haskell-witherable), shellcheck, and system info documentation. Updated Opencode CLI, csvtools, and nethack-git. 2026-08-30 Sunday Demucs Stem Separation Deployment ( https://demucs.drheck.dev/): Added demucs.drheck.dev DNS records to unbound.conf on danknasty ( ~/git/router_config). Built and deployed container running Demucs v4 web service ( app.py, HTML5 Multi-Track Mixer) on dedicated ULA IPv6 ( fd77:7777:5010:0:1337:f1ac:d00d:de3:8000). Supports htdemucs_ft (Fine-tuned 4-Stem), htdemucs (4-Stem), htdemucs_6s (6-Stem with Piano & Guitar), and mdx_extra. Features synchronized multi-track audio playback with per-stem Solo, Mute, Volume faders, and ZIP export. Provisioned Authentik ForwardAuth demucs ProxyProvider and Application in Authentik Embedded Outpost. Configured /etc/nginx/sites-available/demucs reverse proxy with 512MB max body size and 600s read timeout. Added firewall isolation reject rule in /etc/nftables.conf. Created demucs backup and restore scripts with par2 parity in ~/git/backups. 2026-08-29 Saturday Sopro V2 TTS Deployment ( https://sopro.drheck.dev/): Enhanced web interface and backend API to expose configurable acoustic solver Steps (1-64 steps). Added sopro.drheck.dev DNS records to unbound.conf on danknasty ( ~/git/router_config). Built and deployed container running soprotts serve --device cpu --host 0.0.0.0 --port 7860 --no-open on dedicated ULA IPv6 ( fd77:7777:5010:0:1337:f1ac:d00d:509:7860). Provisioned Authentik ForwardAuth sopro ProxyProvider and Application in Authentik Embedded Outpost. Configured /etc/nginx/sites-available/sopro reverse proxy with WebSocket support and pushed to ~/git/nginx. Added firewall isolation reject rule in /etc/nftables.conf and pushed to ~/git/configs. Created sopro backup and restore scripts with par2 parity in ~/git/backups. BentoPDF Deployment ( https://bentopdf.drheck.dev/): Added bentopdf.drheck.dev DNS records to unbound.conf on danknasty ( ~/git/router_config). Deployed ghcr.io/alam00000/bentopdf:latest container on dedicated ULA IPv6 ( fd77:7777:5010:0:1337:f1ac:d00d:bdf:8080). Provisioned Authentik ForwardAuth bentopdf ProxyProvider & Application in Authentik Embedded Outpost. Configured /etc/nginx/sites-available/bentopdf reverse proxy and pushed to ~/git/nginx. Added firewall isolation reject rule in /etc/nftables.conf and pushed to ~/git/configs. Created bentopdf backup and restore scripts with par2 parity in ~/git/backups. SSL Certificate Automation & SSSD TLS: Fixed expired LDAP CertificateKeyPair inside Authentik database via ak shell and restarted authentik-ldap. Deployed /usr/local/bin/update-authentik-cert.sh automated renewal hook in acme.sh configuration with /etc/sudoers.d/acme_ssl NOPASSWD grant. Docker .env & Secret File Hardening: Stripped POSIX extended ACLs ( setfacl -b) across all .env files and SSH keys in /data/docker/. Restricted permissions to 0640 root:docker-data and parent directories to 0750. Universal Admin Account Transition: Transferred administrative roles to akadmin / admin across Authentik, Gitea, Immich, Anchor, BookStack, Nextcloud, Kavita, Navidrome, and RomM. Demoted luke to standard user on web applications while preserving wheel (sudo) access on host OS. PostgreSQL Authentication Hardening: Removed obsolete 0.0.0.0/0 and ::/0 wildcard rules for non-existent infisical role from /data/postgres/data/pg_hba.conf. nftables Egress Isolation ( chain output): Added reject rules ( meta skuid != { 0, 33, 1101 } reject) in chain output for all 26 active ULA IPv6 backend services to prevent unprivileged local processes on ul from bypassing Authentik ForwardAuth. Tracked /etc/nftables.conf in ~/git/configs/nftables.conf. Nginx Hash Table Performance Tuning: Added variables_hash_max_size 2048; and variables_hash_bucket_size 128; to /etc/nginx/nginx.conf and pushed to ~/git/nginx. Authentik 2026.8.0 Outpost & OIDC Remediation: Populated missing authentication_flow ( default-authentication-flow) and invalidation_flow ( default-invalidation-flow) UUIDs across all 19 ProxyProvider database records, fixing Rust Embedded Outpost deserialization crash and restoring ForwardAuth across all services. Updated OpenID email ScopeMapping expression to enforce email_verified = True for Better-Auth (WIPStudio) / SSO client logins. Cockpit Removal: Uninstalled cockpit and cockpit-docker packages; purged /etc/motd.d/cockpit, /etc/issue.d/cockpit.issue, and /run/cockpit. Arch .pacnew Merge & Maintenance: Adopted updated upstream makepkg drop-ins and TPM2 FAPI JSON profiles. Tuned /etc/pacman.conf with DownloadUser = alpm and ParallelDownloads = 5. Cleaned all lingering .pacnew files in /etc/. BMO95 Display Mirroring ( ~/git/audio_config): Installed wl-mirror on bmo95 and configured Sway ( common/sway/config) to mirror DisplayPort DP-3 to HDMI HDMI-A-1 (TV) via zero-copy DMA-BUF. Consumed Sway configuration into ~/git/audio_config and committed changes. Studio Control ( scontrol) UI & Toggle Controls: Added POST /api/power/{device}/{action} supporting individual Sub-GHz RF outlet toggling ( pa, sub, rack, computer, tv, neewer, mrc, all) and POST /api/atx/{action} for PiKVM motherboard ATX button clicks. Added Individual Power Controls view with dedicated toggle buttons and view-switcher button to scontrol.drheck.dev. Checked in and pushed to https://git.drheck.dev/luke/scontrol.git. Kavita Arch Linux / AUR Upgrade & Database Migrations ( ul): Checked SQLite database integrity on /opt/kavita/config/kavita.db ( PRAGMA integrity_check). Created safety database backup at /opt/kavita/config/kavita.db.pre_0.9.1_upgrade. Restarted kavita.service following paru upgrade to Kavita v0.9.1.0. Executed automatic backup ( /opt/kavita/config/temp/migration/0.9.0.2) and applied database migrations ( MetadataSettingFieldRenumber, ManualMigrateOriginalNameBackfill, ManualMigrateNormalizedOriginalNameBackfill, ManualMigrateLastReadDates, ManualMigrationExternalSeriesMetadataIds, ManualMigrationCorrectAuditStatus). Verified service initialization and HTTP 200 on http://[fd77:7777:5010:0:1337:f1ac:d00d:7ead]:5000/. Full Docker Fleet Update & Service Migrations ( ul): Authentik: Created PostgreSQL database backup at /tmp/authentik_pre_upgrade.dump. Upgraded server, worker, and ldap to 2026.8.0. Applied blueprint and Django migrations across all applications. Verified email_verified: True in ak shell scope mapping and tested https://auth.drheck.dev/. Infra (Valkey): Upgraded shared Redis/cache container to valkey/valkey:8.1-alpine (v8.1.9). Verified active session reconnection for Authentik, Immich, and Nextcloud. Immich: Upgraded immich-server and immich-machine-learning to latest v3.1.0 builds with GPU acceleration. Verified https://photos.drheck.dev/. Kasm Workspaces: Upgraded manager container to lscr.io/linuxserver/kasm:latest (Kasm 1.19.0). Initialized internal services and verified https://kasm.drheck.dev/. BookStack: Upgraded bookstack (v26.05.4) and internal mariadb:latest. Verified Laravel migrations and https://wiki.drheck.dev/. Penpot: Upgraded frontend, backend, exporter (v2.17.2), postgres:15.19, and redis:7. Verified https://penpot.drheck.dev/. Anchor: Upgraded image tag to ghcr.io/zhfahim/anchor:0.16.0. Executed database schema migrations and verified https://keep.drheck.dev/. Vaultwarden: Upgraded to vaultwarden/server:latest (v1.37.2). Verified https://pw.drheck.dev/. Navidrome: Upgraded to deluan/navidrome:latest (v0.63.2). Verified web and Subsonic API endpoints on https://navidrome.drheck.dev/ and https://music.drheck.dev/. WebShell / Step-CA: Upgraded smallstep/step-ca:latest (v0.30.2). Verified HTTPS on port 9000. WIPStudio / MinIO: Upgraded storage backend to minio/minio:latest. YOURLS: Upgraded yourls:latest (1.10.5) and internal mariadb:10.11. Drawpile: Upgraded server to drawpile/drawpile-srv:2 (2.3.0). Verified TCP/WebSocket listeners on ports 27750-27752. Stateless Web Services: Upgraded wick ( nginx:alpine), webrcade ( webrcade:latest), excalidraw ( excalidraw:latest), and tldraw ( ratneo/tldraw:latest). Offsite Cloud Backups: Executed full service backups and synced to Google Drive and Proton Drive via rclone ( 2026-08-29_112824). 2026-08-28 Friday Funkwhale Audio Streaming & Range Request Fix ( ul): Diagnosed audio playback failures and stream disconnects on Funkwhale API endpoints ( /api/v1/listen/.../?upload=... and transcoded audio streams). Resolved Nginx reverse proxy byte-range streaming failure ( 206 Partial Content) by updating /data/docker/funkwhale/funkwhale_nginx.template and volume mounting custom /data/docker/funkwhale/mime.types inside the container. Recreated container and verified byte-range audio playback via curl -i -r 0-65535 returning HTTP 206 Partial Content. Updated Funkwhale backup and restore scripts ( /home/luke/git/backups/ul/scripts/backup/funkwhale and restore/funkwhale) to include mime.types and funkwhale_nginx.template in backup tarballs (verified with shellcheck). 2026-08-27 Thursday System Package Maintenance & Kernel Audit ( ul): Executed system package upgrades via paru -Syu / pacman -Syu. Updated mirrorlist and restarted reflector.service. Verified kernel image and module consistency using kernel-check against /boot/vmlinuz-linux-lts. 2026-08-26 Wednesday Studio Control Gateway & ATX Power Automation ( ul & pikvm-studio): Integrated ATX motherboard power button controls ( kvmd-cli atx) into studio-power CLI and synchronized executable between ul and pikvm-studio. Resolved container DNS and IPv6 resolution for bmo95.drheck.dev inside scontrol Docker container on infra_net bridge. Updated and redeployed app.py in /data/docker/scontrol and committed changes to /home/luke/git/scontrol. 2026-08-24 Monday Studio PiKVM Hardware & Perimeter Ingress Security ( ul & pikvm-studio): Configured dual-hostname architecture: studio-kvm-ssh.drheck.dev (direct hardware & SSH) and studio-kvm-web.drheck.dev (Authentik-gated Web UI). Implemented kernel-level outbound firewall isolation in /etc/nftables.conf ( chain output) on ul blocking non-root, non-nginx ( http, UID 33) users from connecting to PiKVM ports 80/443. Configured Pre-Shared Proxy Secret Header ( X-Proxy-Secret) in /etc/nginx/certs/studio_secret.conf (mode 0600, owned by http:http). Configured Mutual TLS (mTLS) with internal CA and client certificate authentication ( /etc/nginx/certs/pikvm_ca.crt, pikvm_client.crt, pikvm_client.key). Updated Authentik ProxyProvider and Application ( Studio PiKVM) with implicit consent ( default-provider-authorization-implicit-consent) and group studio access restriction. Set kvmd.auth.enabled: false in /etc/kvmd/override.yaml (committed in /home/luke/git/pikvm_studio). Updated Studio PiKVM & Studio Control Gateway documentation. Studio Control Gateway Deployment ( scontrol.drheck.dev) ( ul): Developed and deployed asynchronous FastAPI orchestration service ( scontrol) in /data/docker/scontrol binding to dedicated ULA IPv6 address [fd77:7777:5010:0:1337:f1ac:d00d:5c01]:8000. Implemented tactile mobile-optimized dark UI with dynamic state buttons, live monospace console, and status indicators. Generated dedicated service ed25519 key ( /data/docker/scontrol/id_scontrol) registered in luke@studio-kvm-ssh.drheck.dev:~/.ssh/authorized_keys. Integrated real-time Server-Sent Events (SSE) streaming pipeline ( /api/stream/on and /api/stream/off): RF Power Outlets: switches all 7 studio devices via studio-power all on over SSH. ATX Motherboard: queries PiKVM /api/atx and pulses motherboard power button if off. Studio PC Boot: probes DNS hostname bmo95.drheck.dev with a 100-second boot timeout. Soundcards & Audio: audits ALSA/JACK multi-channel recording interfaces. Created Nginx ForwardAuth reverse proxy configuration /etc/nginx/sites-available/scontrol (symlinked to /etc/nginx/sites-enabled/scontrol) with SSE buffering bypass. Configured Authentik ProxyProvider and Application ( Studio Control) bound to Authentik Group studio. Created Git repository at /home/luke/git/scontrol tracking build.sh (verified with shellcheck), Dockerfile, entrypoint.sh, app.py, docker-compose.yml, templates/index.html, and README.md. Created, tested, and verified backup/restore scripts ( scontrol_backup and scontrol_restore in /home/luke/git/backups/ul/scripts/) with par2 10% redundancy. Studio PC ( bmo95) Boot & Audio Stack Audit: Resolved libkmod: ERROR: conf_files_filter_out: Cannot stat directory entry: /etc/modprobe.d/no_intel_sound.conf by replacing early-boot symlink with a direct regular file ( blacklist snd_hda_intel). Verified 4x Echo AudioFire 12 FireWire interfaces ( hw:AudioFire12,0 .. hw:AudioFire12_3,0), Emagic Unitor8 USB MIDI interface, jackd, zita-a2j/ zita-j2a bridges, and autostarting ardour.service DAW instance. SSH Command Tracking Enhancement ( ul): Upgraded /usr/local/bin/audit-log helper script to automatically decode hexadecimal kernel byte strings into plaintext and sort rotated log files ( audit.log.*) in chronological order. Updated SSH Command Tracking Guide documentation. 2026-08-22 Saturday Sub-GHz RF Studio Power Automation ( pikvm-studio): Integrated Great Scott Gadgets YARD Stick One ( 1d50:605b, CC1111) on pikvm-studio to switch 433.92 MHz RF wireless power relays. Reverse-engineered 33-bit Princeton / Fosmon PWM baseband protocol ($T_e = 291.27\text{ }\mu\text{s}$, 3433.22 baud). Identified and resolved critical CC1111 24.0 MHz reference crystal mismatch (recalculated FREQ = 0x121473 from 26.0 MHz default 0x10B071, resolving a 33 MHz carrier transmission error). Resolved double sync gap bug and configured 30-frame contiguous burst streaming (630 bytes, 1.468 s) to satisfy superheterodyne receiver AGC settling. Installed /usr/local/bin/studio-power CLI tool managing all 7 studio devices and whole-studio batch switching. Built standalone C direct transceiver utilities ( yard_tx.c, flipper_replay.c, cc1111_rfcat.h) with libusb-1.0. Created, configured, and pushed Gitea repository at studio_yardstick. Documentation PiKVM GPG / Pass Pinentry TUI Fix ( pikvm-studio): Resolved gpg: public key decryption failed: Screen or window too small when executing gpush pass in subshell pipes. Configured /home/luke/.gnupg/gpg-agent.conf with pinentry-program /usr/bin/pinentry-tty and allow-loopback-pinentry. Configured export GPG_TTY="$(tty)" and gpg-connect-agent updatestartuptty /bye in ~/.bashrc on pikvm-studio. 48-Channel FireWire Synchronous Audio Stack ( bmo95): Consolidated 4x Echo AudioFire12 interfaces into a sample-accurate 48-channel synchronous recording engine on Linux 6.13 PREEMPT_RT kernel. Established hardware color mapping ( Red Master fc79, Blue Slave 1 9815, Green Slave 2 8525, Yellow Slave 3 94e8) with 75Ω BNC Word Clock daisy-chain wiring. Configured master jackd bound to Red with synchronous zita-a2j / zita-j2a duplex ALSA bridges (locked at $\rho = 1.0000000000$). Conducted 10-minute continuous 1 kHz stress test proving absolute phase synchronization (0.000 ms / 0 samples drift across 28.8M samples). Deployed systemd user services ( jack.service, zita-*-{blue,green,yellow}.service) and canonical port alias mapping ( Red_01..12, Blue_01..12, Green_01..12, Yellow_01..12). Documentation Studio Hardware Documentation (Setup Detail) ( ul): Published comprehensive hardware register maps, wiring diagrams, and automation runbooks for the recording studio, audio stack, and RF power systems under the new Setup Detail book. BookStack KaTeX TeX Math & Typography Integration ( ul): Globally enabled KaTeX 0.16.11 client-side LaTeX math rendering across BookStack via app-custom-head. Implemented a CSP strict-dynamic compliant nonced JavaScript loader to dynamically load KaTeX assets without violating browser security policies. Published the Meta: Wiki Capabilities, TeX Math & Typography Reference guide documenting TeX math syntax, Japanese Furigana ( ), callout alert banners, and collapsible accordions. Updated the Agent Guide: Updating BookStack Wiki Pages with typography standards, CLI helper workflows, and operational gotchas. Verified automated backup and restore scripts ( /home/luke/git/backups/ul/scripts/{backup,restore}/bookstack) properly archive and restore database settings and media with par2 parity. 2026-08-20 Thursday pikvm-studio (PiKVM V4 Plus) Configuration & Dotfile Management: 1080p60 Video Pass-Through & Capture Optimization: Imported and applied the v4plus.no-1920x1200 EDID preset via kvmd-edidconf to lock capture to uncropped 1920x1080 @ 60Hz. Enabled gpu_freq=700 in /boot/config.txt for H.264 60fps hardware encoding boost. Configured h264_boost: true and desired_fps: 60 in /etc/kvmd/override.yaml. Chezmoi Configuration Repository Setup: Installed and configured chezmoi on pikvm-studio ( ~/.config/chezmoi/chezmoi.yaml) with sourceDir: ~/git/pikvm_studio and destDir: /. Established clean 2-stage signed Git history: baseline vanilla OS configuration followed by 1080p60 optimizations. Connected and pushed repository to Gitea ( https://git.drheck.dev/luke/pikvm_studio.git) via gpush. SSH Auth Socket & GPG Agent Fixes: Resolved SSH key retention issue in /home/luke/bin/auth_socket.sh by prioritizing existing live agent sockets and testing responsiveness with ssh-add -l rather than relying on obsolete PID suffix parsing. Cleaned up stale lockfiles in ~/.gnupg/ on pikvm-studio and configured pinentry-program /usr/bin/pinentry-curses in ~/.gnupg/gpg-agent.conf. Successfully committed and pushed updated password store entry ( gitea/token/pikvm_studio) via gpush pass. HeckWiki Documentation & Host Maintenance: Authored and verified HeckWiki documentation pages for pikvm-studio and automation. Verified LTS kernel update ( kernel-check), performed planned system reboot on ul, and verified all running systemd services and Docker containers via status_all. 2026-08-19 Wednesday VPS rng Assessment & Infrastructure Monitoring Plan: Inspected VPS rng ( rng.drheck.dev) hardware resources, memory, disk, network, and listening services. Authored and published the Infrastructure Monitoring & Alerting Plan on HeckWiki detailing metrics, alerting strategy, and agent deployment. PiKVM Network Configuration: Verified dual-stack IPv4/IPv6 reachability and latency response ( fping) for pikvm-studio. Added host entry and identity mapping in ~/.ssh/config for direct SSH connectivity. System Package & Media Maintenance: Upgraded host packages via paru -Syu including Java runtime environment updates. Built and installed latest nethack-git (5.0.0 release) from AUR. Restarted plexmediaserver and monitored service journal logs. 2026-08-18 Tuesday ul NVIDIA Driver Upgrade & Container CDI Fix: Diagnosed Immich Outage: Immich server ( immich_server) and machine learning ( immich_machine_learning) exited (127) after host NVIDIA driver upgrade ( 580.173.02 → 580.178.04). The OCI runtime failed mounting /usr/lib/libEGL_nvidia.so.580.173.02 due to obsolete paths in /etc/cdi/nvidia.yaml. Regenerated CDI Spec & Restored Services: Ran nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml to update the CDI specification for 580.178.04, restarted the Immich container stack, and verified healthy container status and restored upstream proxying via Nginx ( photos.drheck.dev). Automated via Pacman Hook: Created /etc/pacman.d/hooks/nvidia-cdi.hook with a PostTransaction trigger on nvidia* and *nvidia* packages to automatically regenerate /etc/cdi/nvidia.yaml on future paru/ pacman driver updates. Opencode Wiki Skill & Knowledge Integration: Embedded HeckWiki / BookStack management rules and helper workflows into global Opencode instructions ( ~/.config/opencode/INSTRUCTIONS.md). Created dedicated skill ~/.config/opencode/skill/update-wiki/SKILL.md for seamless automated wiki editing and querying. Funkwhale Album Track Ordering Fix: Diagnosed track numbering and sort discrepancy in Funkwhale where album and artist views displayed tracks out of sequence. Directly updated track position attributes and calibrated creation_date timestamps in the Funkwhale database via Django management shell ( /app/api/manage.py shell) for Album 10 so both default chronological sorting and album sequence sorting correctly align. 2026-08-17 Monday bmo95 Arch Linux Boot & dmesg Diagnostics & Resolution: Resolved FireWire Audio Crash Loop & Massive dmesg Spam: Diagnosed >50,000 repeating kernel errors ( snd_fireworks fw1.0: oPCR0: plug is already in use) and FireWire CMP stack trace warnings ( sound/firewire/cmp.c:300 / cmp.c:193). Fixed missing ALSA period count in zita-j2a.service and zita-a2j.service by adding -n 3 (required by snd_fireworks driver) and fixed client naming ( -j Up / -j Up_In). Cleaned up set-port-aliases.sh to remove conflicting pkill -9 invocations and background spawns that were killing services mid-handshake and wedging the FireWire CMP isochronous plug state. Fixed hardware clock synchronization sequence: moved clock configuration into firewire-ready.service ( wait-for-firewire.sh) using dynamic GUID-to-node matching ( 0x00148602c583fc79 -> Internal Master, 0x001486a296a59815 -> Word Clock Slave) with a 2-second bus settling delay before starting JACK. Verified full 24-channel capture/playback routing stably running in JACK with zero kernel errors on boot. Udev Rules & System Tuning: Fixed udev substitution syntax in 50-firewire-irq.rules (piping pgrep into xargs to avoid invalid $() substitution error in udevd). Fixed quote syntax and power/control attribute assignment in 50-usb-midi.rules for low-latency Emagic Unitor8 USB MIDI operation. Configured WIRELESS_REGDOM="US" in /etc/conf.d/wireless-regdom. Wireless Regulatory Database: Installed wireless-regdb package to resolve Direct firmware load for regulatory.db failed with error -2. PCIe Correctable AER Timeouts: Added pci=noaer to GRUB_CMDLINE_LINUX_DEFAULT in /home/luke/git/audio_config/common/grub and regenerated GRUB config to eliminate spurious correctable error logging from the ASMedia USB 3.1 controller. ASUS ACPI SuperIO Hardware Monitor Conflict: Added acpi_enforce_resources=lax to GRUB kernel parameters to resolve ACPI OpRegion conflict on ASUS PRIME Z270-A motherboard. 2026-08-16 Sunday bmo95 Audio Workstation Infrastructure & Low-Latency Tuning: Multi-Profile Audio Config Repo: Established /home/luke/git/audio_config git repository tracking audio profiles ( jack-alsa, pipewire), metadata mappings ( filedata.json), and custom management scripts ( apply-profile, consume). Sound Card & Clock Synchronization: Configured dual Echo AudioFire12 interfaces with forced internal clock on primary card (Node 1 / Bottom card) and Word Clock sync on secondary card (Node 2 / Top card). ALSA + JACK2 Engine Setup: Configured JACK2 server via ALSA backend (48kHz / 512 period / 3 buffer) for low-latency audio processing. Systemd User Service Modularization: Modularized JACK audio services into individual systemd user units ( jack.service, firewire-ready.service, zita-j2a.service, zita-a2j.service, a2jmidid.service, jack-aliases.service, patchage.service, ardour.service). Custom Port Aliases & Metadata: Scripted automatic routing and pretty-name metadata ( Down_01.. Down_12 and Up_01.. Up_12) for hardware inputs and outputs. Realtime System & Kernel Tuning: Configured GRUB kernel parameter threadirqs, udev rules for FireWire IRQ thread scheduling ( 50-firewire-irq.rules), HPET/RTC audio timers, sysctl swappiness, and PAM realtime privileges. Software & Documentation: Enabled Ardour 9 integration ( use-jack-pretty-names) and added comprehensive architecture and routing documentation ( README.md). System SSH Audit Logging Setup on bmo95: Installed audit package, enabled auditd, and configured /etc/audit/rules.d/execve.rules with strict auid >= 1000 human session command tracking. Deployed /usr/local/bin/audit-log execution tracking script. 2026-08-12 Wednesday BookStack Wiki & Nginx Reverse Proxy SSO Integration: Reverse Proxy Routing: Configured Nginx /login redirect behavior. Unauthenticated Homepage Auto-Redirect: Implemented DOMContentLoaded JavaScript trigger in BookStack's app-custom-head setting. Ensured unauthenticated visitors accessing https://wiki.drheck.dev/ are automatically forwarded to Authentik ( /login), while keeping deep links (e.g. /books/...) publicly viewable. this is for convenience, not security. I just don't want to click in "login" if I visit wiki/ 2026-08-10 Monday Opencode CLI Agent Configuration & Setup: Configured Opencode settings ( opencode.json / opencode.jsonc) and system instructions. Added instruction mandate to comment bash tool command batches. Deployed global Opencode skills under ~/.config/opencode/skill/. SSH Audit Logging & Command Tracking: Configured Linux auditd rules on ul for auid >= 1000 system call executions. Configured OpenBSD process accounting ( acct) and verbose SSH payload auditing on danknasty. Implemented unified audit-log CLI helper scripts across both machines. Documented SSH command tracking guide and created daily-change-list skill. 2026-08-09 Sunday documenting services in this process saw that kavita permissions were messed up, fixed saw that moebius is like halfway removed but don't know why, leading to the realization that I need to track commands. SSH Command Tracking now I remember why it was removed, because it sucks. The "web service" is just a viewer for multiple drawers connecting with thier own clients. some updates to the crowdsec blocks. realized it has crowed sourced blocking rules, not sure about implementing them though. 2026-08-08 Saturday setting up navidrome, and other audio related software. put albums up on funkwhale power outtage caused IPv6 netblock to change, the IPv6 update scripts still suck. Tried to make them better had AI write a script to pull down missing artwork for Jellyfin noticed collation version mismatch when poking around in posgresql. created a short-link service drheck.dev/$short_link editable at drheck.dev/admin Check Which Databases Have Collation Version Mismatches sudo -u postgres bash -c 'for db in $(psql -At -c "SELECT datname FROM pg_database WHERE datistemplate = false;"); do echo "=== DATABASE: $db ===" psql -d "$db" -c "SELECT 1;" 2>&1 done' Run amcheck to Detect Corrupted B-Tree Indexes sudo -u postgres bash -c ' for db in spacebar immich nextcloud_db anchor authentik postgres gitea nextcloud; do echo "=== AMCHECK: $db ===" psql -d "$db" -c " CREATE EXTENSION IF NOT EXISTS amcheck; SELECT c.relname, bt_index_check(c.oid, true) FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid JOIN pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind = '\\\\\\\\''i'\\\\\\\\'' AND i.indisready AND i.indisvalid AND n.nspname = '\\\\\\\\''public'\\\\\\\\'';" 2>&1 done' Reindex Databases and Refresh Collation Versions sudo -u postgres bash -c ' for db in spacebar immich nextcloud_db anchor authentik postgres gitea nextcloud; do echo "=== REINDEXING & REFRESHING: $db ===" psql -d "$db" -c "REINDEX DATABASE \\\\\\\\"$db\\\\\\\\";" psql -d "$db" -c "ALTER DATABASE \\\\\\\\"$db\\\\\\\\" REFRESH COLLATION VERSION;" done' Verify All Warnings Are Cleared sudo -u postgres bash -c ' for db in $(psql -At -c "SELECT datname FROM pg_database WHERE datistemplate = false;"); do echo "=== VERIFY: $db ===" psql -d "$db" -c "SELECT 1;" 2>&1 done' 2026-08-05 Wednesday updates to crowdsec parsing more SSH errors generated a table of banned IPs but it's not very interesting. Live world view would be cooler 2026-08-04 Tuesday docker updates anchor: 0.13.0 → 0.14.0 authentik: 2026.5.5 → 2026.5.6 bookstack: Updated to latest image immich: v3.0.3 → v3.1.0 infra (valkey): Updated to latest image build kasm: Updated to latest image nextcloud: 34.0.1-apache → 34.0.2-apache penpot: Frontend, backend, exporter, postgres, and redis updated romm: Updated to latest image (5.1.0) vaultwarden: Updated to latest image webrcade: Updated to latest image wick: Updated to latest image installed crowdsec review of publicly accesible service logs (eg: wiki) shows no bot traffic (yet)! review of nginx logs straight from the public IPs shows bot traffic looking for common vulnerabilities, exploits, and staged malware. documentation 2026-08-03 Monday pull all new docker images and prepare for update in preparation for more users, ensure unencrypted backups aren't world readable gdrive backup credentials failed again finally cleaned up google cloud project, renamed properly and created two separate subprojects, google sso, and rclone for ul redid the certs for rclone, tested good ran a backup to repare for docker update and all the online backups had no files except the instructions! setting all the backups to root rendered the rclone script useless. Trying all backup files owned by my user with a plan create a dedicated backup user create a new gpg key for backups backup gpg key move rclone credentials to backup user chown all backups to 700 backup user Profit! 2026-08-02 Sunday Removed old unused "web shells" like the old ttyd, whatever else. Also warpgate. Made webshell awesome. Picked a single font, created a service to show ANSI art on login made big improvements to ansview first render is completely in memory, remove all jumps and relative and exact positions. This allows the final product to be rendered correctly when not placed at the first screen column. due to this success, added an auto wide mode, if artwork can fit on the screen multiple times horizontally, do it also truncate the artwork if viewing on a smaller screen rather than an awful wrap. This is great for the phone shell. 2026-08-01 Saturday Created webshell, and spent a lot time trying to make it customizable per user. This was a failure. 2026-07-31 Friday Set up moebius in KASM, generated a KASM setup guide. 2026-07-30 Thursday giving up on warpgate for now, trying ttyd again with step-ca to tie it into OIDC. I really want an authentik OIDC enabled direct web -> shell. 2026-07-28 Tuesday got warpgate to stop crashing but I'm not sure how I can use it for a web based terminal.. we'll see 2026-07-26 Sunday accidentally blocked the git users with the new firewall rules, allowed it to access the gitea service and we are able to upload again started to set up warpgate as it looks like it can provide a shell that can login with OIDC 2026-07-25 Saturday replaced gemini-cli with opencode, it's way better went through each docker service moving off of the default 1000 UID removed broken web "shells" Made a process to generate short lived ssh certs for web users to get a shell without typing a password but it felt really insecure and I think you could impersonate other users, so I killed it. 2026-07-23 Thursday Set nginx to cache large uploads on the ZFS array as / was filling up with video uploaded from immich Set permanent redirects on short names in the browser for convenience, eg: http://photos -> https://photos.drheck.dev/ 2026-07-22 Wednesday Configured rclone for Google drive and Proton drive. Added this to the backup scripts. Now we have automated on-line backups. created rclone client config in Google's cloud console 2026-07-21 Tuesday transmission would timeout quickly since exposed behind authentik. After some research I modifed the nginx config to start forwarding $host, $remote_addr, $proxy_add_x_forwarded_for, and $scheme. 2026-07-20 Monday IPv6 netblock was wrong again. This time an IPv6 netblock change was detected then the network went out. This caused the API call to desec.io to fail. We were writing the new IPv6 netblock unconditionally though so no more updates were going to happen. Script is updated 1 2 also added logging to the desec.io call in change 2 ^ 2026-07-19 Sunday Tested LDAP users could log in but did not have the ability to choose the window environment LDAP guide updated to fix this problem 2026-07-17 Friday LDAP guide updated to add systemd to shadow for NSS. This was breaking the graphical boot environment on Mitsuki's laptop. write-up 2026-07-16 Thursday router updates Remote IPv6 was not updated in public DNS. The router update script was failing to execute in cron shebang was /usr/local/bin/bash but cron executed /bin/ksh $script... (just openbsd things!) fixed also router IPv6 update script updated to use persistent storage for IPv6 address as a reboot would remove the older address bad firewall rule forwarded all IPv6 traffic arriving on 22 to ul fixed 2026-07-13 Monday created ldap enable doc and test on desktop works! completed SSH cert setup for deskop works! drafted a process to auto generate ssh keys and sign them with authentik 2026-07-12 Sunday created "admin" specific firefox profile on desktop cleared all old disparate passwords for the old individual service passwords fixed www server for mitsuki checked in kanji service in gitea added bounding boxes changed the stroke order color to be more legible and not run into the background 2026-07-11 Saturday created this server change log added "backup_all" script to run each service backup finally checked in backup/restore scripts separated backup output into current and all previous to simplify disk backup modified nginx config for immich, upload cap 30GB -> 100GB NSS + GDM Issue 1. GDM Requests a Dynamic User When GDM starts up it attempts to launch the graphical login screen. For security isolation GDM requests a session for a systemd dynamic user named gdm-greeter from systemd-logind. Systemd dynamically allocated a UID (60578 in this case) and registered the username gdm-greeter to it. Logind triggers the creation of the user's slice and runtime directory systemd[1]: Created slice User Slice of UID 60578. systemd-logind[426]: New session '1' of user 'gdm-greeter' with class 'greeter' and type 'wayland'. 2. Systemd Spawns the User Manager and Invokes PAM To run the background GNOME session, systemd attempts to spawn the per-user systemd instance (user@60578.service). Before launching the service, systemd must authenticate and initialize the user's environment using the PAM stack configured in /etc/pam.d/systemd-user (which inherits /etc/pam.d/system-auth). 3. pam_sss.so Gracefully Rejects the Dynamic User First, the account block in the PAM stack runs pam_sss.so: account sufficient pam_sss.so pam_sss queries the SSSD daemon: "Do you have a user named gdm-greeter?" SSSD queries your LDAP server and finds nothing. SSSD replies: "No, this user is unknown to me." This generates the log entry: 1 (systemd)[508]: pam_sss(systemd-user:account): Access denied for user gdm-greeter: 10 (User not known to the underlying authentication module) Because pam_sss is configured as sufficient, its failure is non-fatal. PAM ignores this rejection and falls through to the next module in the stack: pam_unix.so. 4. pam_unix.so Spawns unix_chkpwd To verify the local account, pam_unix.so spawns the setuid helper binary /usr/bin/unix_chkpwd to check the user's password and shadow database info: 1 account required pam_unix.so 5. The Silent NSS Database Lookup Failure Inside the C code of /usr/bin/unix_chkpwd, the program makes a standard glibc library call to look up the shadow password entry for the username: 1 getspnam("gdm-greeter"); To resolve this call, glibc reads your /etc/nsswitch.conf file to see which databases are registered under the shadow: line. It saw: 1 shadow: compat sss Glibc began querying the modules in order: compat (local files): It searched /etc/shadow for an entry starting with gdm-greeter:. Since gdm-greeter is a dynamic user, no such line exists in /etc/shadow. This returns: NSS_STATUS_NOTFOUND. sss (SSSD): It asked SSSD if it has a cached shadow record for gdm-greeter. SSSD does not. This returns NSS_STATUS_NOTFOUND. Because systemd was missing from that line, glibc reached the end of the databases and stopped. It never queried nss-systemd (which is the only place where GDM's dynamic users are registered). As a result, getspnam("gdm-greeter") returned NULL (No such entry). 6. PAM Crashes, Locking Out the Session unix_chkpwd realized it was completely blind to this user and threw an error: unix_chkpwd[509]: could not obtain user info (gdm-greeter) Because unix_chkpwd failed, pam_unix.so returned a fatal error to PAM. Because pam_unix.so is required, the entire PAM transaction aborted: (systemd)[508]: PAM PAM failed: Authentication service cannot retrieve authentication info Systemd saw the PAM initialization failure and immediately aborted spawning the user manager: (systemd)[508]: user@60578.service: Failed to set up PAM session: Operation not permitted (systemd)[508]: user@60578.service: Failed at step PAM spawning /usr/lib/systemd/systemd: Operation not permitted systemd[1]: Failed to start User Manager for UID 60578. 7. Crash and Lockout Without a running systemd --user manager for UID 60578, the user-level D-Bus broker (org.freedesktop.systemd1) was never launched. When the GDM Wayland session tried to initialize gnome-session, it attempted to talk to the systemd user manager over D-Bus. This lookup failed, causing the greeter worker to throw an unhandled assertion and crash (dumping core): gnome-session-i[518]: Failed to start unit gnome-session@gnome.target: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.freedesktop.systemd1" does not exist systemd-coredump[527]: Process 518 (gnome-session-i) of user 60578 dumped core. GDM attempted to restart this sequence six times, failing every time, until it reached its failure threshold: gdm[455]: Gdm: GdmLocalDisplayFactory: maximum number of display failures reached. Giving up. At this point, GDM completely gave up on opening any graphical interfaces on your screen, leaving the system stuck in the console/tty mode. By adding systemd back to the shadow: line in /etc/nsswitch.conf, the glibc call getspnam("gdm-greeter") was able to query nss-systemd, retrieve the dynamic user's shadow record, pass the PAM stack, start the user manager, and successfully launch the login screen. Web-based Shell Web based SSH Instant Access This is a browser-based SSH terminal service ( webshell.drheck.dev) authenticated via Authentik Single Sign-On (SSO) and short-lived OpenSSH certificates signed by Smallstep ( step-ca), running under rootless Podman Quadlets ( docker-data, UID 1100). 1. Architectural Overview [ Browser Client ] │ ▼ (HTTPS / WebSocket) [ Nginx Reverse Proxy (/etc/nginx/sites-available/webshell) ] │ ├── auth_request ──► [ Authentik Embedded Outpost ] (SSO Auth & Claims) │ ▼ (HTTP / WebSocket + X-Authentik-Username Header) [ Webshell Rootless Container (ttyd + login-wrapper.sh) on eth77 fd77:...:771d:7681 ] │ ├── Step SSH Certificate Sign ──► [ Step-CA Container on webshell_net (step-ca:9000) ] │ ▼ (SSH with Ephemeral Ed25519 Certificate over IPv6) [ Host ul OpenSSH Daemon on [fd77:7777:5010:0:1337:f1ac:d00d:337]:22 ] Key Components step-ca: Smallstep Certificate Authority issuing 5-minute ephemeral Ed25519 user certificates ( step-ca.container on webshell.network). webshell: Debian Bookworm container running static ttyd 1.7.4, step CLI 0.25.0, and login-wrapper.sh ( webshell.container on webshell.network). Authentik: Provides OIDC integration for step-ca and ForwardAuth single sign-on ( webshell.drheck.dev). Nginx: Terminates TLS and proxies HTTP/WebSocket connections while passing authenticated identity headers ( X-Authentik-Username). OpenSSH ( sshd): Configured to trust step-ca signed user certificates for passwordless OS login on host ul. 2. Network & ULA IPv6 Allocation Every web-facing service on host ul binds to a unique ULA IPv6 address on interface eth77 ( fd77:7777:5010:0:1337:f1ac:d00d:.../64). step-ca IPv6: fd77:7777:5010:0:1337:f1ac:d00d:ca (Port 9000, also used locally by /data/minecraft/ipctl) webshell IPv6: fd77:7777:5010:0:1337:f1ac:d00d:771d (Port 7681) Private Bridge Network: webshell.network ( webshell_net, serving step-ca container DNS via aardvark-dns) Host ul ULA IPv6 SSH Target: fd77:7777:5010:0:1337:f1ac:d00d:337 (Port 22) 3. Step 1: Rootless Podman Quadlet Configuration ( step-ca & webshell) A. Quadlet Unit Definitions ( /data/docker/webshell/) Both Quadlets are symlinked into /home/docker-data/.config/containers/systemd/ and managed by user@1100.service: /data/docker/webshell/step-ca.container: [Unit] Description=Smallstep SSH & X.509 CA (Rootless Podman) After=network-online.target Wants=network-online.target [Container] Pull=never ContainerName=step-ca Image=docker.io/smallstep/step-ca:latest Network=webshell.network AddHost=auth.drheck.dev:169.254.1.2 PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:ca]:9000:9000 EnvironmentFile=/data/docker/webshell/.env Volume=/data/docker/webshell/step-ca:/home/step:rw [Service] Restart=always RestartSec=5 [Install] WantedBy=default.target /data/docker/webshell/webshell.container: [Unit] Description=Webshell Browser SSH Terminal (Rootless Podman) After=network-online.target step-ca.service Requires=step-ca.service [Container] Pull=never ContainerName=webshell Image=localhost/webshell:latest Network=webshell.network PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:771d]:7681:7681 Environment=STEP_CA_URL=https://step-ca:9000 Volume=/data/docker/webshell/login-wrapper.sh:/usr/local/bin/login-wrapper.sh:ro Volume=/data/docker/webshell/known_hosts:/etc/ssh/ssh_known_hosts:ro Volume=/etc/passwd:/etc/ul_passwd:ro Volume=/data/docker/webshell/step-ca/certs/root_ca.crt:/etc/step-ca/root_ca.crt:ro Volume=/data/docker/webshell/step-ca/secrets/webshell_svc_pwd:/etc/step-ca/webshell_svc_pwd:ro Volume=/data/docker/webshell/index.html:/etc/ttyd/index.html:ro Exec=-W -H X-Authentik-Username -t theme={"background":"#000000"} -I /etc/ttyd/index.html /usr/local/bin/login-wrapper.sh [Service] Restart=always RestartSec=5 [Install] WantedBy=default.target B. CA Directory Permissions (Rootless User Namespace) Inside docker-data's rootless user namespace ( docker-data:624288:65536), the step user inside smallstep/step-ca maps cleanly to an isolated unprivileged subordinate UID on the host with zero world permissions ( o-rwx). 4. Step 2: Host ul OpenSSH Daemon Trust & Web Session Banner Setup To allow sshd on host ul ( [fd77:7777:5010:0:1337:f1ac:d00d:337]:22) to accept certificates signed by step-ca, enforce strict known_hosts checking, and serve a dedicated login banner exclusively for web sessions: Extract the CA user public key: (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec step-ca step certificate inspect --format json /home/step/certs/user_ca_key.pub) Append the public key ( ecdsa-sha2-nistp256 ... or ssh-ed25519 ...) to /etc/ssh/drheck_ca_user_key.pub. Verify that /etc/ssh/sshd_config.d/10-drheck-certs.conf contains: TrustedUserCAKeys /etc/ssh/drheck_ca_user_key.pub HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub Match Address fd77:7777:5010:771d::/64 Banner /etc/ssh/webshell_banner.txt Web Session Banner & Rotation Script: /etc/profile.d/webshell_banner.sh executes when a web session connects. It randomly selects an ANSI art file from /data/nfo/ansi_list.txt (42,000+ files), renders it via /usr/local/bin/ansview --wide-dupe across terminal width, right-aligns the file path, and exports $LAST_ANSI for the /usr/local/bin/bad_ansi removal command. The static fallback banner is stored at /etc/ssh/webshell_banner.txt. Generate /data/docker/webshell/known_hosts from host ul's public host key: echo "fd77:7777:5010:0:1337:f1ac:d00d:337 $(cat /etc/ssh/ssh_host_ed25519_key.pub)" > /data/docker/webshell/known_hosts chmod 644 /data/docker/webshell/known_hosts Reload OpenSSH daemon: sudo systemctl reload sshd 5. Step 3: Web Terminal Backend ( ttyd + login-wrapper.sh) Setup A. Dockerfile ( /data/docker/webshell/Dockerfile) Rootless Podman ttyd Listener Note: Do NOT pass -6 -i :: under rootless Podman. When rootless Podman forwards traffic from host [fd77:...:771d]:7681 into webshell.network, the forwarded socket arrives over container IPv4 ( 10.89.X.X:7681). Omitting -6 -i :: allows ttyd to bind all interfaces inside the container cleanly instead of rejecting IPv4 forwards with 502 Bad Gateway. FROM debian:bookworm-slim ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt-get install -y --no-install-recommends \ ca-certificates curl openssh-client bash procps libnss-sss \ && sed -i 's/passwd: files/passwd: files sss/' /etc/nsswitch.conf \ && rm -rf /var/lib/apt/lists/* # Install static ttyd 1.7.4 & step CLI 0.25.0 RUN curl -sSL https://github.com/tsl0922/ttyd/releases/download/1.7.4/ttyd.x86_64 -o /usr/local/bin/ttyd \ && chmod +x /usr/local/bin/ttyd \ && curl -sSL https://dl.step.sm/gh-release/cli/docs-step-cli/v0.25.0/step_linux_0.25.0_amd64.tar.gz | tar -xz -C /tmp \ && mv /tmp/step_0.25.0/bin/step /usr/local/bin/step \ && chmod +x /usr/local/bin/step \ && rm -rf /tmp/step_0.25.0 COPY login-wrapper.sh /usr/local/bin/login-wrapper.sh RUN chmod +x /usr/local/bin/login-wrapper.sh EXPOSE 7681 ENTRYPOINT ["/usr/local/bin/ttyd"] CMD ["-W", "-H", "X-Authentik-Username", "-t", "theme={\"background\": \"#000000\"}", "-I", "/etc/ttyd/index.html", "/usr/local/bin/login-wrapper.sh"] B. Backend Login Wrapper ( /data/docker/webshell/login-wrapper.sh) #!/usr/bin/env bash set -euo pipefail # Resolve username across CGI-style environment variables exported by ttyd/proxy TARGET_USER="${X_AUTHENTIK_USERNAME:-${TTYD_USER:-${HTTP_X_AUTHENTIK_USERNAME:-${AUTHENTIK_USERNAME:-}}}}" if [[ -z "${TARGET_USER}" ]]; then echo "Authentication Error: Missing Authentik identity." >&2 exit 1 fi # Strict alphanumeric regex check if [[ ! "${TARGET_USER}" =~ ^[a-z_][a-z0-9_-]{0,31}$ ]]; then echo "Authentication Error: Invalid username format." >&2 exit 1 fi # Retrieve numeric UID from host passwd database TARGET_UID=$(awk -F: -v u="${TARGET_USER}" '$1 == u {print $3}' /etc/ul_passwd 2>/dev/null || true) if [[ -z "${TARGET_UID}" ]]; then echo "Authentication Error: User does not exist on target host." >&2 exit 1 fi # Reject system accounts (UID < 1000), admin, and docker* accounts if (( TARGET_UID < 1000 )) || [[ "${TARGET_USER}" == "admin" ]] || [[ "${TARGET_USER}" == docker* ]]; then echo "Authentication Error: Access to privileged or system account is forbidden." >&2 exit 1 fi KEY_DIR="$(mktemp -d -t ssh_XXXXXX)" KEY_PATH="${KEY_DIR}/id_ed25519" cleanup() { rm -rf "${KEY_DIR}" } trap cleanup EXIT INT TERM # Generate ephemeral Ed25519 keypair silently ssh-keygen -q -t ed25519 -f "${KEY_PATH}" -N "" >/dev/null 2>&1 # Sign user certificate via JWK provisioner (--sign flag is mandatory for non-interactive execution) if command -v step >/dev/null 2>&1 && [[ -n "${STEP_CA_URL:-}" ]] && [[ -f "/etc/step-ca/webshell_svc_pwd" ]]; then step ssh certificate "${TARGET_USER}" "${KEY_PATH}.pub" --sign --provisioner="webshell-svc" --provisioner-password-file="/etc/step-ca/webshell_svc_pwd" --ca-url="${STEP_CA_URL}" --root="/etc/step-ca/root_ca.crt" --not-after=5m --force >/dev/null 2>&1 || true fi # Login passwordless to host ul SSH daemon over local ULA IPv6 with strict known_hosts checking exec ssh -o StrictHostKeyChecking=yes -i "${KEY_PATH}" "${TARGET_USER}@fd77:7777:5010:0:1337:f1ac:d00d:337" "$@" 6. Step 4: Authentik Single Sign-On Provisioning ( ak shell) To configure Authentik ForwardAuth without encountering 500 Internal Server Error, Redirect URI Error, or ERR_TOO_MANY_REDIRECTS, execute the following Python provisioning script in authentik-server: (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -i authentik-server ak shell) << 'EOF' from authentik.core.models import Application from authentik.providers.proxy.models import ProxyProvider, ProxyMode from authentik.providers.oauth2.models import RedirectURI, RedirectURIMatchingMode, RedirectURIType from authentik.outposts.models import Outpost SERVICE_NAME = "webshell" EXTERNAL_HOST = f"https://{SERVICE_NAME}.drheck.dev" # 1. Inherit default scope property mappings from a working provider template template_provider = ProxyProvider.objects.get(name="cockpit") # 2. Create ProxyProvider with complete OAuth, redirect URI, and cookie settings provider, _ = ProxyProvider.objects.get_or_create( name=SERVICE_NAME, defaults={ "external_host": EXTERNAL_HOST, "mode": ProxyMode.FORWARD_SINGLE, "authorization_flow_id": "f9ad5326-f668-4d1f-861d-43b47f0a002e", "cookie_domain": "drheck.dev", "grant_types": ["authorization_code", "client_credentials", "password"], "redirect_uris": [ RedirectURI( matching_mode=RedirectURIMatchingMode.STRICT, url=f"{EXTERNAL_HOST}/outpost.goauthentik.io/callback?X-authentik-auth-callback=true", redirect_uri_type=RedirectURIType.AUTHORIZATION ), RedirectURI( matching_mode=RedirectURIMatchingMode.STRICT, url=f"{EXTERNAL_HOST}?X-authentik-auth-callback=true", redirect_uri_type=RedirectURIType.AUTHORIZATION ), ] } ) provider.property_mappings.set(template_provider.property_mappings.all()) provider.save() # 3. Create Application app, _ = Application.objects.get_or_create( slug=SERVICE_NAME, defaults={ "name": SERVICE_NAME.capitalize(), "meta_launch_url": f"{EXTERNAL_HOST}/", "open_in_new_tab": True, "provider": provider, } ) # 4. Mandatory: Assign provider to authentik Embedded Outpost outpost = Outpost.objects.get(name="authentik Embedded Outpost") if provider not in outpost.providers.all(): outpost.providers.add(provider) outpost.save() print(f"Successfully provisioned {SERVICE_NAME} and attached to Embedded Outpost.") EOF 7. Step 5: Nginx Reverse Proxy Configuration ( /etc/nginx/sites-available/webshell) map $http_upgrade $proxy_target_connection_webshell { default upgrade; "" ""; } server { listen 443 ssl; listen [::]:443 ssl; http2 on; server_name webshell.drheck.dev; ssl_certificate /etc/nginx/certs/fullchain.pem; ssl_certificate_key /etc/nginx/certs/privkey.pem; client_max_body_size 64M; # Authentik ForwardAuth Subrequest Configuration auth_request /outpost.goauthentik.io/auth/nginx; error_page 401 = @goauthentik_proxy_signin; auth_request_set $auth_cookie $upstream_http_set_cookie; add_header Set-Cookie $auth_cookie; auth_request_set $authentik_username $upstream_http_x_authentik_username; auth_request_set $authentik_email $upstream_http_x_authentik_email; # Per-User Custom Fonts Route location /user-fonts/ { auth_request /outpost.goauthentik.io/auth/nginx; auth_request_set $authentik_username $upstream_http_x_authentik_username; alias /home/$authentik_username/.config/webshell/fonts/; types { font/woff2 woff2; font/ttf ttf; font/otf otf; } add_header Cache-Control "public, max-age=86400"; } location / { proxy_pass http://[fd77:7777:5010:0:1337:f1ac:d00d:771d]:7681; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $proxy_target_connection_webshell; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Authentik-Username $authentik_username; proxy_set_header X-Authentik-Email $authentik_email; } location /outpost.goauthentik.io { proxy_pass https://auth.drheck.dev/outpost.goauthentik.io; proxy_set_header Host auth.drheck.dev; proxy_set_header X-Original-URL $scheme://$http_host$request_uri; proxy_set_header X-Forwarded-Host $http_host; proxy_set_header X-Forwarded-Proto $scheme; add_header Set-Cookie $auth_cookie; auth_request_set $auth_cookie $upstream_http_set_cookie; proxy_pass_request_body off; proxy_set_header Content-Length ""; } location @goauthentik_proxy_signin { internal; add_header Set-Cookie $auth_cookie; return 302 /outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri; } } 8. Step 6: Backup & Restore Scripts Every service on host ul uses shellcheck-verified backup and restore scripts located in /home/luke/git/backups/ul/scripts/ with 10% par2 redundancy: /home/luke/git/backups/ul/scripts/backup/webshell: Archives /data/docker/webshell/ ( .env, step-ca.container, webshell.container, webshell.network, step-ca/) to webshell_data.tar.xz, and system configuration/scripts ( /etc/ssh/webshell_banner.txt, /etc/ssh/drheck_ca_user_key.pub, /etc/ssh/sshd_config.d/10-drheck-certs.conf, /etc/nginx/sites-available/webshell, /etc/profile.d/webshell_banner.sh, /usr/local/bin/bad_ansi, /data/nfo/ansi_list.txt) to webshell_system_config.tar.xz, generating .par2 recovery files via par2 create -r10. /home/luke/git/backups/ul/scripts/restore/webshell: Verifies archives with par2 verify, restores configurations into /data/docker/webshell/ and /, and restarts step-ca.service and webshell.service. 9. Initial Assumptions & Attempts vs. Actual Working Configuration Technical Area Initial Assumption / Attempt Actual Working Configuration & Root Cause Prevention Standard Outpost Assignment ( 500 Internal Server Error) Creating ProxyProvider in Authentik automatically activates ForwardAuth routing. Creating a provider does not assign it to an Outpost. Unassigned hostnames return 404 Not Found from /outpost.goauthentik.io/auth/nginx, which Nginx converts to 500 Internal Server Error. Explicitly attach providers to authentik Embedded Outpost ( outpost.providers.add(provider)) before enabling Nginx traffic. Redirect URIs ( Redirect URI Error) Instantiating ProxyProvider sets default authorization callback URLs. The Django ORM defaults redirect_uris to []. Authentik rejects callback URIs against an empty database list. Explicitly define RedirectURI objects for both /outpost.goauthentik.io/callback and root / in ProxyProvider.redirect_uris. Cookie Domain & Scopes ( ERR_TOO_MANY_REDIRECTS) Providers operate without defining cookie_domain, grant_types, or property_mappings. Without property_mappings, the OAuth request omits critical scopes ( ak_proxy). Without explicit cookie_domain='drheck.dev', Authentik refuses to grant ForwardAuth proxy tokens, causing an endless redirect loop. Clone property_mappings from cockpit and explicitly set cookie_domain='drheck.dev' and grant_types. CGI Env Mapping ( Missing Authentik identity) ttyd -H X-Authentik-Username exports X_AUTHENTIK_USERNAME directly to child scripts. ttyd transforms HTTP headers into standard CGI environment variables ( HTTP_X_AUTHENTIK_USERNAME). Always check CGI-style fallback variations: TARGET_USER="${X_AUTHENTIK_USERNAME:-${TTYD_USER:-${HTTP_X_AUTHENTIK_USERNAME:-${AUTHENTIK_USERNAME:-}}}}". Rootless Port Forwarding vs. -6 -i :: ( 502 Bad Gateway) Keeping legacy Docker "-6", "-i", "::" flags works identically in rootless Podman. Under rootless Podman, PublishPort=[fd77:...:771d]:7681:7681 forwards traffic into webshell_net over IPv4 ( 10.89.X.X:7681). Passing -6 -i :: restricts ttyd to IPv6-only and rejects forwarded connections. Omit -6 -i :: in webshell.container Exec= so ttyd listens on all interfaces inside the container namespace. Container Script Updates ( old script execution) Modifying login-wrapper.sh on host ul and restarting runs the updated script only if mounted. Restarting only re-reads bind-mounted files, not COPYed image layers. Bind-mount operational scripts ( Volume=/data/docker/webshell/login-wrapper.sh:/usr/local/bin/login-wrapper.sh:ro) in webshell.container. CA Secret Privilege Restrictions Administrative or personal user accounts own the CA keys and JWK provisioner secret. Standard unprivileged or administrative ( wheel) user accounts should not have read access to CA private keys or provisioner passwords. Isolate /data/docker/webshell/step-ca inside docker-data's rootless subordinate UID namespace with o-rwx permissions. SSH Client Host Warnings ( Warning: Permanently added...) Invoking ssh with /dev/null as known_hosts prints key addition warnings on every login. Connecting without a populated known_hosts file causes OpenSSH to log a warning to stderr whenever an ephemeral host entry is added. Mount a static /etc/ssh/ssh_known_hosts file generated from /etc/ssh/ssh_host_ed25519_key.pub and enforce StrictHostKeyChecking=yes. Cockpit Login MOTD Banner ( Web console: https://ul...:9090) Cockpit's MOTD issue updater ( update-issue) injects :9090 URLs into /etc/motd.d/. Cockpit's cockpit-issue.service inspects local interfaces and writes direct port URLs to /run/cockpit/issue, which /etc/motd.d/cockpit symlinks into login greetings. Remove /etc/motd.d/cockpit and /etc/issue.d/cockpit.issue symlinks and mask cockpit-issue.service ( systemctl mask cockpit-issue.service). Dynamic Host UID Verification Hardcoding system username lists (`root admin ...`) in wrapper scripts to block system accounts. Web-Session-Only Login Banner Printing greetings via script echo or editing server-wide /etc/motd. Script echo output is lost before terminal attachment, while /etc/motd changes affect standard SSH logins. Configure Match Address fd77:7777:5010:771d::/64 in /etc/ssh/sshd_config.d/10-drheck-certs.conf with Banner /etc/ssh/webshell_banner.txt. Customization Webshell Configuration Guide Per-user web terminal preferences are stored in ~/.config/webshell/. Directory Structure ~/.config/webshell/ ├── config.json # Theme, font size, and cursor settings └── fonts/ # Dir for local fonts └── font.woff2 # (Optional) Local custom font file (.woff2, .ttf, or .otf) config.json Options Key Type Default Description fontFamily String 'JetBrains Mono', monospace Font family stack fontSize Number 15 Font size in pixels cursorStyle String "block" Cursor shape ( "block", "underline", or "bar") cursorBlink Boolean true Enable or disable cursor blinking theme Object See below Color palette overrides Theme Options ( theme object) Key Description background Terminal background color (e.g. "#000000") foreground Default text color (e.g. "#e0e0e0") cursor Cursor color (e.g. "#ffffff") black – white Standard 8 ANSI colors brightBlack – brightWhite High-intensity 8 ANSI colors Example ~/.config/webshell/config.json { "fontFamily": "'JetBrains Mono', 'Fira Code', monospace", "fontSize": 15, "cursorStyle": "block", "cursorBlink": true, "theme": { "background": "#000000", "foreground": "#e0e0e0", "cursor": "#00ff00", "green": "#00ff00", "cyan": "#00ffff" } } Using Local Font Files Location Place your font file in ~/.config/webshell/ as one of the following: font.woff2 (Create your own if needed! See below) font.ttf font.otf The webshell will automatically encode and load your font file into the web browser upon login. Determining Font Family Name from a Font File fc-scan will return the font family name for you. It is possible to return multiple aliases so you can use the first one to simplify the process. $ fc-scan GoogleSansCodeNerdFontMono-SemiBoldItalic.ttf --format "%{family}\n" | cut -d, -f1 GoogleSansCode Nerd Font Mono Use woff2 files instead of ttf or otf These are normal font files, but compressed so it will be nicer for web use: # one at a time for f in *.ttf *.otf; do woff2_compress "$f"; done # use all those cores! parallel woff2_compress ::: *.[ot]tf On 181 of my fonts the size dropped from 381M to 220M. KASM Container Build Archived / Decommissioned (2026-10-19): Kasm Workspaces on ul has been decommissioned. This page is retained as an archival reference for container packaging and KasmVNC configuration. This guide covers the technical workflow for packaging desktop GUI applications into custom Docker images for Kasm Workspaces (version 1.19.0+), registering them in the database and web interface, configuring advanced features, and understanding display server architecture (X11 vs. Wayland). 1. Selecting a Base Image Kasm Workspaces runs applications inside containerized desktop environments streamed via KasmVNC. Always extend an official Kasm 1.19 base image to ensure compatibility with session management, audio forwarding, clipboard synchronization, and dynamic resizing: Base Image Base OS Desktop Environment Recommended Use Case kasmweb/terminal:1.19.0-rolling-weekly Ubuntu 24.04 (Noble) XFCE4 + Terminal GUI utilities, editors, lightweight tools kasmweb/core-ubuntu-noble:1.19.0 Ubuntu 24.04 (Noble) Minimal XFCE4 Core Clean custom application desktop builds kasmweb/core-debian-bookworm:1.19.0 Debian 12 (Bookworm) Minimal XFCE4 Core Debian-focused software packaging kasmweb/desktop:1.19.0-rolling-weekly Ubuntu 24.04 (Noble) Full XFCE4 + Apps Full desktop environments 2. Dockerfile Packaging Workflow Key Packaging Rules User Isolation: Official Kasm base images run under non-root user 1000:1000 ( kasm-user). Switch to USER root for package installation and explicitly switch back to USER 1000 at the end of the Dockerfile. Ubuntu 24.04 Package Renaming ( libasound2t64): In Ubuntu 24.04 (Noble), the 64-bit time_t transition renamed ALSA runtime packages (e.g. libasound2 $\rightarrow$ libasound2t64). Use libasound2t64 (or libasound2*) to prevent dependency resolution issues on 24.04 bases. Autostart Configuration: To launch an application automatically when the workspace opens, copy an XFCE .desktop shortcut into /home/kasm-user/.config/autostart/. Custom Startup Scripts: For applications requiring pre-launch environment variables or wrapper logic, place a script in /dockerstartup/custom_startup.sh. Example Dockerfile ( kasmweb/moebius:latest) FROM kasmweb/terminal:1.19.0-rolling-weekly USER root # Install required X11, GTK3, and audio runtime dependencies (Ubuntu 24.04 Noble) RUN apt-get update && \ apt-get install -y --no-install-recommends \ wget curl libnotify4 libnss3 libxss1 libxtst6 xdg-utils \ libsecret-1-0 libgbm1 libasound2t64 && \ wget -q -O /tmp/Moebius.deb https://github.com/blocktronics/moebius/releases/latest/download/Moebius.deb && \ apt-get install -y /tmp/Moebius.deb && \ rm -rf /var/lib/apt/lists/* /tmp/Moebius.deb && \ mkdir -p /home/kasm-user/Desktop /home/kasm-user/.config/autostart && \ cp /usr/share/applications/moebius.desktop /home/kasm-user/Desktop/moebius.desktop && \ cp /usr/share/applications/moebius.desktop /home/kasm-user/.config/autostart/moebius.desktop && \ chmod +x /home/kasm-user/Desktop/moebius.desktop && \ chown -R 1000:1000 /home/kasm-user/Desktop /home/kasm-user/.config USER 1000 Building inside Standalone Kasm Containers When running standalone Kasm Workspaces ( lscr.io/linuxserver/kasm), Docker images must be built inside the Kasm container engine ( kasm): sudo docker exec -i kasm docker build -t kasmweb/moebius:latest - < Dockerfile 3. Registering the Workspace in the Web Interface Automated Database Registration ( kasm_db) In standalone Kasm deployments, workspaces can be registered directly in the PostgreSQL database ( kasm_db). Each entry in the images table defines the container parameters, and group_images grants access to user groups. [!IMPORTANT] Unused foreign-key UUID columns ( session_banner_id, server_id, zone_id, filter_policy_id, server_pool_id) must be explicitly set to NULL. Empty strings ( '') cause PostgreSQL UUID syntax errors. -- 1. Insert Workspace definition into images table INSERT INTO images ( image_id, cores, description, docker_registry, docker_token, docker_user, image_src, enabled, available, friendly_name, hash, memory_bytes, name, session_banner_id, session_banner_force_disabled, x_res, y_res, run_config, volume_mappings, persistent_profile_path, restrict_to_network, restrict_network_names, allow_network_selection, restrict_to_server, server_id, exec_config, restrict_to_zone, zone_id, filter_policy_id, filter_policy_force_disabled, categories, require_gpu, gpu_count, gpu_graphics_acceleration, gpu_video_encoding, hidden, notes, override_egress_gateways, image_type, server_pool_id, link_url, cpu_allocation_method, uncompressed_size_bytes, launch_config, rdp_client_type, is_remote_app, remote_app_icon, remote_app_name, remote_app_program, remote_app_args, created_at, updated_at, enforce_workspace_persistence ) VALUES ( 'a1111111-60eb-4000-a000-0000000060eb', 2, 'Moebius ANSI and ASCII Art Editor full desktop client.', 'https://index.docker.io/v1/', NULL, NULL, 'img/thumbnails/terminal.png', true, true, 'Moebius', NULL, 2902458368, 'kasmweb/moebius:latest', NULL, false, 1280, 720, '{"hostname": "moebius"}', '{}', '/data/kasm_profiles/{username}/', false, '[]', false, false, NULL, '{}', false, NULL, NULL, false, '["Design"]', false, 0, '["MESA"]', '["SW"]', false, NULL, false, 'Container', NULL, NULL, 'Inherit', 1500000000, '{}', NULL, false, NULL, NULL, NULL, NULL, NOW(), NOW(), true ) ON CONFLICT (image_id) DO NOTHING; -- 2. Assign image to default All Users group (68d557ac-4cac-42cc-a9f3-1c7c853de0f3) INSERT INTO group_images (group_image_id, group_id, image_id) VALUES ( 'a2222222-60eb-4000-a000-0000000060eb', '68d557ac-4cac-42cc-a9f3-1c7c853de0f3', 'a1111111-60eb-4000-a000-0000000060eb' ) ON CONFLICT (group_image_id) DO NOTHING; Manual UI Registration Navigate to Kasm Admin Panel -> Workspaces -> Add Workspace. Workspace Type: Container Friendly Name: Moebius Docker Image: kasmweb/moebius:latest Cores / Memory: 2 / 2768 (MB) Persistent Profile Path: /data/kasm_profiles/{username}/ Click Save. 4. Advanced Configuration Options When packaging complex software, evaluate these key operational options: 1. Persistent Profiles ( persistent_profile_path) Purpose: Retains user files, application settings, and browser profiles across session restarts. Configuration: Set persistent_profile_path to /data/kasm_profiles/{username}/. Behavior: When a session launches, Kasm bind-mounts the user's persistent directory to /home/kasm-user/. Set enforce_workspace_persistence = true to prevent unmanaged ephemeral sessions. 2. GPU Hardware Acceleration (AMD, Intel & NVIDIA) Purpose: Offloads WebGL, 3D rendering, compute, and video stream encoding to host GPUs. Configuration for AMD & Intel GPUs: Pass /dev/dri and /dev/kfd device nodes in run_config: { "devices": [ "/dev/dri:/dev/dri", "/dev/kfd:/dev/kfd" ], "group_add": ["render", "video"] } Graphics Acceleration: Set gpu_graphics_acceleration: ["MESA"] (RADV Vulkan / OpenGL). Hardware Video Encoding: Set gpu_video_encoding: ["VAAPI", "SW"] for hardware H.264/HEVC/AV1 encoding. Set require_gpu = true and gpu_count = 1. Configuration for NVIDIA GPUs: Ensure NVIDIA Container Toolkit is enabled on the host. Set gpu_graphics_acceleration: ["NVIDIA"] and gpu_video_encoding: ["NVENC", "SW"]. 3. Bidirectional Audio & Microphone Forwarding Purpose: Streams 48 kHz stereo audio output and enables low-latency WebRTC microphone input. Configuration: Kasm base images include ALSA and PulseAudio ( pulseaudio -D) bridged to WebRTC / WebSocket audio codecs. Ensure packages install libasound2t64 and pulseaudio-utils. Enable microphone pass-through in launch_config if recording or voice chat is required: { "allow_microphone": true } 4. Gamepad & Controller Passthrough Purpose: Passes browser HTML5 Gamepad API inputs directly into container /dev/input/js* virtual joystick devices. Configuration: In launch_config, enable gamepad forwarding: { "allow_gamepad": true } 5. Network Isolation & Security ( restrict_to_network) Purpose: Restricts container egress or connects workspaces to internal backend VLANs/docker networks. Configuration: Set restrict_to_network = true and specify target Docker network names in restrict_network_names: ["infra_net", "isolated_workspaces"] 5. Display Server Architecture: X11 vs. Wayland 1. Current KasmVNC Standard (X11) Architecture: Official Kasm Workspaces desktop containers ( kasmweb/desktop, kasmweb/terminal) operate on the X11 display server protocol using Xvfb / Xorg virtual framebuffers on display :1. Why X11 is Default: Input Injection: Keyboard and mouse event simulation across web browsers maps reliably to X11 input extensions ( libxtst6). Dynamic Resolution Resizing: Automatic desktop resizing when browser windows resize relies on xrandr / X11 Resize and Rotate extensions. Clipboard Synchronization: Web-to-desktop clipboard sharing uses standard X11 selection buffers ( PRIMARY and CLIPBOARD). 2. Wayland & Xwayland Support Can Kasm run Wayland?: Yes. Modern KasmVNC releases support Wayland compositors (such as Sway or Weston) via Xwayland and virtual DRM/KMS backend framebuffers. Limitations in Wayland Mode: Clipboard & Capture: Pure Wayland restricts client-to-client screen capture and clipboard access by design. Applications running natively under Wayland require xdg-desktop-portal-wlr or compositor-specific protocols to share clipboards with KasmVNC. Application Compatibility: Most packaged Linux GUI applications (Electron, GTK3, Qt5) run under X11 or Xwayland by default. For production packaging in Kasm Workspaces, X11 remains the recommended display server unless testing specialized Wayland-only compositors. GPG Keys Current Luke https://keys.openpgp.org/search?q=lukeisgreat@gmail.com pub rsa4096 2019-11-30 [SC] 9F9AC4567EDC8E2B86E5CB9475D6600BEF4E8E8F uid Luke Tidd sub rsa4096 2019-11-30 [E] -----BEGIN PGP PUBLIC KEY BLOCK----- Comment: 9F9A C456 7EDC 8E2B 86E5 CB94 75D6 600B EF4E 8E8F Comment: Luke Tidd xsFNBF3ifkcBEADU/riY8J3FRkZ+2qVvMQPHiwqEmZYCeSEtCsFxuXamEfRMhPEO bD2B6DXsHWXYw4g14Q0TF0A5OezSIXUt3au0d9AiSSFJtlXdUCAber5CGfQT44IU UgjBSww3FPTp9QJ7t7eUbNluablI4bt82piFJEgc0619bXmUMjZOyQps7xeFdRD0 FKNnSs7rgNgvZdTHwOvoqy0JKky/YKLgTU4jRieuVNT1mdR+IxfzbOzS7mJxa1Ur RX/FbrDp6+INMPFWoHOJaspc5jAcTtUAkjREyZmgn/Ki/KAe8A6agFseK7gKX3SS XIPhAgHfRbS9xr9vG5+QOlsksKqlEzGOxjgRGtloXBkk4H2oOJNgQfGccKl4ADel PZ3tXJGZlphZ/86kHFt1XNgr+/wsFpE5XCMiHiFo63t6jGEIQr2QgfJNZSTv1Lrr B1gt8lakaTHEOAdJafb0VAGFQPxNiiGYP6bfh9dXeKWOg0sdyBWCS0hHv29neEal rwrYTLNeEVDSxbMgw+dG53CKWZTkvRuPgGK+hcjPXOmByqFJzUJSHCHwpH073ret UgP8SGpkcC+7Sv3HP/qBe4UNAXrQnSXi2oPdHejaLXryCxEmLx+1mycGTkhkfXQ2 XFZsLWm8FSbXsnzELFPsB4gxZqOlL1k6zEk5gyW/KH+IkFulf/pRf9QQgQARAQAB zSFMdWtlIFRpZGQgPGx1a2Vpc2dyZWF0QGdtYWlsLmNvbT7CwZQEEwEIAD4CGwMF CwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQSfmsRWftyOK4bly5R11mAL706OjwUC aUVLcwUJDUQArAAKCRB11mAL706OjyQyD/9jlYzHboeOyabPqkscgGunDhBnNjrA uWSd2tyhP3/vRtv3vY8Vzp0VNdxXDD16oKmCZgj85qmpCl4ZhctZ1enyhDJKKeK/ JdwrdvZIL1atNufkzl3mko5wSISRK0dnDfOWhTOtsDFScIsQyBCOe41eU0eg3Jp/ 2SlEu3kQC6WIUXk5CeziHlPCMfoK3GS/i7caKwixelE2Ju35yDzTnrBIhN3Zykpj Fy/qn5M7TDlZZU+JzV6kKu7Q3rTnWkDJMcwv7a20P0chhZnonxKN/p5ktlgYMcWn KXkt9lGv3Kssu5CIAO9qIRTZMck0j2gxI1QCcFtRE+ZwPV2ToX12YunQAWbAhwEP scfPOYb08cNPCoZy7xvrRsY+hlxx47sc9iHvxHdqzf6B36QDyzEoJ6hSgHQYXiV5 A0dSLFDwJWxx4VfzYO9UDaKm6FwT0ocD/eZ+ZfPS4dcae0ftFBnj/S+Y7pxW3skU TywRZKsu+XQakL4chGtOi42Ii8sQMTWc50Qfc82ynnXDqlnD8s+VUcUmCYBhN4J+ QFuzIS4Dk7m8Zp6gSM8Vbr5ZYAsLkSm4ekAZcus8RBxZhODWo4l18G4EWxmLlJyz H/I5ztbnuJlUT20ZbKBjth8xT/X8SHJQPqtZ1sT1iPZGuoqEZpU81pIfLAi8Bey+ 5eon0j1/mFKnu8LBlAQTAQgAPgIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgBYh BJ+axFZ+3I4rhuXLlHXWYAvvTo6PBQJe4kecBQkC4PzVAAoJEHXWYAvvTo6PnvwP /3jg9RcHaDxnTgi/J8HaXE6AtLgZ05Qku+TW/nYE+gyL3/EuHk8TfwyqiuBv85oa oMHR9b3IP3fUZ5cdWVi/npEMm5/Tfv19DzYzVw84cFxGBDO2GnwshW9HnhBn56UG YDJ1J/8u9VutkC33JirAqd1DCAWn1KZOV6M9AFxvF5lSPXwMEXRiQr9A8rVjR8oc 6mTQKMluW6D918F6v+4MqtB7eamsMvym2ixNnyW3SuqPrvevEZRPmOv61g3e/vS8 00tBlcjAveE85rTgjZKdyXkUTD8FhPzqXW6pHx6AzhiqSS4FLwk9yLtMjC85RHvQ Texoc5WS9SwVPXhKpEc4QXIS66L5jf4mJNqdyUAeAqrbbRqcCQsSlMe3SU+9CMS0 coa4piedhO6wirFnH8sWxN2CQVCRIwrZqZDlNBEaNQinyBWmr0c+VN0EQwdv4p2c ImYpb1GhUE/FsuW2TC6EJppITz3ipTBvD1MYBdpho1TzizAQympMKmTJMrdup2FM zfVTs7q8pXDQIQCwZjYismLUIhRVLp0YJldm/uTg/UhN/R7JrQ9bmbAKjcfXcXIw 1t4ORsApEO+GkAk6YTO759CqrZw0Cg5OoaVZprg83yT+77Y9GE+OEkpzbj8n8bLo C81vgMzDkMvkk/n2klm4O+lKCd3QyK00sRhSShmIqtv9zsFNBF3ifkcBEADNeG+l 45CuegVep4ti1q6J3AAj1kDeZfNYF5IMCNmgC2mLoxQzxGUd2aPdr1zOFqo2B2TP m/FvXooZMa4Iocue+/BIpxUqJNGqCEv6B3/EzUQzomOSiy8yxm6fvMpBhS32+KKq AdhRyq32dUBdEwy20+7EcYi2sFGdJXaqf0NAM5VXp1UHx/4Q0SCdezswi8WU1a03 cDYfAc/GF1ZPucYjCa9yB3stbBTHc3VBgcbcWmpRB2Lv063e3NswLyqrKMWBxYZq HK+h/deYMRh3fmmL4OeUmzT2e1CvJxBEl9OsTo1Q1NAnByRhee+3AwLwLsA3a4jz NDz4igGhLVFS0yEWc7xQtYePfQi/0uXLezujAxrNT7ugAKyPzxtIG3Qto2O0Q+zP 5BzOFZ8XyCuRCPojd2pN6u2/Z2dQpUtwH+fbQ8c5Xui356asqBhpBCjH/nV360bd WgKNpmgnSbIGkqoUr2LjXZFA2Qt+N+HvUFl+zEm11OwFuuXvu6waaSiCkkevuA+E Ec/hsVujUkj437vF2S3hQsNcmuHtfEg4ld/d1Be4lu86tUAhsTjeyQYKhHBP5dKj qkUJwrkVP+f0zRVigedof5LG38XGxcv2KOmW2y9Pg+BIncPLHYK9pg4vJ1qvvTp2 nThVOfT7tKpiOhkbeQOvhI+DJLCedvXkcYperwARAQABwsF8BBgBCAAmAhsMFiEE n5rEVn7cjiuG5cuUddZgC+9Ojo8FAmlFS3MFCQ1EAKwACgkQddZgC+9Ojo/uiw/+ I52aAPI7S2AFX8BLD6rHCzzqNKx822e36yK3AUny6UUv51ou5YeDpO3H+O8U8bZX +L4GRT1VDUBR/B75AlBdZXtu9RTQ5AfXJYgoK8j796jH3cNTgN5iOrREjYCrFrqc V0etLOrCVXvhsBbOuznLgDoN+n3tQnme0zghB+RINoWZaOji9Ut3FaOWxzcoB6dC Swtg1kZqxhWN1VKtwSbNGX1Q9uqrtR6AoRbtG0CX8S9CDFEidDTonngspY5ICj6i Pq6Wf9DgOqT+fpwDGjV9gjM6etY6TV1vA1pLfwf7JB04faFN4KxnN7UHPvd6R2qa NJ7QfOsGHsTh5sugV2IOAuazl/xUhn3iyftSQ1F2Q+9vRhWW0AgOhWEQzVuFYq8O IB4BnzT5rt6rrsLyKHVOgc6T+3lVP8o3c2s8TXCcbYnbtiy6dlAwrLfSpxcAPXWi K4M4rS5DoxCkgt/sTdMT9BqYVCJjR7ud8Ffn76OfIM6EqYwENAPQqZPHgFE0fzOR kV9thvsvMne/a1zr6Ys1RXK3Yfv+EyxuLvH4houZ0vHfMgzrxOop5/X7u+2TcPaX XWG+U8t+SWngnYDTq93Nwg82rNazKftlYFR8TuMCYedJc+BC1+guhGkqxImA/GlZ 0wBvFkvp73MrOoPaVxHgK8f/TGZK9tggNwIo5vTOeYvCwXwEGAEIACYCGwwWIQSf msRWftyOK4bly5R11mAL706OjwUCXuJH3QUJAuD9FgAKCRB11mAL706Oj2YQD/9q Jzkn1G/palkUfAA6lFisfSY7Ej0INmxJZpI/RHOiOipcZvh2PWzAfTh1w0yLtLw+ D+vEedRpGVKvqS+CjUAKyV6Z3uP48cy93FPnLi/kiDq2eCkB+GZYTNjP2IDsLR5Z YFj/qK5olpmNff3BOMIRR55QIWT0ETgI2oG1KSJ/iqQYBxpQQ51gyWqroP8XaX0W 0Dcii4pscXL/Mj6okDO5QlN7o7pQOGShkWcsqmIbGJ3DOdArGZpi3Cm6zdBarsKg mgKE8XIVZ3XGynUh0K74TIZMGqErX3/RX6Tg8qb2T0vnSwagN8hpI5EyxGquwD0G FUyXtvW6xi2jjCQUzfE59QCpkxOL4vIsf3fRmP9UtlSgf8MhDaTR5Fjqs8C3YoHZ lAK16NmDP+rjkxdlhiPsKRS4I3AOStbtORMPB4ig4MRzRNFUtxZJgFN0yEkf42QL QOBMude+jIsi+zrwI1NqimpqTyKw248s9n+5Fir5rGYj9y3cfBFCqqfz5JqjGknf UiFXaG7MrWFz9S/Dy8for+IFUjHrs22TMGJGZL89gblbGHFqQIjFxTsj/f1ehijk G/WOWDXRMukLUBLk/R3N/sQz9lfWIGi4XdMHk2UU32OQKZU2BgB/xCNQ7U52LnlI Ga1nOpOXmBUUtXKfOgJmGU17aKNMa5SGS72XN4kWPA== =0mJb -----END PGP PUBLIC KEY BLOCK----- Obsolete Luke https://pgp.mit.edu/pks/lookup?op=get&search=0xAD7D030C881A4A29 Type bits/keyID Date User ID pub 1024D/881A4A29 2004-12-26 *** KEY REVOKED *** [not verified] Luke Tidd Public Key Server -- Get "0xad7d030c881a4a29" -----BEGIN PGP PUBLIC KEY BLOCK----- Version: SKS 1.1.6 Comment: Hostname: pgp.mit.edu mQGiBEHOIAIRBACJI1tNSFOEJ0F2xPOpbtnaxqiwoseGKK6oNAFN5k8VntQVQrir5n9A8ocp uFPsB47dsHyBn7X8bFuR1Umz7tgO09lrvfulCvGRLpmSkm2E1VbOEj9IBekbng8ByzjGVLOQ RCAWoqxzdGLSxcTUGutTNRaA0RJms1CJeqQze2PYNwCggeJa5orn6FhZZcmVaaRx4iIxKkcD /1ICPOhXlO1Q0Q1zWQYjuMJIstA7pGeb+wMmN2JEHgra+ClpBU2ZafkHBz+SPR1K85F8ufZY deXcvc3gruBtYrKdlucRbelMT0gMWOtw/2f+mX5n0zvbzJQXIJN+rRJfqs9VVN5emRisvAWy 60Gja/pGBTzkwhM1+4Qt7S1iyQ4qA/9diGL6Ft0oph0zMYZSKAYIArkyEB19UrwX1mhenxsw f7WXrOoW/Vq3y6TBldtfGTclBqpMwgpbfOr5HtYoer007izKIQVzyI2UfEPkCdSc7mlvy3RY rejVMFwgwZ4rUy5miAqmcPYoz8b2ECE4KCt0M0eW0vfk+vMp7Jb4f4N0xYhJBCARAgAJBQJB 7yRNAh0AAAoJEK19AwyIGkopkNEAmgKSZ+rREOPVofe3aNP3KTrxEEh8AJ0UnNPfYJUY8Xk8 GspO0SfenryshrQdTHVrZSBUaWRkIDxjNjBAYmVsbHNvdXRoLm5ldD6IXgQTEQIAHgUCQc4g AgIbAwYLCQgHAwIDFQIDAxYCAQIeAQIXgAAKCRCtfQMMiBpKKbYhAJ9kqM6c2GGCosftce/Y nNDyfkqYegCfa/7irE2R2MKtm8x9O/v5qSMALC65BA0EQc4hbBAQAIHmEr8VvBJ838gTAv++ 8snvnrckpTfLenSUdG5IbbcJrJyWEwMC3V3i669RMpsDO0t8Z9mSa8B12A8pN3g1DqHJAJv6 WRnH3FOOkcI54ypBTUJBYhp8FYOxFEEi7B0a8c4w3/Zic3mXdnvbkHa3n6OnJ11ugFrIVJUz wmynfk69QmaHnY9st0NC/TqRLmcq2iWEbdk76GBA0stiitjAIYwSEkRXZzMYyq3QwJM0FLfg U2xuves+PjfnXbxiEi9l13FSXerhgZWxVkdGIp+3fmuBqa1dUrJLsMLM8IalEKiAue9vFqgH cF/ZVBKpCQnyAZLih9DBzvRF8kiqTgayyjIN+oF7kYIZuwbIBjP1PwFF/ILwKUMIG5uTdyJ1 C0yQj3HC/R7qjB1QRP7pQVvUrjgI6W7oT0t0Cy8thchW5PJmR48zQIepOQodc/EKsLL2l2ii BdTcXdTBsJbOzelMIMnOJaedOqLEZvNFFCIGmoPumm+mOVXAzzOY2R48n4oTE1xwkk3clZ6G 2Lhk0yvdGYsW6+TrhXx53O+JSzOqz4C96SKclIhrVmzJAi+gmIX1giiWnW3jwgrAFFvEoLFa 4egF//beOPVaBFMHe6bXabMLsG4dst+f3O6h1tJk79VTurTHLb4JVrSg7nOJ+Ye03BXFpGzG bftEIwJR5cHn6lurAAMFD/wMQsd/3DgkrlO5diVnBvVuLVrCewKj49aP9j69zsc18BLFcLKu I7/JJBOhiyvAeYiWXF1Z2rdzkHA32eolkNoRyN2zBh+17U9BZxXtiLlqf3y+YmpqzJexn4t1 yXLh23EcHIdSwcQtAH2VRFf+MQHlQY9Z7omQRGILyS6dvBLz0iHHR/XdtMk+6dxlQ+7a/r6s rPAsct5nKa0D8UhYeUzE7xSp7RvH9fBQBM+XRb5Zk/gF7acj5lklO0a4CdLHcDXlxGyLrdyg EZo4HdvK4E2xYm/bnLRMh8uFACWlKOwyi+dpd20R2l6V+dSEvgsRR+WR9PGCel+QlL1tjV6p WKnGvK3WbGPNvNanto1wn0HlbtMBwKUgy3+DzAkpjdjrrJy1tgSnfy+XBw5PkvICx16Za6hT cTcyTxmpwK13UZ9M6+oXEDOaFCF33U+HrR5EL9VMkRpkwJ1qzIwjCjsdlkvEQmN9eZNnakr6 9SZXDVHcQDE3OqmzycutBMmyVRJupcnE+GEHe4eWvEcf+4C9lEcyMxSdI/Pn71UWJvODqOVL wma+4Md83L2JBvUHRSgLuI8AY9cfx9R7EoBa8gEtlRP+CqWpStxl597ZR2qJYFk/gtL4hh3N t0dotxone2fozW1xn7cJnz8qhpchABW1GtpWntawuzwNpCVAT87TeUbbz4hJBBgRAgAJBQJB ziFsAhsMAAoJEK19AwyIGkopb4wAniDf4xRYStRvlozVTmUnqYl2hj+PAJ4tXfl2456EOvqp 5/XNFKNEpAmuaA== =z7x2 -----END PGP PUBLIC KEY BLOCK----- CrowdSec & OpenBSD PF Perimeter Security Setup 1. Overview & Architectural Concept This document outlines the distributed intrusion detection and perimeter enforcement setup across the local infrastructure ( drheck.dev). [ Public Internet (WAN) ] │ ▼ ┌─────────────────────────────────┐ │ danknasty (OpenBSD Router) │ │ ─────────────────────────────── │ │ • OpenBSD PF Kernel Firewall │ │ • Kernel SSH Rate Limiter │ │ • Syslog SSH Forwarder (UDP 514)│ │ • CrowdSec PF Firewall Bouncer │ └────────────────┬────────────────┘ │ Drops banned IPs at WAN edge │ Stream bans via IPv6 LAN ([fd77:...:c70d]:8080) ▼ ┌─────────────────────────────────┐ │ ul (Arch Linux Server) │ │ ─────────────────────────────── │ │ • Dedicated IPv6: ...:c70d/64 │ │ • Nginx Web Reverse Proxy │ │ • Remote Syslog Ingestion (514) │ │ • CrowdSec LAPI Decision Engine │ │ • 53-Hour Ban Policy Engine │ │ • Public Active Enriched JSON │ │ • Append-Only NDJSON Stream │ └─────────────────────────────────┘ Design Philosophy & Conventions Dedicated Service IPv6 Address ( ul): Per local network convention ( 20-wired.network), CrowdSec LAPI listens on a dedicated IPv6 ULA: fd77:7777:5010:0:1337:f1ac:d00d:c70d on port 8080 (LAPI) and UDP port 514 (Syslog Ingestion). Offload All Heavy Detection to Powerful Hardware ( ul): All heavy log analysis, regex pattern matching, SQLite/LAPI database operations, and scenario evaluations run on the Arch Linux server ( ul). Nginx Web Probes & Direct IP Hits: Parses Nginx access logs to detect scanner bots (e.g., .env harvesting, WordPress probes, webshell attempts, open proxy CONNECT attempts, and direct IP connection attempts). Router SSH Probes, Invalid Users & Root Attempts: Receives OpenBSD SSH authentication logs ( /var/log/authlog) forwarded over UDP IPv6 syslog from danknasty to detect SSH brute-force attempts, instant invalid username bans, and instant root attempt bans. 53-Hour Ban Policy: Issues a global 53-hour ban for any detected web probe, direct IP scan, or SSH brute-force/invalid user/root attempt. Perimeter Enforcement at Gateway ( danknasty): The OpenBSD router ( danknasty) runs only the lightweight CrowdSec PF Bouncer ( crowdsec_firewall_bouncer) and native syslogd. Banned IPs are pulled from ul and injected into OpenBSD PF's kernel table. Banned packets are dropped at the network interface before entering the local network. Zero-Overhead Edge SSH Protection ( danknasty): OpenBSD PF tracks TCP connection rates directly in kernel state space ( max-src-conn-rate 3/30). Any bot opening more than 3 SSH connections in 30 seconds is added to the OpenBSD PF table and dropped with 0% CPU and ~0 MB RAM usage. LAN Subnet & Dynamic IPv6 Whitelisting: Private subnets ( 10.0.0.0/24, fd77:7777:5010::/64), loopbacks ( 127.0.0.1/32, ::1/128), and the dynamic public ISP IPv6 prefix (auto-detected via /usr/local/bin/crowdsec-update-ipv6-whitelist) are whitelisted on ul so internal LAN users and SSH typos can never trigger bans. Rich Context & OSINT Enriched Export ( ul): Every banned IP is automatically enriched with Reverse DNS (PTR), Country, Region, City, Coordinates (Lat/Lon), Timezone, ISP, Organization, and Autonomous System (ASN) details. Active Array File: /var/log/crowdsec-bans-public.json ( chmod 644). Append-Only NDJSON Stream: /var/log/crowdsec-bans.json ( chmod 644). Detection Latency & Pipeline Timeline Protection Layer Target Threat Latency Mechanism / Action Layer 1: OpenBSD PF Kernel Limiter SSH Connection Spikes Instant (Nanoseconds / ~0s) In-kernel TCP state tracking ( max-src-conn-rate 3/30) instantly moves abusive IPs into table. Layer 2: CrowdSec Distributed Ban Web Probes, SSH Invalid Users & Root Attempts ~5 seconds (Max 10s) Syslog/Nginx logs stream to ul. ul triggers 53h ban. danknasty bouncer pulls decision and injects IP into OpenBSD PF . CrowdSec Distributed Pipeline Timeline: Event Generation: sshd-session or Nginx emits log event ( < 1 ms). Log Forwarding: UDP IPv6 syslog or Nginx log tailer streams log to ul ( < 1 ms). Scenario Trigger & Ban Creation: CrowdSec parses line, leaks scenario bucket, and writes 53h ban to LAPI database ( ~20–50 ms). Router Bouncer Polling: danknasty bouncer polls ul LAPI over IPv6 LAN ( update_frequency: 10s, average ~5 seconds). Kernel Table Injection: Bouncer executes pfctl -t crowdsec-blacklists -T add ( ~10 ms). 2. Implementation & Configuration Details A. Central Decision Engine on ul Interface Configuration ( /etc/systemd/network/20-wired.network): # crowdsec Address=fd77:7777:5010::1337:f1ac:d00d:c70d/64 Listen URI ( /etc/crowdsec/config.yaml): Bound to dedicated IPv6 ULA [fd77:7777:5010:0:1337:f1ac:d00d:c70d]:8080: api: server: listen_uri: "[fd77:7777:5010:0:1337:f1ac:d00d:c70d]:8080" Nginx Log Acquisition ( /etc/crowdsec/acquis.d/nginx.yaml): filenames: - /var/log/nginx/*.log - /var/log/nginx/access.log labels: type: nginx Router SSH Syslog Ingestion ( /etc/crowdsec/acquis.d/danknasty-ssh.yaml): source: syslog listen_addr: "fd77:7777:5010:0:1337:f1ac:d00d:c70d" listen_port: 514 labels: type: syslog LAN & Dynamic IPv6 Whitelist Override ( /etc/crowdsec/parsers/s02-override/00-lan-whitelist.yaml): name: custom/lan-whitelist description: "Whitelist internal LAN subnets and dynamic public IPv6 prefix" filter: "evt.Meta.source_ip != nil" whitelist: reason: "Exempt internal LAN subnets and dynamic public IPv6 prefix" cidr: - "10.0.0.0/24" - "fd77:7777:5010::/64" - "127.0.0.1/32" - "::1/128" - "2603:3001:626:9d80::/64" # Auto-updated by crowdsec-update-ipv6-whitelist Custom Honeypot Scenario ( /etc/crowdsec/scenarios/honeypot-endpoints.yaml): type: leaky name: custom/honeypot-endpoints description: "Instant trigger on bot/scanner endpoints" filter: | evt.Parsed.request in ["/.env", "/cgi-bin/authLogin.cgi", "/IndoSec.php", "/alfa.php", "/luxe.php", "/BDKR28WP.php"] || evt.Parsed.request matches "(?i)^/(wp-admin|wp-login|wp-includes|\\.env|cgi-bin|alfa\\.php|1\\.php|222\\.php|8\\.php|shell\\.php)" capacity: 1 leakspeed: 1h groupby: evt.Meta.source_ip blackhole: 1m labels: service: nginx type: honeypot_scan remediation: true Custom SSH Invalid User Scenario ( /etc/crowdsec/scenarios/ssh-invalid-user.yaml): Instant 53-hour ban for any attempt to log in using an invalid username: type: leaky name: custom/ssh-invalid-user description: "Instant 53h ban for any SSH login attempt using an invalid user" filter: | evt.Meta.log_type == "ssh_failed-auth" && ( evt.Parsed.sshd_invalid_user != "" || evt.Meta.target_user != "" ) capacity: 1 leakspeed: 1h groupby: evt.Meta.source_ip blackhole: 1m labels: service: ssh type: bruteforce remediation: true Custom SSH Root Attempt Scenario ( /etc/crowdsec/scenarios/ssh-root-login-attempt.yaml): Instant 53-hour ban for any attempt targeting the root account: type: leaky name: custom/ssh-root-login-attempt description: "Instant 53h ban for any SSH login attempt targeting user root" filter: | evt.Meta.log_type == "ssh_failed-auth" && ( evt.Meta.target_user == "root" || evt.Parsed.sshd_invalid_user == "root" || evt.Parsed.sshd_user == "root" ) capacity: 1 leakspeed: 1h groupby: evt.Meta.source_ip blackhole: 1m labels: service: ssh type: bruteforce remediation: true Direct IP Access Scenario ( /etc/crowdsec/scenarios/direct-ip-access.yaml): Instant 53-hour ban for any HTTP/HTTPS connection made directly by IP address or invalid host header: type: leaky name: custom/direct-ip-access description: "Instant 53h ban for any HTTP/HTTPS connection made directly to IP address or non-domain host" filter: | evt.Meta.log_type == "http_access-log" && ( evt.Meta.http_status in ["444", "400", "403"] || evt.Parsed.status in ["444", "400", "403"] || evt.Parsed.target_fqdn matches "^([0-9]{1,3}\\.){3}[0-9]{1,3}$" || evt.Parsed.http_host matches "^([0-9]{1,3}\\.){3}[0-9]{1,3}$" ) capacity: 1 leakspeed: 1h groupby: evt.Meta.source_ip blackhole: 1m labels: service: nginx type: direct_ip_scan remediation: true 53-Hour Ban Policy ( /etc/crowdsec/profiles.yaml): name: default_ip_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 53h notifications: - file_default on_success: break --- name: default_range_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Range" decisions: - type: ban duration: 53h notifications: - file_default on_success: break File Notification Plugin ( /etc/crowdsec/notifications/file.yaml): Appends real-time events upon trigger to /var/log/crowdsec-notifications.log: type: file name: file_default log_level: info group_wait: 0s group_threshold: 1 format: | {{range . -}} {{$alert := . -}} {{range .Decisions -}} {"timestamp":"{{$alert.CreatedAt}}","ip":"{{.Value}}","action":"{{.Action}}","duration":"{{.Duration}}","reason":"{{.Scenario}}","country":"{{$alert.Source.Cn}}","as_name":"{{$alert.Source.AsName}}"} {{end -}} {{end -}} log_path: "/var/log/crowdsec-notifications.log" rotate: enabled: false Host Firewall Rules ( /etc/nftables.conf): ip6 saddr fd77:7777:5010::/64 ip6 daddr fd77:7777:5010:0:1337:f1ac:d00d:c70d tcp dport 8080 accept comment "Allow CrowdSec LAPI from local IPv6 LAN" ip6 saddr fd77:7777:5010::/64 ip6 daddr fd77:7777:5010:0:1337:f1ac:d00d:c70d udp dport 514 accept comment "Allow CrowdSec Syslog Ingestion from local IPv6 LAN" Automated OSINT & Threat Pipeline ( /usr/local/bin/crowdsec-export-bans): Runs every 60 seconds ( crowdsec-export-bans.timer): Updates dynamic public IPv6 whitelist ( /usr/local/bin/crowdsec-update-ipv6-whitelist). Ingests all router SSH auth failures in real time via native UDP IPv6 syslog ( auth.info;authpriv.info -> port 514), triggering instant sub-second bans ( custom/ssh-invalid-user via type: trigger). Updates /var/log/crowdsec-bans-public.json (active JSON array) and appends newly discovered ban events with full GeoIP/ASN/rDNS metadata to /var/log/crowdsec-bans.json ( chmod 644). B. Perimeter Enforcement & Syslog Forwarding on danknasty (Router) Syslog Forwarding Configuration ( /etc/syslog.conf): auth.info;authpriv.info @[fd77:7777:5010:0:1337:f1ac:d00d:c70d]:514 Note: OpenBSD syslogd(8) requires the :514 port suffix when enclosing IPv6 addresses in square brackets @[...]. Without :514, syslogd silently skips UDP forwarding. Bouncer Configuration ( /etc/crowdsec-firewall-bouncer/crowdsec-firewall-bouncer.yaml): mode: pf retry_initial_connect: true update_frequency: 10s api_url: http://[fd77:7777:5010:0:1337:f1ac:d00d:c70d]:8080/ api_key: deny_action: DROP blacklists_ipv4: crowdsec-blacklists blacklists_ipv6: crowdsec6-blacklists pf: anchor_name: "" Router Firewall Rules ( /etc/pf.conf): # CrowdSec Blacklist Tables table persist table persist block drop quick from block drop quick from # Table for SSH Brute-force Auto-Blocking table persist block drop quick from # SSH rules with stateful connection tracking pass in quick on $ext proto tcp to self port ssh \ flags S/SA keep state \ (max-src-conn 10, max-src-conn-rate 3/30, overload flush global) 3. Interaction, Operations & Data Format A. Enriched Public JSON Data Schema The exported JSON files ( /var/log/crowdsec-bans-public.json and /var/log/crowdsec-bans.json) contain rich OSINT, network, and threat context for each banned IP: { "decision_id": 4, "alert_id": 4, "ip": "119.30.119.3", "ip_version": "IPv4", "origin": "crowdsec", "reason": "crowdsecurity/netgear_rce", "action": "ban", "duration": "52h59m18s", "country": "Pakistan", "country_code": "PK", "region": "Punjab", "city": "Sialkot", "latitude": 32.426, "longitude": 74.3644, "timezone": "Asia/Karachi", "isp": "Mobilink GSM, Pakistan Mobile Communication Ltd.", "org": "Linkdotnet Telecom Limited", "as_name": "AS58470 IX Peering for Mobilink and Link Direct International.", "rdns": "static-host119-30-119-3.link.net.pk", "target_uri": "/setup.cgi?next_file=netgear.cfg&todo=syscmd&cmd=rm+-rf+/tmp/*...", "user_agent": "-", "method": "GET", "status_code": "301", "created_at": "2026-08-05T02:44:21Z" } B. Viewing Active System State & OpenBSD PF Kernel Tables 1. View Active Bans on ul (CrowdSec LAPI) # View all active decisions (untruncated): sudo cscli decisions list -a # Query if a specific IP is banned in CrowdSec: sudo cscli decisions list --ip 2. View Active Enriched JSON Array cat /var/log/crowdsec-bans-public.json 3. Tail the Append-Only Ban Stream (NDJSON) tail -f /var/log/crowdsec-bans.json 4. View Active Bouncers & Connection Status sudo cscli bouncers list 5. View Detection Metrics & Statistics Shows lines read via Nginx logs and remote router syslog: sudo cscli metrics 6. View Active OpenBSD PF Kernel Tables on Router ( danknasty) # List all blocked IPs in CrowdSec's PF Radix Tree table: SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t crowdsec-blacklists -T show" # View packet & byte drop statistics for the CrowdSec PF table: SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -v -t crowdsec-blacklists -T show" # Test if a specific IP is currently blocked in the OpenBSD PF kernel table: SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t crowdsec-blacklists -T test " # View OpenBSD SSH kernel brute-force bans: SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t bruteforce -T show" C. Manual Interactivity & Ban Management 1. Manually Ban an IP for 53 Hours sudo cscli decisions add --ip --duration 53h --reason "Manual administrative ban" 2. Remove a Ban (Unban an IP) sudo cscli decisions delete --ip 3. Force Immediate Synchronization & OSINT Enrichment sudo /usr/local/bin/crowdsec-export-bans 4. Inspect Recent Detection Alerts sudo cscli alerts list sudo cscli alerts inspect 5. Flush Banned IP Tables on the Router (Emergency Reset) SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t crowdsec-blacklists -T flush" D. Whitelisting Local Trusted Networks Internal LAN subnets and your dynamic public IPv6 prefix are protected from accidental bans via /etc/crowdsec/parsers/s02-override/00-lan-whitelist.yaml: name: custom/lan-whitelist description: "Whitelist internal LAN subnets from all CrowdSec scenarios" filter: "evt.Meta.source_ip != nil" whitelist: reason: "Exempt internal LAN subnets" cidr: - "10.0.0.0/24" - "fd77:7777:5010::/64" - "127.0.0.1/32" - "::1/128" - "2603:3001:626:9d80::/64" After modifying whitelists or scenarios, reload CrowdSec: sudo systemctl reload crowdsec E. Adding More Scanner Endpoints /etc/crowdsec/scenarios/honeypot-endpoints.yaml Current Configuration: type: leaky name: custom/honeypot-endpoints description: "Instant trigger on bot/scanner endpoints" filter: | evt.Parsed.request in ["/.env", "/cgi-bin/authLogin.cgi", "/IndoSec.php", "/alfa.php", "/luxe.php", "/BDKR28WP.php"] || evt.Parsed.request matches "(?i)^/(wp-admin|wp-login|wp-includes|\\.env|cgi-bin|alfa\\.php|1\\.php|222\\.php|8\\.php|shell\\.php)" capacity: 1 leakspeed: 1h groupby: evt.Meta.source_ip blackhole: 1m labels: service: nginx type: honeypot_scan remediation: true How to Add More Scanner Endpoints Add new exact paths to the evt Parsed.request in [...] list (e.g. "/setup.php", "/config.json"), or append regex patterns to evt.Parsed.request matches "(?i)...". Reload Crowd $ sudo cscli scenarios reload $ sudo systemctl reload crowdsec Service List Runtime Architecture Note: All 30 containerized applications on ul run under rootless Podman Quadlets ( docker-data, UID 1100) fronted by Nginx TLS + Authentik SSO on dedicated eth77 ULA IPv6 addresses. See podman for full operational reference. Anchor Anchor is a fast, self-hosted collaborative note-taking and checklist application integrated with Authentik OIDC Single Sign-On and backed by PostgreSQL on ul. AudioMass AudioMass is a full-featured, open-source web-based audio waveform editor and recorder. Running entirely inside the browser using Web Audio and WebAssembly, it provides users with a desktop-class digital audio editing workspace without requiring local software installation. Users can load audio files, record microphone input, view real-time visual spectrums, cut, copy, paste, trim, and mix audio tracks. The tool includes an extensive suite of built-in DSP effects and processing tools, such as equalizers, gain adjusters, compressors, pitch shifters, delay, reverb, and noise gates. It allows quick audio editing, podcast clips creation, or sound snippet modifications with instant rendering and export in WAV or MP3 formats directly from the browser. Authentik Identity Provider Authentik is the central OpenID Connect (OIDC), ForwardAuth proxy, and LDAPS ( ldap.drheck.dev:636) Single Sign-On identity provider powering unified authentication across web apps and Linux workstations on drheck.dev. BentoPDF BentoPDF is a privacy-preserving, browser-based PDF manipulation toolkit gated behind Authentik ForwardAuth for merging, splitting, compressing, rotating, watermarking, and converting PDF documents with zero external cloud uploads. BookStack BookStack is a wiki platform designed to make storing and organizing information simple and structured. BookStack breaks content down into an intuitive real-world book hierarchy: Shelves, Books, Chapters, and Pages. This visual structure ensures that technical documentation, standard operating procedures, personal notes, and knowledge bases remain easy to search, navigate, and maintain. BookStack offers both WYSIWYG rich text and Markdown editors, built-in version control with page revisions, image management, cross-linking, and granular role-based access controls. Single Sign-On support: yes, with SSO can visit the site and be signed in directly. Camera Hub ( cam) DrHeck Camera Hub provides live low-latency video and MP3 audio streaming, two-way RTSP ONVIF backchannel talk, YOLOv8n + hardware AI event capture timelines, 5X optical zoom controls, floodlight/IR controls, and a Google TV 10-foot UI for the PoE Reolink RLC-811A cameras, restricted to the camera_access group. Console Games (Arcade Launcher) Console Games is a web-based retro gaming launcher and front-end interface built on top of webЯcade. It provides end users with an interactive, TV- and controller-friendly library for browsing and launching classic console video games directly in modern web browsers using WebAssembly emulation cores. Users can connect gamepad controllers via USB or Bluetooth, save game states, customize controller mappings, and stream classic retro titles across desktop computers, mobile devices, and smart TVs without installing native emulators or software dependencies. Degoog Search Degoog is a self-hosted privacy metasearch engine at https://search.drheck.dev ( https://s.drheck.dev) backed by host Valkey caching without tracking or ad profiling. Demucs Stem Splitter Demucs is a GPU-accelerated music source separation studio running on the AMD Radeon AI PRO R9700 ( ROCm), splitting songs into isolated vocals, drums, bass, guitar, piano, and accompaniment stems. Drawpile Drawpile is a free, open-source collaborative drawing program that allows multiple artists to draw on the same canvas simultaneously over the internet. The server component hosted on drheck.dev manages shared session state, brush strokes, layer ordering, and user permissions for real-time multiplayer digital art sessions. Users can host private or public drawing rooms, sketch together in real time, record and playback drawing sessions, lock specific layers to prevent accidental edits, and chat with co-creators. Authentik integration ensures authenticated users can easily jump into ongoing canvas sessions or manage drawing servers. This is just the server, create a room and connect with your client. Excalidraw Excalidraw is a virtual, hand-drawn style collaborative whiteboard application designed for quick sketching, architecture diagrams, wireframes, and visual brainstorming. Its distinctive hand-drawn aesthetic removes the pressure of high-fidelity graphics, encouraging rapid ideation and clear visual communication. Featuring real-time end-to-end encryption, infinite canvas navigation, rich shape libraries, customizable arrows, text blocks, and exported PNG/SVG diagrams, Excalidraw allows end users to draft system architectures, mind maps, and workflow diagrams quickly within their browser. Funkwhale Funkwhale is a self-hosted, decentralized audio and music streaming platform built on open web protocols (ActivityPub). It allows end users to import, organize, stream, and share their personal music collections and podcasts across devices while connecting with audio libraries on other federated instances. End users can create custom playlists, follow public audio channels, listen to high-quality audio streams in the web application or via Subsonic-compatible mobile apps, and share music tracks with friends without depending on centralized commercial streaming services. Forgejo Forgejo is a self-hosted, community-governed lightweight software forge that provides complete software development workflows similar to GitHub. It offers source code hosting, Git repository management, pull requests, code reviews, issue tracking, wikis, and organization management in a fast, low-footprint application. Integrated with Authentik via Single Sign-On (OIDC), users can log into Forgejo with their unified credentials to push and pull Git repositories, manage SSH keys, configure webhook integrations, run automated CI/CD pipelines, and collaborate on software projects. Grill Monitor ( FireBoard 2) Local cloud-free BLE-to-Wi-Fi telemetry and multi-channel temperature graphing dashboard ( grill.drheck.dev / fireboard.drheck.dev) for the hardware-switched Seeed XIAO ESP32-C3 + FireBoard 2 6-probe smoker thermometer. Immich Immich is a high-performance, self-hosted photo and video management solution designed as a privacy-respecting alternative to Google Photos or Apple Photos. It provides seamless background photo backup from mobile devices (iOS and Android) as well as an intuitive web interface for browsing, searching, and sharing media libraries. Immich includes machine-learning features such as face recognition, object detection, spatial map views based on EXIF GPS data, album sharing, and multi-user support with quota management. SSO authentication via Authentik allows smooth, secure onboarding across all web and mobile client applications. Jellyfin Jellyfin is a free, open-source media system that puts users in control of managing and streaming their personal movies, TV shows, music, and live TV broadcasts. Unlike commercial alternatives, Jellyfin contains no tracking, premium paywalls, or central server dependencies, running entirely self-hosted. Jellyfin automatically fetches metadata, artwork, and subtitle tracks for media libraries, transcoding video streams on-the-fly for smooth playback across web browsers, smart TVs, Roku, Fire TV, and mobile devices. Authentik LDAP integration allows unified credentials for all household members. Kanji Practice Kanji Practice is a specialized web application built for Japanese language learners to practice stroke order. There are two modes. In the first, each stroke will be animated in the correct order in a single pane, or there is an view showing each character being assembled one stroke at a time in successive panes. Kavita Kavita digital library designed specifically for reading and managing digital comics, manga, light novels, and traditional e-books. It automatically organizes files (EPUB, CBZ, CBR, PDF) into series, volumes, and chapters with cover art and metadata tagging. Kavita includes built-in responsive web readers tailored for mobile devices, tablets, and desktop screens, offering custom reading modes (continuous scroll, dual page, manga right-to-left mode), bookmark sync, reading progress tracking, and user library permissions. LibreSprite LibreSprite is an open-source animated sprite editor and pixel art tool compiled to WebAssembly. Running 100% client-side in the browser, it provides a full-featured digital drawing and animation workspace for creating 2D game assets, animated GIFs, and pixel graphics without requiring local software installation. Users can draw pixel artwork, construct frame-by-frame sprite animations, manage custom color palettes, and export finished graphics as sprite sheets or animated images, bringing classic pixel art editing into a cloud-hosted web environment. Minecraft Control Plane Minecraft Control Plane ( mc-manager) is the unified web administration and world provisioning studio for all Minecraft servers hosted on ul. Protected behind Authentik ForwardAuth, it provides real-time server lifecycle management ( On-Demand wake-on-join via HAProxy, Always On, and Manual), a live WebSocket PTY server console ( mc-runner), interactive cubiomes 2D biome & structure seed previews, and a built-in Minecraft Text, MOTD, Color Gradient & 8-Bit Pixel Icon Studio with live Multiplayer Server List ping updates. It also integrates with automated network, HAProxy SNI, and DNS provisioning ( ipctl using short-lived step-ca SSH certificates and external config files) as well as 3D world maps such as PokeLovers BlueMap. Navidrome Navidrome ( navidrome.drheck.dev / music.drheck.dev) is a modern, lightweight, open-source music server and streamer that gives users complete freedom to listen to their personal music collection from anywhere. Built with high performance in mind, Navidrome indexes large music libraries effortlessly with low CPU and memory overhead. Navidrome features a responsive web UI and full compatibility with Subsonic/Airsonic mobile clients (such as DSub, amposer, and Symfonium). Users can stream music in original quality or on-the-fly transcoded formats, create playlists, mark favorites, and share music with family members. Nextcloud & Collabora Online Nextcloud is a productivity and cloud storage platform that provides file storage and access, documents, calendars, contacts, and communication tools. It serves as a privacy-focused alternative to proprietary cloud services like Google Drive and Microsoft 365. End users can sync files across desktop and mobile clients, edit office documents collaboratively in real time via Collabora Online ( office.drheck.dev), schedule events, manage task lists, and share files securely via public password-protected web links. Authentik OIDC Single Sign-On grants secure access across all Nextcloud modules. Penpot Penpot is the first open-source, web-based design and prototyping tool built natively on open web standards (SVG). Created as an alternative to proprietary platforms like Figma, Penpot allows design and development teams to craft UI/UX wireframes, interactive component prototypes, and vector illustrations. Because Penpot uses SVG as its native domain model, vector designs render with absolute fidelity for web Developers. It supports real-time multiplayer collaboration, shared design systems, component libraries, layout grids, and CSS code generation. Single Sign-On via Authentik enables instant user authentication. Pixelorama Pixelorama is a free, open-source pixel art editor built with the Godot engine and deployed to the web for creating 2D sprites, tilesets, pixel animations, and retro game art directly in the browser. It features a rich toolkit tailored specifically for pixel artists, including custom pencil and eraser brushes, color palettes, onion skinning for frame-by-frame animation, layer management, pixel-perfect lines, tile mode previewing, and export options to sprite sheets or animated GIFs. Plex Plex is a comprehensive media streaming application that organizes personal video, audio, and photo collections into a clean, beautifully formatted media streaming service. Plex automatically matches files against online databases to attach movie posters, plot summaries, cast lists, trailers, and themes. End users can stream content seamlessly across a vast ecosystem of devices, including smart TVs, streaming sticks, mobile phones, game consoles, and web browsers, with intelligent server-side video transcoding to match client device capabilities and network bandwidth. RomM (ROM Manager) RomM (ROM Manager) is a modern web application designed for organizing, scanning, and managing retro video game ROM libraries. RomM connects with online databases (such as IGDB and SteamGridDB) to automatically enrich game titles with high-resolution box art, screenshots, release dates, genre tags, and game descriptions. End users can browse their game collection by console platform, search titles, manage save states and ROM variants, and launch games directly in the browser using integrated web emulators or send game files to local emulation devices. Sopro Text-to-Speech Sopro is a self-hosted neural text-to-speech synthesis web studio gated behind Authentik ForwardAuth on ul. Studio Control Gateway ( scontrol) Studio Control ( scontrol.drheck.dev / studio-kvm-web.drheck.dev) orchestrates one-touch RF power sequencing, ATX motherboard ignition, and Echo AudioFire 12 / Ardour DAW readiness across the recording studio rack. SubOCR Japanese Subtitle Study Platform SubOCR extracts burned-in Japanese video subtitles via ROCm PP-OCRv5 GPU inference on the AMD Radeon AI PRO R9700, refines dialogue via local qwen2.5:32b, and generates Kanji-only furigana, JLPT tags, conjugation tables, and interactive video study decks. Sunshine Game Stream Host Bare-metal KMS/DRM Gamescope + Sunshine low-latency streaming server running directly on the AMD Radeon AI PRO R9700 with Steam libraries hosted on ZFS /data/games/steam. tldraw tldraw is a collaborative, infinite-canvas drawing tool and visual whiteboarding app designed for instant sketching, diagramming, and visual note-taking. Built around speed and simplicity, tldraw provides a minimal canvas interface with immediate tool accessibility. End users can create vector shapes, freehand ink lines, sticky notes, connective arrows, and text labels. Supporting multiplayer real-time collaboration and SVG/PNG exports, tldraw is ideal for quick technical diagrams, architecture brainstorming, and informal remote collaboration. Transmission Transmission is a fast, lightweight BitTorrent client equipped with a clean web interface for managing peer-to-peer file transfers remotely. The web UI allows users to monitor active downloads and uploads, add new torrent files or magnet links, adjust bandwidth rate limits, and organize downloads into destination folders. Protected behind Authentik ForwardAuth reverse-proxy authentication, users can safely monitor long-running torrent transfers and manage media downloads on drheck.dev from any web browser without exposing unauthenticated torrent control endpoints to the public internet. Vaultwarden Vaultwarden ( pw.drheck.dev / vault.drheck.dev) is a lightweight, self-hosted implementation of the Bitwarden secrets and password management server API written in Rust. It provides full compatibility with official Bitwarden desktop applications, browser extensions, mobile apps, and CLI tools while consuming minimal memory resources. End users can securely store end-to-end encrypted login credentials, credit cards, secure notes, personal identities, and two-factor authentication (TOTP) seed keys. Enforced with Authentik OIDC Single Sign-On and strict password policies, Vaultwarden ensures seamless, secure password management across all user devices. (secrets) Vikunja Task Manager Vikunja is a self-hosted project, Kanban board, Gantt chart, and task management platform backed by host PostgreSQL, host Valkey caching, and Authentik OIDC Single Sign-On. Wavacity Wavacity is an open-source web browser port of the popular Audacity audio editing application. Compiled to WebAssembly, it brings Audacity's iconic multi-track audio editing workspace directly into modern web browsers without requiring local software installation or native binaries. Users can import multi-track audio files, perform non-destructive editing, apply audio filters, cut and align tracks, inspect frequency spectrums, and export finished master tracks in standard audio formats, carrying the familiar Audacity interface into a cloud-hosted web environment. webЯcade Setup webЯcade is a cloud-based retro gaming ecosystem that allows users to play classic console and arcade games directly inside modern web browsers across desktop computers, mobile phones, tablets, and smart TVs. The webЯcade setup application allows users and administrators to configure game feeds, set up cloud save game synchronization, map custom controller inputs, and manage retro emulator settings for seamless web-based retro gaming. Webshell Webshell is a web-based terminal interface (powered by ttyd and short-lived 5-minute step-ca SSH certificates) that provides authenticated users with direct, browser-based command-line shell access to host environment tools. Protected behind Authentik Single Sign-On, Webshell enables administrators and authorized users to safely perform terminal administration tasks, check system diagnostics, run scripts, and manage files from mobile devices or workstations without needing native SSH client software. Wick Editor Wick Editor is an open-source, web-based tool for creating animations, games, and interactive code projects. Combining the accessibility of classic Flash software with modern web standards (HTML5/JavaScript), Wick Editor makes interactive media creation approachable directly in the browser. Users can draw vector artwork, build timeline-based frame animations, program interactivity with beginner-friendly JavaScript code snippets, attach sound effects, and export projects as interactive HTML5 games, standalone web apps, animated GIFs, or video files. WIPStudio WIPStudio (Work-In-Progress Studio) is a self-hosted creative media and web development studio platform designed for managing ongoing multimedia projects, interactive web prototypes, and creative assets. Integrated with Authentik via OpenID Connect (OIDC), WIPStudio provides authenticated creators with a unified workspace to draft visual concepts, organize project files, review media builds, and collaborate on experimental digital projects. YOURLS Shortener ( drheck.dev) YOURLS powers custom short links on https://drheck.dev/, with public HTTP 301 resolution and private admin/API management protected behind Authentik ForwardAuth ( /yourls-api.php and /admin/). SSH Command Tracking Guide Detail of command execution auditing and tracking across human SSH sessions and AI agent sessions ( amux). Part 1: Arch Linux ( ul, bmo95, studio-kvm-ssh) The Linux Audit Framework ( auditd) captures execve system calls triggered by human user sessions ( auid >= 1000), recording all interactive and SSH payload commands while strictly filtering out background system and rootless Podman container noise. 1. Installation & Service Setup Install the audit package and enable the auditd daemon: $ sudo pacman -S --noconfirm audit $ sudo systemctl enable --now auditd 2. Configuration Rules & Interactive User Filtering Create /etc/audit/rules.d/execve.rules with Audit User ID filtering ( auid >= 1000): CRITICAL: Do NOT include catch-all execve rules without -F auid>=1000. Background system daemons and rootless Podman containers started by user@1100.service (e.g. MinIO healthchecks, Authentik workers) have auid = 4294967295 ( unset). An unfiltered rule generates ~8 MB of log noise every 7 minutes, rotating out entire log histories within 30 minutes. $ sudo tee /etc/audit/rules.d/execve.rules > /dev/null << 'EOF' # Delete pre-existing rules -D # Set kernel buffer size -b 8192 # Log all command executions strictly for human/login users (auid >= 1000) -a always,exit -F arch=b64 -S execve -F auid>=1000 -k remote_commands -a always,exit -F arch=b32 -S execve -F auid>=1000 -k remote_commands EOF Load rules into the active kernel: $ sudo auditctl -D && sudo augenrules --load $ sudo auditctl -e 1 Verify active rules: $ sudo auditctl -l Process UID ( uid) vs Audit User ID ( auid) Process UID ( uid=1100 docker-data & subordinate 624288– 689823): Controls unprivileged host kernel UIDs for rootless Podman containers. Audit User ID ( auid / loginuid): Assigned exclusively by PAM when a human authenticates via SSH or TTY ( auid = 1000). Background containers started via systemd lingering ( user@1100.service) have auid = 4294967295 ( unset). Result: Filter by auid >= 1000. Part 2: Agent Execution Auditing ( amux / systemd journal) When AI agents or automated scripts run multi-line code via write-pane -f or write-pane -b in amux, those commands can bypass standard auditd execve logging (because they are streamed directly into an already-running shell's PTY or an interactive Python/Node REPL). To ensure complete, tamper-resistant auditing and forensic recovery: 1. Structured Journal Logging Every call to write-pane in amux automatically logs directly into the systemd journal with structured fields: SYSLOG_IDENTIFIER=amux AMUX_PANE_ID: Target pane ( %0, %1, etc.) AMUX_WINDOW_NAME: Hierarchical task window (e.g. mc-pokelovers-bluemap) AMUX_SOURCE: Original script file path or base64 AMUX_ARCHIVE_PATH: Permanent snapshot location in /data/amux/scripts/ AMUX_PAYLOAD_SIZE: Injected byte count Querying agent execution logs: # Stream live agent injections $ journalctl -t amux -f # Inspect detailed metadata for recent agent commands $ journalctl -t amux -n 20 -o verbose 2. Permanent Script Archival ( /data/amux/scripts/) Before any byte is written to the terminal PTY, amux automatically saves a bit-for-bit snapshot to: /data/amux/scripts/YYYY-MM-DD_HHMM.SS_.sh Durability: Stored on /data with ownership luke:luke. Zero Loss: Even if an agent immediately removes a temporary file from /dev/shm after execution, the exact script executed is preserved permanently. Disaster Recovery: /data/amux/scripts/ is automatically backed up during the daily backup run via /home/luke/git/backups/ul/scripts/backup/amux. Part 3: Python Command Inspection & Summarization ( audit-log) The /usr/local/bin/audit-log Python utility parses, decodes, and summarizes EXECVE records from /var/log/audit/audit.log.*. CLI Command Options & Usage # 1. Default view: last 20 executed commands $ audit-log # 2. View specific number of entries (e.g. last 50 commands) $ audit-log 50 $ audit-log -n 50 # 3. Stream all recorded command history $ audit-log all $ audit-log --all # 4. Filter by specific day (YYYY-MM-DD) $ audit-log --day 2026-08-29 # 5. Filter date range (since date / until date) $ audit-log --since 2026-08-25 --clean # 6. Filter repetitive background noise (tmux status bars, pollers) $ audit-log --clean -n 30 # 7. Regex / substring search across command history $ audit-log --search "funkwhale|podman" --clean # 8. Deduplicate identical commands in output $ audit-log --unique --since 2026-08-28 # 9. Generate automated daily change log summary outline $ audit-log --daily --since 2026-08-25 --clean Part 4: OpenBSD (Router danknasty) OpenBSD uses native kernel process accounting ( acct) to track binary executions, alongside syslog-based sshd verbose logging and doas auditing. 1. Kernel Process Accounting ( acct) Enable Accounting: # Create log file $ doas touch /var/account/acct # Enable accounting at boot $ echo "accounting=YES" | doas tee -a /etc/rc.conf.local # Start accounting daemon immediately $ doas /usr/sbin/accton /var/account/acct 2. Real-Time Command Inspection ( audit-log) Query process execution history using the matching audit-log helper script: # View the last 20 executed commands (Default) $ doas audit-log # View a specific number of recent commands (e.g. 50 entries) $ doas audit-log 50 # View ALL recorded command history $ doas audit-log all 3. SSH Verbose Payload & doas Logging To log non-interactive SSH connections and payload commands to /var/log/authlog: Edit /etc/ssh/sshd_config: LogLevel VERBOSE Reload sshd: $ doas rcctl reload sshd Monitor logs: $ doas tail -f /var/log/authlog Agent's Guide to Wiki Agent Guide: Updating BookStack Wiki Pages This guide outlines how AI agents can directly query, read, edit, and update pages in the self-hosted BookStack Wiki (https://wiki.drheck.dev/). 1. Golden Rules for Agents 🚨 MANDATORY RULE: READ LIVE CONTENT FIRST BEFORE EDITING NEVER generate or overwrite page content from memory or scratch without first reading the current live page and its latest revisions. Humans frequently make live edits in the BookStack browser interface between agent turns. Overwriting a page without reading its latest markdown and page_revisions destroys user contributions, custom links, code blocks, and embedded images. Every page update MUST follow this sequence: Fetch live markdown to a file via /home/luke/bin/read-bookstack-page . Query attached images from the images table. Incorporate existing content, user edits, and image tags into the updated document. Deploy changes with /home/luke/bin/update-bookstack-page . Pull down the live rendered page and verify HTML and KaTeX formula output via /home/luke/bin/verify-bookstack-page . 🚨 MANDATORY RULE: PULL DOWN & VERIFY THE RENDERED PAGE & FORMULAS AFTER EVERY EDIT Mathematical formulas ( $$...$$ and $...$), signal chains ( \longrightarrow), matrices, and TeX output are desired and encouraged across the wiki—but once an edit is made, you MUST pull down the live rendered page and check the output. Never assume a formula rendered properly just because the Markdown source looks right or LENGTH(html) is non-zero in MariaDB. BookStack's server-side CommonMark parser transforms backslash-punctuation sequences before client-side KaTeX runs, and unquoted shell expansions can corrupt either the page source or the global app-custom-head KaTeX loader. After every wiki edit, you MUST: Run /home/luke/bin/verify-bookstack-page (also executed automatically by update-bookstack-page and create-bookstack-page), which pulls down the live rendered page over HTTPS ( https://wiki.drheck.dev/books//page/) and validates: Every inline