# System Setup # System Change log - 2026-10-20 Tuesday - **Forgejo Two-Factor Authentication (`git.drheck.dev`) Cipher Migration Fix (`ul`)**: - Resolved `HTTP 500 Internal Server Error` on `POST /user/two_factor` (`UserSignIn: chacha20poly1305: message authentication failed`) caused by legacy Gitea `AES-CFB` (`md5(SECRET_KEY)`) + double-Base64 ciphertext remaining in `two_factor.secret` after the Gitea $\longrightarrow$ Forgejo `16.0.6` migration. - Decrypted the existing 64-character Base32 TOTP seed without requiring authenticator re-enrollment, re-sealed `two_factor.secret` using Forgejo `16.0.6`'s `modules/keying` HKDF-SHA256 (`totp` subkey) + `XChaCha20-Poly1305` AEAD with `ColumnAndID("secret", id)` big-endian associated data, verified all 27 `forgejo doctor check --all` checks (`OK`), and refreshed PAR2 backups. - **Docker Package Removal, Root NVMe Reclamation & ZFS `data/podman` Dataset Migration (`ul`)**: - Uninstalled `docker` (`1:29.9.1-1`), `docker-compose` (`5.6.0-1`), and `containerd` (`2.4.2-1`) via `pacman -Rns` and purged `/var/lib/docker`, `/var/lib/containerd`, and `/etc/docker` (`43 GB` freed on `/`). - Provisioned dedicated ZFS dataset `data/podman` (`compression=lz4`, `xattr=sa`, `acltype=posix`, `dnodesize=auto`, `atime=off`, `mountpoint=/home/docker-data/.local/share/containers`) and migrated Podman's `67 GB` image/layer store from the root NVMe onto the 160 TB ZFS pool via two-pass `rsync -aHAX`. - Cleaned `39 GB` of obsolete package tarballs from `/var/cache/pacman/pkg` (`paccache -rk2` + `paccache -ruk0`) and enabled weekly `paccache.timer`. - 2026-10-19 Monday - **Complete 30-Service / 40-Container Rootless Podman Cutover & Docker Decommissioning (`ul`)**: - Decommissioned `docker.service`, `docker.socket`, and `containerd.service` completely (`inactive` / disabled), removed all 10 leftover Docker kernel bridges (`docker0` + `br-*`), archived legacy artifacts into `/data/backups/legacy_docker_archive/`, and replaced `docker-prune.timer` with rootless `/etc/systemd/system/podman-prune.{service,timer}` (`User=docker-data`). - Migrated all remaining 19 services (29 containers) alongside the initial 11 standalone services—totaling **all 30 containerized services (41 containers)**—to rootless Podman Quadlets (`docker-data`, `UID 1100`, subordinate range `docker-data:624288:65536`, `Pull=never` offline boot resilience, `/home/docker-data/.config/containers/systemd/*.{container,network}`). - Migrated shared Valkey to native host `valkey.service` bound strictly to `127.0.0.1:6379` with per-service ACL users (`authentik`, `immich`, `nextcloud`, `vikunja`, `degoog`), `default` disabled, and dangerous admin commands disabled; locked host `postgresql.service` strictly to `127.0.0.1:5432` and upgraded the legacy MD5 role password to `SCRAM-SHA-256`. - Configured host HAProxy Layer-4 TCP passthrough (`/etc/haproxy/conf.d/ldap.cfg`) relaying ports `389`/`636` to rootless `authentik-ldap` (`:3389`/`:6636`) and removed `/etc/nginx/certs:/certs:ro` from all Authentik containers so zero host TLS/mTLS keys (`pikvm_client.key`, `studio_secret.conf`) are mounted inside containers. - Remediated security & operational findings uncovered during migration: - Enabled `YOURLS_PRIVATE=true`, rotated weak `yourls` credentials (`admin_pass_9213` / `yourls_db_pass_9213`), and gated `https://drheck.dev/yourls-api.php` behind Authentik ForwardAuth while keeping short-link `301` redirects public. - Removed `/home/luke:/host_home:ro` mount from `subocr`, restricted Collabora WOPI wildcard `https://.*` strictly to `cloud\.drheck\.dev`, set `DISABLE_SIGN_UP=true` and `0770` permissions on `wipstudio`, and archived the deleted upstream `minio/minio` image at `/data/docker/wipstudio/minio_image_backup.tar.xz`. - Fixed silent data-loss gaps in `/home/luke/git/backups/ul/scripts/`: pointed `backup/nextcloud` + `restore/nextcloud` at live `nextcloud_db` (instead of stale `nextcloud`) + `config.php`; migrated `penpot` Postgres & assets out of `/var/lib/docker/volumes` onto ZFS `/data/docker/penpot/{postgres_data,assets}` and added `assets` to `backup/penpot`; added `pg_dumpall -U funkwhale` to `backup/funkwhale`; added `authentik-*.container` to `backup/authentik`; and expanded `backup/systemd` to archive `/etc/containers`, `/etc/haproxy`, `/etc/nftables.conf`, and `/home/docker-data/.config/containers/systemd`. - Fixed `HTTP 500` ForwardAuth errors on secondary hostname aliases (`music.drheck.dev` $\longrightarrow$ `navidrome.drheck.dev`, `scontroll.drheck.dev` $\longrightarrow$ `scontrol.drheck.dev`) and OIDC callback host mismatches (`vault.drheck.dev` $\longrightarrow$ `pw.drheck.dev`, `bookstack.drheck.dev` $\longrightarrow$ `wiki.drheck.dev`) with dedicated TLS `301` redirects across `/etc/nginx/sites-available/`. - Pruned a stale `aardvark-dns` entry in `/run/user/1100/containers/networks/aardvark-dns/penpot_net` (`10.89.13.2` ahead of `10.89.13.5`) that caused `penpot-backend` Lettuce Redis timeouts, fixed root-owned cache generation in `/home/luke/bin/read-bookstack-page` (`-u abc -e HOME=/config`), and updated `/usr/local/bin/{update-authentik-cert.sh,mc-bluemap-provision,degoog-update}`, `/data/minecraft/ipctl`, `/etc/nftables.conf`, and all 43 PAR2-verified service backups. - **Backup Architecture Decoupling (`ul`)**: - Refactored `/home/luke/git/backups/ul/scripts/run_all_backups` so default invocations execute fast local snapshots and `par2` verification only (`--all` runs the full local + cloud + ISO pipeline). - Extracted standalone `/home/luke/git/backups/ul/scripts/stage_cloud_archive`, `/home/luke/git/backups/ul/scripts/upload_cloud_archive`, and `/home/luke/git/backups/ul/scripts/stage_iso_archive` scripts and updated [Backup Process](https://wiki.drheck.dev/books/system-setup/page/backup-process). - **amux Synchronous Command Waiting (`ul`)**: - Implemented native event-driven command completion waiting in `amux` (`/home/luke/git/amux`, installed at `/usr/local/bin/tmux-custom`) via `read-pane -w` (`-S `, `-e `, `-E `, `-O `, `-t `), eliminating polling loops. - Added `atx-exec` helper to `/home/luke/.agent_profile` and updated [amux - Agent modified tmux](https://wiki.drheck.dev/books/system-setup/page/amux-agent-modified-tmux) and [SSH & Agent Command Tracking Guide](https://wiki.drheck.dev/books/system-setup/page/ssh-command-tracking-guide). - **Gitea $\longrightarrow$ Forgejo Migration (`ul`)**: - Migrated [git.drheck.dev](https://git.drheck.dev) from Gitea (`28.1.0-1`) to Forgejo (`16.0.6-1`), renaming ZFS dataset `data/gitea` to `data/forgejo` (`/data/forgejo`), PostgreSQL database/role to `forgejo`, configuration to `/etc/forgejo/app.ini`, and working state to `/var/lib/forgejo` (`User=git`). - Added `/etc/tmpfiles.d/forgejo.conf` (`git:git` override) to prevent pacman `30-systemd-tmpfiles.hook` from resetting `/var/lib/forgejo` and `/etc/forgejo/app.ini` ownership to `forgejo:forgejo` during package upgrades. - Updated Nginx (`/etc/nginx/sites-available/forgejo`), Authentik OIDC provider/application (`Forgejo` / `forgejo`), `pass` entries (`forgejo/*`), `repo_tools` (`forgejo-create`, `forgejo-delete`, `forgejo-Visibility`, `gpush`), and backup/restore scripts (`/home/luke/git/backups/ul/scripts/{backup,restore}/forgejo`). - **Kasm Decommissioning & Rootless Podman Quadlet Migration (`ul`)**: - Decommissioned `kasm` (`kasm.drheck.dev`, `/data/docker/kasm`, PostgreSQL `kasm_db`, ULA IPv6 `:ca53`, Authentik OIDC app, Nginx config, Unbound DNS, and `/data/kasm_profiles`) and removed obsolete `/data/docker/sunshine` container stack. - Configured rootless Podman (`6.1.4-1`) with `passt` user-mode networking, subordinate UID/GID ranges (`docker-data:100000:65536`), ZFS-backed graphroot (`/data/podman/docker-data/storage`), and systemd user lingering (`loginctl enable-linger docker-data`). - Migrated 11 standalone single-container services (`audiomass`, `bentopdf`, `excalidraw`, `tldraw`, `wick`, `webrcade`, `drawpile`, `navidrome`, `vaultwarden`, `sopro`, `demucs`) from rootful Docker Compose to rootless Podman systemd Quadlets (`/home/docker-data/.config/containers/systemd/*.container`). - Updated `/home/luke/bin/status_all`, backup/restore scripts, [Service List](https://wiki.drheck.dev/books/system-setup/page/service-list), [Service Update Process](https://wiki.drheck.dev/books/system-setup/page/service-update-process), and authored [podman](https://wiki.drheck.dev/books/system-setup/page/podman). - 2026-10-10 Saturday - **Minecraft Server Control Plane, Live Console, Seed Studio & Text/Icon Studio (`ul`, `danknasty`)**: - Built and deployed `mc-manager` (`/usr/local/bin/mc-manager`, `mc-manager.service`), `mc-runner` (`/usr/local/bin/mc-runner`, `minecraft@.service` PTY wrapper with `/run/mc-manager/.sock`), and `mc-seed-render` (`/usr/local/bin/mc-seed-render`, native `cubiomes` biome & structure map generator) at [mc.drheck.dev](https://mc.drheck.dev) behind Nginx and Authentik ForwardAuth. - Added **Minecraft Text, MOTD, Color Gradient & 8-Bit Pixel Icon Studio** to both the **+ New World** wizard and existing server **Configuration** panel (Unicode font styles, Java-safe symbols, `§` RGB gradient interpolator, live server-list preview, and 64×64 8-bit pixel badge/icon generator). - Updated `/etc/haproxy/minecraft.lua` and HAProxy backend rules so all Multiplayer Server List pings (`state == 1`) query `mc-manager` (`BuildLiveStatusJSON`), overlaying live `motd` (`server.properties`) and `favicon` (`server-icon.png`) onto running server status responses without requiring a server restart. - Updated `/data/minecraft/ipctl` to authenticate programmatically to `danknasty` using short-lived (5-minute) SSH user certificates issued by `step-ca` (`webshell-svc` provisioner) and trusted via `/etc/sshd_config.d/10-drheck-certs.conf` on `danknasty`. - Externalized all dynamic `ipctl` state out of code-reviewed base configs into `/etc/systemd/network/20-wired.network.d/minecraft.conf`, `/etc/haproxy/conf.d/minecraft_backends.cfg`, `/etc/haproxy/minecraft_hosts.map`, and `/var/unbound/etc/unbound.minecraft.conf` (included from `unbound.conf` and parsed recursively by `update_dns.py`, `dns_danger_sync.py`, and `verify_dns.sh`), with tracked mirrors in `/data/minecraft/.network/`. - Provisioned `desertworld.drheck.dev` (`fd77:7777:5010::1337:3c:d00d:2/64`) and updated [Minecraft Worlds](https://wiki.drheck.dev/books/system-setup/page/minecraft-worlds) and [Service List](https://wiki.drheck.dev/books/system-setup/page/service-list). - 2026-09-27 Sunday - **amux (Agent-Modified tmux) & Persistent Session Infrastructure (`ul`)**: - Created custom tmux fork (`amux`) built from `/home/luke/git/tmux` and installed at `/usr/local/bin/tmux-custom`. - Implemented native `write-pane` (`writep`) command supporting `-b ` and `-f ` with `-e` (auto-enter) to inject scripts directly into PTY buffers with zero quote escaping or keystroke lag. - Built structured audit logging directly into `cmd-write-pane.c` using `sd_journal_send` (`SYSLOG_IDENTIFIER=amux`, `AMUX_PANE_ID`, `AMUX_WINDOW_NAME`, `AMUX_ARCHIVE_PATH`). - Configured automatic pre-execution script archival to persistent storage in `/data/amux/scripts/` (`YYYY-MM-DD_HHMM.SS_.sh`). - Configured dedicated `atx` agent session in `/home/luke/bin/tmux_control` on socket `/tmp/tmux-1000/tmux_luke_atx`, styled with orange status bar (`#ff8800`), `C-b` prefix, and persistent agent environment in `~/.agent_profile`. - Added backup and restore scripts for amux (`/home/luke/git/backups/ul/scripts/backup/amux` and `/restore/amux`). - Documented architecture and audit procedures on [amux - Agent modified tmux](https://wiki.drheck.dev/books/system-setup/page/amux-agent-modified-tmux) and [SSH & Agent Command Tracking Guide](https://wiki.drheck.dev/books/system-setup/page/ssh-command-tracking-guide). - **Minecraft Datapack Enhancements (`pokelovers`) (`ul`)**: - Installed 5 verified Minecraft 26.3 datapacks in `/data/minecraft/poke lovers/world/datapacks/`: - `fast_leaf_decay` (rapid leaf decay upon log breakage) - `graves` (death chest/grave saving inventory and XP) - `invisframes` (invisible item frames via shears while sneaking) - `player-drops-head` (player head drops on death) - `shulker-drops-two-shells` (guaranteed 2 shulker shells) - Broadcasted in-game countdown notices to online players prior to executing live `/reload`. - **BlueMap 3D Web Map & Authentik SSO Integration (`pokelovers`) (`ul`)**: - Deployed BlueMap 5.28 standalone CLI renderer running as `bluemap-pokelovers.service` in watch mode (`-w -u`), serving live world updates and Server-Sent Events (SSE) on port 8100. - Configured NGINX reverse proxy on port 443 with wildcard SSL (`*.drheck.dev`) in `/etc/nginx/sites-available/pokelovers`. - Gated `https://pokelovers.drheck.dev` behind Authentik ForwardAuth proxy provider and application on embedded outpost, protecting map coordinates and player builds behind authentication. - 2026-09-26 Saturday - **Zero-Config Minecraft HAProxy SNI Stream Routing (`ul`, `danknasty`)**: - Decommissioned application-level proxy (Velocity) to preserve native vanilla Diffie-Hellman encryption and Microsoft `online-mode=true` authentication. - Deployed HAProxy 3.4.5 on dual-stack endpoints (`10.0.0.113:25565` and `[fd77:7777:5010::1337:f1ac:d00d:3c]:25565`). - Implemented `/etc/haproxy/minecraft.lua` using non-draining buffer peeking (`txn.req:dup()`) to inspect the raw unencrypted Minecraft handshake packet (`0x00`), dynamically extracting the SNI virtual host (`pokelovers.drheck.dev`). - Routed traffic via `/etc/haproxy/minecraft_hosts.map` directly to dedicated internal ULA addresses on standard port 25565 (`[fd77:7777:5010::1337:3c:d00d:1]:25565`). - Implemented native fallback responder (`mc_offline`) returning custom protocol packets when backends are stopped: - Multiplayer Server List MOTD: `"[Server] is currently offline. Ping Luke"` - Multiplayer Join Screen: Immediate disconnect with red bold error: `"[Server] is currently offline. Ping Luke"` - **Network & DNS Automation (`ipctl`) (`ul`, `danknasty`)**: - Developed `/data/minecraft/ipctl` (symlinked in `/home/luke/bin/ipctl`). - Automates ULA IPv6 allocation (`fd77:7777:5010::1337:3c:d00d:/64`) on `eth77` with `systemd-networkd` reload. - Automates HAProxy backend and map configuration reload. - Integrated SSH agent validation checking for key `ultra-arikui` (`SHA256:7Xr0vCxgfpPhbKE3hN9fE2IC9y4x3Kj+kYmXwzgJY3s`). - Implemented 7-step automated DNS provisioning over SSH to `danknasty`: updates `~/git/router_config/unbound.conf`, validates via `unbound-checkconf`, deploys to `/var/unbound/etc/`, restarts `unbound`, verifies local resolution, and triggers `dns_danger_sync.py --force` to push public A and AAAA records to deSEC.io. - Created dedicated backup and restore scripts in `/home/luke/git/backups/ul/scripts/backup/minecraft_router` and `/restore/minecraft_router`. - Documented entire architecture and quickstart HOWTO on HeckWiki page 136 ([Minecraft Worlds](https://wiki.drheck.dev/books/system-setup/page/minecraft-worlds)). - **Firewall Configuration for Minecraft Port 25565 (`danknasty`)**: - Configured `mc_ipv4` (`10.0.0.113`), `mc_gu` (`$gu_prefix:1337:f1ac:d00d:3c`), and `mc_ula` (`$ula_prefix:1337:f1ac:d00d:3c`) in `~/git/router_config/pf.conf`. - Added inbound port 25565 forwarding rules for IPv4 NAT and IPv6 ULA redirection. - Validated with `pfctl -nf`, deployed to `/etc/pf.conf`, loaded into the kernel, and verified reachability externally from `rng`. - 2026-09-21 Monday - **Native Sunshine & Gamescope Migration on ul**: - Decommissioned legacy `steam-headless` container; migrated to bare-metal Gamescope DRM compositor + Sunshine host on AMD Radeon AI PRO R9700 (`1002:7551`, Navi 48). - Injected AOC 1080p EDID on `DP-1` via mkinitcpio & systemd-boot loader options. - Configured Steam library paths directly to 151TB ZFS pool (`/data/games/steam/steamapps`). - Resolved PS5 DualSense controller disconnection: disabled Steam proprietary HIDAPI polling (`SDL_JOYSTICK_HIDAPI_PS5=0`, `SDL_JOYSTICK_HIDAPI=0`) to prevent `hid_read failure` timeout on virtual `uhid` devices. - Added `-e` (`--steam`) integration flag to Gamescope launch script to ensure proper Wayland focus relaying to Steam, enabling Steam Input in-game instead of dropping to Desktop mouse layout. - Documented complete architecture, launch scripts, and controller gotchas on HeckWiki page 130 ([Sunshine Remote Setup](https://wiki.drheck.dev/books/systems/page/sunshine-remote-setup)). - 2026-09-21 Monday - **Steam Storage Migration to 151TB ZFS Pool (`/data/games/steam`) (`ul`)**: - Discovered root NVMe filesystem at 85% utilization (71GB free). - Consolidated all Steam games, runtimes, manifests, and shader caches from temporary docker volumes to `/data/games/steam/steamapps/` on the 151TB ZFS pool. - Replaced `~/.local/share/Steam/steamapps` and `~/.steam/steam/steamapps` with permanent atomic symlinks pointing to `/data/games/steam/steamapps/`. - Cleaned `libraryfolders.vdf` ensuring 100% of game downloads route directly to `/data/games/steam/` with zero bytes hitting the root SSD. - **TTY1 Getty Deactivation & Display Console Fix (`ul`)**: - Disabled `getty@tty1.service` (`systemctl disable --now getty@tty1.service`) to prevent Linux console login prompts from broadcasting over KMS on `DP-1`. - 2026-09-20 Sunday - **PS5 DualSense Controller Hardware Support & uhid Activation (`ul`)**: - Identified PlayStation 5 DualSense controller emulation failure in Sunshine due to missing `/dev/uhid` (User HID) device node. - Loaded `uhid` kernel module and made persistent across reboots in `/etc/modules-load.d/uhid.conf`. - Configured udev rule in `/etc/udev/rules.d/85-sunshine-uinput.rules` granting `input` group permissions to both `/dev/uinput` and `/dev/uhid` (`MODE="0660", GROUP="input"`). - **Native Gamescope + Headless DRM Stack Migration (`ul`)**: - Decommissioned legacy `steam_headless` Docker container to eliminate Xorg CPU dummy-driver bottleneck (`xf86-video-dummy`) and software rasterization (15 FPS). - Installed native Arch Linux gaming stack: `gamescope`, `steam`, `vulkan-radeon`, `lib32-vulkan-radeon`, `pipewire-pulse`, `wireplumber`, `seatd`, `socat`, and `sunshine-bin` (AUR). - Granted real-time scheduling capabilities to Gamescope (`setcap cap_sys_nice=eip /usr/bin/gamescope`) and rootless DRM frame capture to Sunshine (`setcap cap_sys_admin,cap_sys_nice=eip /usr/bin/sunshine`). - Configured `seatd` daemon with group `video` override in `/etc/systemd/system/seatd.service.d/group.conf` to allow Gamescope DRM session management without an active logind seat. - Configured Gamescope launcher script `/home/luke/bin/launch-gamescope-steam.sh` driving connector `DP-1` via `--backend drm` with RADV Vulkan acceleration on the AMD Radeon AI PRO R9700. - **Headless Virtual Display via Boot-Time EDID Injection (`ul`)**: - Resolved `*ERROR* No EDID found on connector: DP-1` kernel spam on headless AMD GPU. - Sourced verified 1080p 144Hz CEA-861 DisplayPort EDID (`aoc_1080p.bin`) in `/usr/lib/firmware/edid/`. - Packaged EDID into early initramfs via `FILES=(/usr/lib/firmware/edid/aoc_1080p.bin)` in `/etc/mkinitcpio.conf` and regenerated boot image via `mkinitcpio -P`. - Configured bootloader kernel parameters in `/boot/loader/entries/arch.conf`: `drm.edid_firmware=DP-1:edid/aoc_1080p.bin video=DP-1:1920x1080@60e`. - Pinned Sunshine KMS capture explicitly to `adapter_name = /dev/dri/card1` and `output_name = DP-1`, preventing BMC graphics capture (`VGA-1`). - **Dual-Stack IPv4/IPv6 `socat` Bridge (`ul`)**: - Bound native Sunshine directly to dedicated IPv6 ULA address `[fd77:7777:5010:0:1337:f1ac:d00d:4799]`. - Deployed `/usr/local/bin/sunshine-ipv4-bridge.sh` and systemd unit `/etc/systemd/system/sunshine-ipv4-bridge.service` listening on `10.0.0.112` and forwarding all GameStream TCP and UDP ports bidirectionally to IPv6 to prevent asymmetric routing drops on dual-stack LANs. - **Client Authentication & Stale Certificate Remediation (`ul`)**: - Diagnosed `Warning: SSL Verification error :: Client certificate identity is not enabled` (LizardByte issue #5696) caused by duplicate client certificates with conflicting UUIDs in `sunshine_state.json`. - Cleared stale device records in `sunshine_state.json` allowing clean one-time client pairing across desktop and TV clients. - **NetworkManager D-Bus Mock for Steam (`ul`)**: - Diagnosed Steam Gamepad UI offline status (`An internet connection is required to load this content` / slashed Wi-Fi icon) caused by absence of NetworkManager on `systemd-networkd` host. - Implemented lightweight Python D-Bus daemon `/usr/local/bin/fake-networkmanager.py` and service `/etc/systemd/system/fake-networkmanager.service` reporting `NM_STATE_CONNECTED_GLOBAL` (70) and `NM_CONNECTIVITY_FULL` (4) to satisfy Steam network checks. - **Documentation**: - Authored and published comprehensive architecture and setup runbook on HeckWiki: [sunshine remote setup](https://wiki.drheck.dev/books/system-setup/page/sunshine-remote-setup) (Page 130). - 2026-09-19 Saturday - **Sunshine Network Tuning & Steam Container Bootstrap (`ul`)**: - Diagnosed container web assets and API communication issues on `josh5/steam-headless`. - Configured Sunshine CORS origins (`csrf_allowed_origins = https://sunshine.drheck.dev`) and dual-stack network settings in `sunshine.conf`. - Bootstrapped Steam runtime inside container and configured Steam Big Picture startup commands. - Authored and published initial HeckWiki guide: [Steam Streaming](https://wiki.drheck.dev/books/computer/page/steam-streaming) (Page 129). - Updated LDAP group enrollment documentation (`ldap-sssd-client-enrollment-guide`, Page 25) with corrected repository clone instructions. - 2026-09-18 Friday - **Immich Machine Learning AMD ROCm Migration (`ul`)**: - Migrated `immich_machine_learning` from NVIDIA CUDA to native AMD ROCm: pulled `ghcr.io/immich-app/immich-machine-learning:v3.1.0-rocm`. - Updated `/data/docker/immich/.env` (`IMMICH_MACHINE_LEARNING_GPU_TYPE=rocm`). - Recreated Immich stack and verified PyTorch ROCm device detection (`torch.cuda.is_available() = True` on AMD Radeon AI PRO R9700 `gfx1201`). - Verified VA-API hardware video acceleration on `/dev/dri/renderD128` inside `immich_server`. - **Sunshine & Steam-Headless Initial Deployment (`ul` & `danknasty`)**: - Provisioned dedicated internal IPv4 (`10.0.0.112`) and IPv6 ULA (`fd77:7777:5010:0:1337:f1ac:d00d:4799`). - Updated `unbound.conf` in `~/git/router_config` on `danknasty` with `sunshine.drheck.dev` A and AAAA records; deployed via `./sync_unbound`. - Synced public DNS records via `dns_danger_sync.py` on deSEC.io. - Configured `/etc/nginx/sites-available/sunshine` with Authentik ForwardAuth and deployed reverse proxy. - Provisioned `sunshine` application and ProxyProvider in Authentik embedded outpost. - Deployed `josh5/steam-headless` container in `/data/docker/sunshine/docker-compose.yml`. - 2026-09-17 Thursday - **Tmux & SSH Auth Environment Maintenance (`ul`)**: - Re-initialized Tmux Plugin Manager (`tpm`) and checked plugin compatibility (`check_tmux_version.sh`). - Maintained persistent SSH agent socket symlinks. - 2026-09-15 Tuesday - **System Package Maintenance & Initramfs Rebuild (`ul`)**: - Upgraded system packages via `paru -Syu`. - Rebuilt `initramfs-linux-lts.img` via ALPM hooks. - Verified systemd service health. - 2026-09-12 Saturday - **System Package Maintenance & LTS Kernel Update (`ul`)**: - Upgraded system packages via `paru -Syu` / `pacman -Syu`. - Kernel update to `6.18.52-1-lts` and regenerated `initramfs-linux-lts.img`. - Re-verified NVIDIA kernel module conftest compilation checks for `580.178.04`. - Media library maintenance: sorted and updated Adventure Time subtitle filenames. - 2026-09-08 Tuesday - 2026-09-11 Friday - **System Maintenance & Driver Toolchain Compilation (`ul`)**: - Executed routine system maintenance and package updates across `ul`. - Kernel module conftest compilation and pacman keyring verification. - 2026-09-05 Saturday - **OpenBSD Router Reinstallation & CrowdSec Perimeter Restoration (`danknasty`)**: - Reinstalled and configured `crowdsec-firewall-bouncer` connecting to `ul`'s LAPI over IPv6 ULA (`http://[fd77:7777:5010:0:1337:f1ac:d00d:c70d]:8080/`). - Restored OpenBSD PF firewall tables (``, ``, ``) with rate limiting (`max-src-conn-rate 3/30`), loading ~23,880 blacklisted attacker IPs directly into kernel tables. - Verified remote syslog forwarding of `auth,authpriv.info` over IPv6 UDP to `ul`'s CrowdSec syslog parser. - **Router Network Mapping & Dynamic IPv6 DNS Cron Jobs (`danknasty`)**: - Aligned network interface roles after hardware reinstallation (`em0` WAN, `em1` LAN with delegated `/64` prefix). - Updated interface configurations across `/etc/hostname.em0`, `/etc/hostname.em1`, `/etc/dhcp6leased.conf`, `/etc/rad.conf`, and `/etc/dhcpd.conf`. - Deployed dynamic deSEC.io IPv6 DNS updater (`/etc/update_dns.py`) and keepalive monitor (`/etc/ipv6_keepalive.sh`). - Restored root crontab on `danknasty` to run `/etc/update_dns.py` every minute and `/etc/ipv6_keepalive.sh` every 5 minutes. - Triggered manual sync: verified all 43 domain AAAA records updated on deSEC and verified public IPv6 reachability via `rng` to `https://auth.drheck.dev/`. - **Router Disaster Recovery & Reinstall Survival Automation (`danknasty`)**: - Created single-command idempotent provisioning script `~/git/router_config/bootstrap.sh` (`doas ./bootstrap.sh`): installs package dependencies, copies system configs, sets file permissions and ownership, restores root crontab, applies sysctl, and starts all services. - Created zero-touch OpenBSD post-install hook `~/git/router_config/install.site` to automatically pull repository and execute `bootstrap.sh` inside `chroot /mnt` at OS installation completion. - Added `/etc/sysctl.conf` (`net.inet.ip.forwarding=1`, `net.inet6.ip6.forwarding=1`) to repo with automated activation via `sysctl -f`. - Added package manifests to repo: `pkglist` (generated via `pkg_info -mz` for native OpenBSD `pkg_add -I -l` restoration) and `packages.txt` (human-readable catalog with descriptions). - Added `/etc/dhcpd.conf` (DHCPv4 server) and `/var/unbound/etc/unbound.conf` (DNS resolver) deployment and service management into `bootstrap.sh`. - Backed up all router configuration templates into `~/git/router_config` (`boot.conf`, `ttys`, `doas.conf`, `crontab.root`, `dhcp6leased.conf`, `rad.conf`, `pf.conf`, `syslog.conf`, `rc.conf.local`, `mygate`, `crowdsec-firewall-bouncer.yaml`). - **Nextcloud Landing Page & occ Runbook Guide (`ul`)**: - Configured post-authentication redirect on `cloud.drheck.dev` to default directly to `/apps/files/files` (`occ config:system:set defaultapp files`, `occ config:app:set core defaultpage /apps/files/files`). - Authored and published comprehensive [Nextcloud occ Guide](https://wiki.drheck.dev/books/system-setup/page/nextcloud-occ) on HeckWiki documenting command-line administration, alias setups, user management, and maintenance mode operations. - 2026-09-04 Friday - **Danknasty Router Hardware & Platform Diagnostics (`danknasty`)**: - Identified router hardware platform: Broachlink NOAH4C V10 (Intel Atom E3845 quad-core @ 1.91GHz, 4GB RAM, 4x Intel I210 Gigabit Ethernet ports `em0`-`em3`). - Enabled RJ45 serial console access on `com0` at 115200 baud via `/etc/boot.conf` (`stty com0 115200`, `set tty com0`) and `/etc/ttys` (`tty00 std.115200 vt220 on secure`). - Diagnosed periodic SSH responsiveness latency: traced kernel system CPU lockups and interrupt activity using `vmstat -i`, `sensorsd`, and `bsd.re-config`. - **Split-Horizon Unbound Authoritative LAN Resolution (`danknasty`)**: - Analyzed Unbound configuration on `danknasty` acting as authoritative LAN resolver (`local-zone` / `local-data` with `aa` flag set in LAN responses). - Evaluated DNS high-availability options: Git-based Unbound configuration replication vs authoritative secondary migration to NSD. - 2026-09-03 Thursday - **SSH CA Documentation Modularization (HeckWiki)**: - Cleanly partitioned SSH Certificate Authority documentation into two focused reference runbooks under the `LDAP+PAM+SSH CA` book: 1. [SSH CA setup](https://wiki.drheck.dev/books/ldappamssh-ca/page/ssh-ca-setup) (Page 26): Core architecture, Ed25519 CA keys, Vaultwarden storage, signing principles, and Certificate Revocation Lists (KRL). 2. [SSH Client and Host Onboarding Guide](https://wiki.drheck.dev/books/ldappamssh-ca/page/ssh-client-and-host-onboarding-guide) (Page 112): Step-by-step out-of-band bootstrap and verification workflows for onboarding new clients and servers without circular SSH dependencies. - 2026-09-02 Wednesday - **OpenBSD Router IPv6 Dynamic DNS Rewrite (`update_dns.py`) (`danknasty`)**: - Diagnosed HTTP 400 bulk update failures on deSEC.io caused by regex string parsing errors and duplicate records in `unbound.conf`. - Replaced legacy bash/awk script (`update_dns.sh`) with robust Python 3 implementation (`update_dns.py`) using Python's native `ipaddress` library and typed dictionary deduplication. - Added unit test suite `test_update_dns.py` with 9 unit tests verifying zone record parsing, address synthesis, and deduplication. - Removed duplicate `libresprite.drheck.dev` entry and cleaned up interleaved records in `unbound.conf`. - Deployed `/etc/update_dns.py` (0755 root:wheel), removed `/etc/update_dns.sh`, and signed/committed all changes to `~/git/router_config`. - **CrowdSec Log Synchronization & False-Positive Ban Remediation (`ul` & `danknasty`)**: - Investigated repetitive `/var/log/authlog` session spam on `danknasty`. - Identified redundant SSH polling in `crowdsec-sync-authlog` on `ul` running via `crowdsec-export-bans.timer` every 60s. - Disabled unnecessary SSH scraping, relying on native real-time UDP IPv6 syslog forwarding (`auth,authpriv.info`) from `danknasty` to CrowdSec on `ul`. - Resolved false-positive ban on VPS `rng` (`66.186.37.205`) in `` PF table caused by unprivileged TCP connect scans. - **SSH CA Inventory Tracking & Streamlined Key Signing (`ul`)**: - Implemented key archival and inventory management in `/usr/local/bin/sign-host-key` and `/usr/local/bin/sign-user-key` archiving signed public keys in `/etc/ssh/ca_inventory/{hosts,users}` with mandatory labeling for revocation tracking. - Built revocation utility `/usr/local/bin/revoke-ssh-key`. - Documented out-of-band onboarding workflow on HeckWiki. - 2026-08-31 Monday - **System Package Maintenance & Toolchain Updates (`ul`)**: - Executed system upgrades via `paru` / `pacman`. - Updated Haskell libraries (`haskell-uuid-types`, `haskell-vector`, `haskell-witherable`), `shellcheck`, and system info documentation. - Updated Opencode CLI, `csvtools`, and `nethack-git`. - 2026-08-30 Sunday - **Demucs Stem Separation Deployment (`https://demucs.drheck.dev/`)**: - Added `demucs.drheck.dev` DNS records to `unbound.conf` on `danknasty` (`~/git/router_config`). - Built and deployed container running Demucs v4 web service (`app.py`, HTML5 Multi-Track Mixer) on dedicated ULA IPv6 (`fd77:7777:5010:0:1337:f1ac:d00d:de3:8000`). - Supports `htdemucs_ft` (Fine-tuned 4-Stem), `htdemucs` (4-Stem), `htdemucs_6s` (6-Stem with Piano & Guitar), and `mdx_extra`. - Features synchronized multi-track audio playback with per-stem Solo, Mute, Volume faders, and ZIP export. - Provisioned Authentik ForwardAuth `demucs` ProxyProvider and Application in Authentik Embedded Outpost. - Configured `/etc/nginx/sites-available/demucs` reverse proxy with 512MB max body size and 600s read timeout. - Added firewall isolation reject rule in `/etc/nftables.conf`. - Created `demucs` backup and restore scripts with `par2` parity in `~/git/backups`. - 2026-08-29 Saturday - **Sopro V2 TTS Deployment (`https://sopro.drheck.dev/`)**: - Enhanced web interface and backend API to expose configurable acoustic solver `Steps` (1-64 steps). - Added `sopro.drheck.dev` DNS records to `unbound.conf` on `danknasty` (`~/git/router_config`). - Built and deployed container running `soprotts serve --device cpu --host 0.0.0.0 --port 7860 --no-open` on dedicated ULA IPv6 (`fd77:7777:5010:0:1337:f1ac:d00d:509:7860`). - Provisioned Authentik ForwardAuth `sopro` ProxyProvider and Application in Authentik Embedded Outpost. - Configured `/etc/nginx/sites-available/sopro` reverse proxy with WebSocket support and pushed to `~/git/nginx`. - Added firewall isolation reject rule in `/etc/nftables.conf` and pushed to `~/git/configs`. - Created `sopro` backup and restore scripts with `par2` parity in `~/git/backups`. - **BentoPDF Deployment (`https://bentopdf.drheck.dev/`)**: - Added `bentopdf.drheck.dev` DNS records to `unbound.conf` on `danknasty` (`~/git/router_config`). - Deployed `ghcr.io/alam00000/bentopdf:latest` container on dedicated ULA IPv6 (`fd77:7777:5010:0:1337:f1ac:d00d:bdf:8080`). - Provisioned Authentik ForwardAuth `bentopdf` ProxyProvider & Application in Authentik Embedded Outpost. - Configured `/etc/nginx/sites-available/bentopdf` reverse proxy and pushed to `~/git/nginx`. - Added firewall isolation reject rule in `/etc/nftables.conf` and pushed to `~/git/configs`. - Created `bentopdf` backup and restore scripts with `par2` parity in `~/git/backups`. - **SSL Certificate Automation & SSSD TLS**: - Fixed expired LDAP `CertificateKeyPair` inside Authentik database via `ak shell` and restarted `authentik-ldap`. - Deployed `/usr/local/bin/update-authentik-cert.sh` automated renewal hook in `acme.sh` configuration with `/etc/sudoers.d/acme_ssl` NOPASSWD grant. - **Docker `.env` & Secret File Hardening**: - Stripped POSIX extended ACLs (`setfacl -b`) across all `.env` files and SSH keys in `/data/docker/`. - Restricted permissions to `0640 root:docker-data` and parent directories to `0750`. - **Universal Admin Account Transition**: - Transferred administrative roles to `akadmin` / `admin` across Authentik, Gitea, Immich, Anchor, BookStack, Nextcloud, Kavita, Navidrome, and RomM. - Demoted `luke` to standard user on web applications while preserving `wheel` (sudo) access on host OS. - **PostgreSQL Authentication Hardening**: - Removed obsolete `0.0.0.0/0` and `::/0` wildcard rules for non-existent `infisical` role from `/data/postgres/data/pg_hba.conf`. - **`nftables` Egress Isolation (`chain output`)**: - Added reject rules (`meta skuid != { 0, 33, 1101 } reject`) in `chain output` for all 26 active ULA IPv6 backend services to prevent unprivileged local processes on `ul` from bypassing Authentik ForwardAuth. - Tracked `/etc/nftables.conf` in `~/git/configs/nftables.conf`. - **Nginx Hash Table Performance Tuning**: - Added `variables_hash_max_size 2048;` and `variables_hash_bucket_size 128;` to `/etc/nginx/nginx.conf` and pushed to `~/git/nginx`. - **Authentik 2026.8.0 Outpost & OIDC Remediation**: - Populated missing `authentication_flow` (`default-authentication-flow`) and `invalidation_flow` (`default-invalidation-flow`) UUIDs across all 19 `ProxyProvider` database records, fixing Rust Embedded Outpost deserialization crash and restoring ForwardAuth across all services. - Updated OpenID `email` ScopeMapping expression to enforce `email_verified = True` for Better-Auth (WIPStudio) / SSO client logins. - **Cockpit Removal**: - Uninstalled `cockpit` and `cockpit-docker` packages; purged `/etc/motd.d/cockpit`, `/etc/issue.d/cockpit.issue`, and `/run/cockpit`. - **Arch `.pacnew` Merge & Maintenance**: - Adopted updated upstream makepkg drop-ins and TPM2 FAPI JSON profiles. - Tuned `/etc/pacman.conf` with `DownloadUser = alpm` and `ParallelDownloads = 5`. - Cleaned all lingering `.pacnew` files in `/etc/`. - **BMO95 Display Mirroring (`~/git/audio_config`)**: - Installed `wl-mirror` on `bmo95` and configured Sway (`common/sway/config`) to mirror DisplayPort `DP-3` to HDMI `HDMI-A-1` (TV) via zero-copy DMA-BUF. - Consumed Sway configuration into `~/git/audio_config` and committed changes. - **Studio Control (`scontrol`) UI & Toggle Controls**: - Added `POST /api/power/{device}/{action}` supporting individual Sub-GHz RF outlet toggling (`pa`, `sub`, `rack`, `computer`, `tv`, `neewer`, `mrc`, `all`) and `POST /api/atx/{action}` for PiKVM motherboard ATX button clicks. - Added **Individual Power Controls** view with dedicated toggle buttons and view-switcher button to `scontrol.drheck.dev`. - Checked in and pushed to `https://git.drheck.dev/luke/scontrol.git`. - **Kavita Arch Linux / AUR Upgrade & Database Migrations (`ul`)**: - Checked SQLite database integrity on `/opt/kavita/config/kavita.db` (`PRAGMA integrity_check`). - Created safety database backup at `/opt/kavita/config/kavita.db.pre_0.9.1_upgrade`. - Restarted `kavita.service` following `paru` upgrade to Kavita `v0.9.1.0`. - Executed automatic backup (`/opt/kavita/config/temp/migration/0.9.0.2`) and applied database migrations (`MetadataSettingFieldRenumber`, `ManualMigrateOriginalNameBackfill`, `ManualMigrateNormalizedOriginalNameBackfill`, `ManualMigrateLastReadDates`, `ManualMigrationExternalSeriesMetadataIds`, `ManualMigrationCorrectAuditStatus`). - Verified service initialization and HTTP 200 on `http://[fd77:7777:5010:0:1337:f1ac:d00d:7ead]:5000/`. - **Full Docker Fleet Update & Service Migrations (`ul`)**: - **Authentik**: Created PostgreSQL database backup at `/tmp/authentik_pre_upgrade.dump`. Upgraded `server`, `worker`, and `ldap` to `2026.8.0`. Applied blueprint and Django migrations across all applications. Verified `email_verified: True` in `ak shell` scope mapping and tested `https://auth.drheck.dev/`. - **Infra (Valkey)**: Upgraded shared Redis/cache container to `valkey/valkey:8.1-alpine` (v8.1.9). Verified active session reconnection for Authentik, Immich, and Nextcloud. - **Immich**: Upgraded `immich-server` and `immich-machine-learning` to latest `v3.1.0` builds with GPU acceleration. Verified `https://photos.drheck.dev/`. - **Kasm Workspaces**: Upgraded manager container to `lscr.io/linuxserver/kasm:latest` (Kasm 1.19.0). Initialized internal services and verified `https://kasm.drheck.dev/`. - **BookStack**: Upgraded `bookstack` (v26.05.4) and internal `mariadb:latest`. Verified Laravel migrations and `https://wiki.drheck.dev/`. - **Penpot**: Upgraded `frontend`, `backend`, `exporter` (v2.17.2), `postgres:15.19`, and `redis:7`. Verified `https://penpot.drheck.dev/`. - **Anchor**: Upgraded image tag to `ghcr.io/zhfahim/anchor:0.16.0`. Executed database schema migrations and verified `https://keep.drheck.dev/`. - **Vaultwarden**: Upgraded to `vaultwarden/server:latest` (v1.37.2). Verified `https://pw.drheck.dev/`. - **Navidrome**: Upgraded to `deluan/navidrome:latest` (v0.63.2). Verified web and Subsonic API endpoints on `https://navidrome.drheck.dev/` and `https://music.drheck.dev/`. - **WebShell / Step-CA**: Upgraded `smallstep/step-ca:latest` (v0.30.2). Verified HTTPS on port 9000. - **WIPStudio / MinIO**: Upgraded storage backend to `minio/minio:latest`. - **YOURLS**: Upgraded `yourls:latest` (1.10.5) and internal `mariadb:10.11`. - **Drawpile**: Upgraded server to `drawpile/drawpile-srv:2` (2.3.0). Verified TCP/WebSocket listeners on ports 27750-27752. - **Stateless Web Services**: Upgraded `wick` (`nginx:alpine`), `webrcade` (`webrcade:latest`), `excalidraw` (`excalidraw:latest`), and `tldraw` (`ratneo/tldraw:latest`). - **Offsite Cloud Backups**: Executed full service backups and synced to Google Drive and Proton Drive via rclone (`2026-08-29_112824`). - 2026-08-28 Friday - **Funkwhale Audio Streaming & Range Request Fix (`ul`)**: - Diagnosed audio playback failures and stream disconnects on Funkwhale API endpoints (`/api/v1/listen/.../?upload=...` and transcoded audio streams). - Resolved Nginx reverse proxy byte-range streaming failure (`206 Partial Content`) by updating `/data/docker/funkwhale/funkwhale_nginx.template` and volume mounting custom `/data/docker/funkwhale/mime.types` inside the container. - Recreated container and verified byte-range audio playback via `curl -i -r 0-65535` returning HTTP `206 Partial Content`. - Updated Funkwhale backup and restore scripts (`/home/luke/git/backups/ul/scripts/backup/funkwhale` and `restore/funkwhale`) to include `mime.types` and `funkwhale_nginx.template` in backup tarballs (verified with `shellcheck`). - 2026-08-27 Thursday - **System Package Maintenance & Kernel Audit (`ul`)**: - Executed system package upgrades via `paru -Syu` / `pacman -Syu`. - Updated mirrorlist and restarted `reflector.service`. - Verified kernel image and module consistency using `kernel-check` against `/boot/vmlinuz-linux-lts`. - 2026-08-26 Wednesday - **Studio Control Gateway & ATX Power Automation (`ul` & `pikvm-studio`)**: - Integrated ATX motherboard power button controls (`kvmd-cli atx`) into `studio-power` CLI and synchronized executable between `ul` and `pikvm-studio`. - Resolved container DNS and IPv6 resolution for `bmo95.drheck.dev` inside `scontrol` Docker container on `infra_net` bridge. - Updated and redeployed `app.py` in `/data/docker/scontrol` and committed changes to `/home/luke/git/scontrol`. - 2026-08-24 Monday - **Studio PiKVM Hardware & Perimeter Ingress Security (`ul` & `pikvm-studio`)**: - Configured dual-hostname architecture: `studio-kvm-ssh.drheck.dev` (direct hardware & SSH) and `studio-kvm-web.drheck.dev` (Authentik-gated Web UI). - Implemented kernel-level outbound firewall isolation in `/etc/nftables.conf` (`chain output`) on `ul` blocking non-root, non-nginx (`http`, UID 33) users from connecting to PiKVM ports 80/443. - Configured Pre-Shared Proxy Secret Header (`X-Proxy-Secret`) in `/etc/nginx/certs/studio_secret.conf` (mode `0600`, owned by `http:http`). - Configured Mutual TLS (mTLS) with internal CA and client certificate authentication (`/etc/nginx/certs/pikvm_ca.crt`, `pikvm_client.crt`, `pikvm_client.key`). - Updated Authentik `ProxyProvider` and `Application` (`Studio PiKVM`) with implicit consent (`default-provider-authorization-implicit-consent`) and group `studio` access restriction. - Set `kvmd.auth.enabled: false` in `/etc/kvmd/override.yaml` (committed in `/home/luke/git/pikvm_studio`). - Updated [Studio PiKVM & Studio Control Gateway](https://wiki.drheck.dev/books/system-setup/page/studio-pikvm) documentation. - **Studio Control Gateway Deployment (`scontrol.drheck.dev`) (`ul`)**: - Developed and deployed asynchronous FastAPI orchestration service (`scontrol`) in `/data/docker/scontrol` binding to dedicated ULA IPv6 address `[fd77:7777:5010:0:1337:f1ac:d00d:5c01]:8000`. - Implemented tactile mobile-optimized dark UI with dynamic state buttons, live monospace console, and status indicators. - Generated dedicated service ed25519 key (`/data/docker/scontrol/id_scontrol`) registered in `luke@studio-kvm-ssh.drheck.dev:~/.ssh/authorized_keys`. - Integrated real-time Server-Sent Events (SSE) streaming pipeline (`/api/stream/on` and `/api/stream/off`): 1. RF Power Outlets: switches all 7 studio devices via `studio-power all on` over SSH. 2. ATX Motherboard: queries PiKVM `/api/atx` and pulses motherboard power button if off. 3. Studio PC Boot: probes DNS hostname `bmo95.drheck.dev` with a 100-second boot timeout. 4. Soundcards & Audio: audits ALSA/JACK multi-channel recording interfaces. - Created Nginx ForwardAuth reverse proxy configuration `/etc/nginx/sites-available/scontrol` (symlinked to `/etc/nginx/sites-enabled/scontrol`) with SSE buffering bypass. - Configured Authentik `ProxyProvider` and `Application` (`Studio Control`) bound to Authentik Group **`studio`**. - Created Git repository at `/home/luke/git/scontrol` tracking `build.sh` (verified with `shellcheck`), `Dockerfile`, `entrypoint.sh`, `app.py`, `docker-compose.yml`, `templates/index.html`, and `README.md`. - Created, tested, and verified backup/restore scripts (`scontrol_backup` and `scontrol_restore` in `/home/luke/git/backups/ul/scripts/`) with `par2` 10% redundancy. - **Studio PC (`bmo95`) Boot & Audio Stack Audit**: - Resolved `libkmod: ERROR: conf_files_filter_out: Cannot stat directory entry: /etc/modprobe.d/no_intel_sound.conf` by replacing early-boot symlink with a direct regular file (`blacklist snd_hda_intel`). - Verified 4x Echo AudioFire 12 FireWire interfaces (`hw:AudioFire12,0` .. `hw:AudioFire12_3,0`), Emagic Unitor8 USB MIDI interface, `jackd`, `zita-a2j`/`zita-j2a` bridges, and autostarting `ardour.service` DAW instance. - **SSH Command Tracking Enhancement (`ul`)**: - Upgraded `/usr/local/bin/audit-log` helper script to automatically decode hexadecimal kernel byte strings into plaintext and sort rotated log files (`audit.log.*`) in chronological order. - Updated [SSH Command Tracking Guide](https://wiki.drheck.dev/books/system-setup/page/ssh-command-tracking-guide) documentation. - 2026-08-22 Saturday - **Sub-GHz RF Studio Power Automation (`pikvm-studio`)**: - Integrated Great Scott Gadgets YARD Stick One (`1d50:605b`, CC1111) on `pikvm-studio` to switch 433.92 MHz RF wireless power relays. - Reverse-engineered 33-bit Princeton / Fosmon PWM baseband protocol ($T_e = 291.27\text{ }\mu\text{s}$, 3433.22 baud). - Identified and resolved critical CC1111 24.0 MHz reference crystal mismatch (recalculated `FREQ = 0x121473` from 26.0 MHz default `0x10B071`, resolving a 33 MHz carrier transmission error). - Resolved double sync gap bug and configured 30-frame contiguous burst streaming (630 bytes, 1.468 s) to satisfy superheterodyne receiver AGC settling. - Installed `/usr/local/bin/studio-power` CLI tool managing all 7 studio devices and whole-studio batch switching. - Built standalone C direct transceiver utilities (`yard_tx.c`, `flipper_replay.c`, `cc1111_rfcat.h`) with `libusb-1.0`. - Created, configured, and pushed Gitea repository at [studio_yardstick](https://git.drheck.dev/luke/studio_yardstick). - [Documentation](https://wiki.drheck.dev/books/setup-detail/page/studio-power-automation) - **PiKVM GPG / Pass Pinentry TUI Fix (`pikvm-studio`)**: - Resolved `gpg: public key decryption failed: Screen or window too small` when executing `gpush pass` in subshell pipes. - Configured `/home/luke/.gnupg/gpg-agent.conf` with `pinentry-program /usr/bin/pinentry-tty` and `allow-loopback-pinentry`. - Configured `export GPG_TTY="$(tty)"` and `gpg-connect-agent updatestartuptty /bye` in `~/.bashrc` on `pikvm-studio`. - **48-Channel FireWire Synchronous Audio Stack (`bmo95`)**: - Consolidated 4x Echo AudioFire12 interfaces into a sample-accurate 48-channel synchronous recording engine on Linux 6.13 PREEMPT_RT kernel. - Established hardware color mapping (`Red` Master `fc79`, `Blue` Slave 1 `9815`, `Green` Slave 2 `8525`, `Yellow` Slave 3 `94e8`) with 75Ω BNC Word Clock daisy-chain wiring. - Configured master `jackd` bound to Red with synchronous `zita-a2j` / `zita-j2a` duplex ALSA bridges (locked at $\rho = 1.0000000000$). - Conducted 10-minute continuous 1 kHz stress test proving absolute phase synchronization (0.000 ms / 0 samples drift across 28.8M samples). - Deployed systemd user services (`jack.service`, `zita-*-{blue,green,yellow}.service`) and canonical port alias mapping (`Red_01..12`, `Blue_01..12`, `Green_01..12`, `Yellow_01..12`). - [Documentation](https://wiki.drheck.dev/books/setup-detail/page/48-channel-firewire-audio-stack) - **Studio Hardware Documentation ([Setup Detail](https://wiki.drheck.dev/books/setup-detail)) (`ul`)**: - Published comprehensive hardware register maps, wiring diagrams, and automation runbooks for the recording studio, audio stack, and RF power systems under the new [Setup Detail](https://wiki.drheck.dev/books/setup-detail) book. - **BookStack KaTeX TeX Math & Typography Integration (`ul`)**: - Globally enabled **KaTeX 0.16.11** client-side LaTeX math rendering across BookStack via `app-custom-head`. - Implemented a CSP `strict-dynamic` compliant nonced JavaScript loader to dynamically load KaTeX assets without violating browser security policies. - Published the [Meta: Wiki Capabilities, TeX Math & Typography Reference](https://wiki.drheck.dev/books/wiki/page/meta-wiki-capabilities-tex-math-typography-reference) guide documenting TeX math syntax, Japanese Furigana (``), callout alert banners, and collapsible accordions. - Updated the [Agent Guide: Updating BookStack Wiki Pages](https://wiki.drheck.dev/books/system-setup/page/agents-guide-to-wiki) with typography standards, CLI helper workflows, and operational gotchas. - Verified automated backup and restore scripts (`/home/luke/git/backups/ul/scripts/{backup,restore}/bookstack`) properly archive and restore database settings and media with `par2` parity. - 2026-08-20 Thursday - pikvm-studio (PiKVM V4 Plus) Configuration & Dotfile Management: - 1080p60 Video Pass-Through & Capture Optimization: - Imported and applied the `v4plus.no-1920x1200` EDID preset via `kvmd-edidconf` to lock capture to uncropped 1920x1080 @ 60Hz. - Enabled `gpu_freq=700` in `/boot/config.txt` for H.264 60fps hardware encoding boost. - Configured `h264_boost: true` and `desired_fps: 60` in `/etc/kvmd/override.yaml`. - Chezmoi Configuration Repository Setup: - Installed and configured `chezmoi` on `pikvm-studio` (`~/.config/chezmoi/chezmoi.yaml`) with `sourceDir: ~/git/pikvm_studio` and `destDir: /`. - Established clean 2-stage signed Git history: baseline vanilla OS configuration followed by 1080p60 optimizations. - Connected and pushed repository to Gitea (`https://git.drheck.dev/luke/pikvm_studio.git`) via `gpush`. - SSH Auth Socket & GPG Agent Fixes: - Resolved SSH key retention issue in `/home/luke/bin/auth_socket.sh` by prioritizing existing live agent sockets and testing responsiveness with `ssh-add -l` rather than relying on obsolete PID suffix parsing. - Cleaned up stale lockfiles in `~/.gnupg/` on `pikvm-studio` and configured `pinentry-program /usr/bin/pinentry-curses` in `~/.gnupg/gpg-agent.conf`. - Successfully committed and pushed updated password store entry (`gitea/token/pikvm_studio`) via `gpush pass`. - HeckWiki Documentation & Host Maintenance: - Authored and verified HeckWiki documentation pages for [pikvm-studio](https://wiki.drheck.dev/books/system-setup/page/pikvm-studio) and [automation](https://wiki.drheck.dev/books/system-setup/page/automation). - Verified LTS kernel update (`kernel-check`), performed planned system reboot on `ul`, and verified all running systemd services and Docker containers via `status_all`. - 2026-08-19 Wednesday - VPS rng Assessment & Infrastructure Monitoring Plan: - Inspected VPS `rng` (`rng.drheck.dev`) hardware resources, memory, disk, network, and listening services. - Authored and published the [Infrastructure Monitoring & Alerting Plan](https://wiki.drheck.dev/books/monitoring/page/infrastructure-monitoring-alerting-plan) on HeckWiki detailing metrics, alerting strategy, and agent deployment. - PiKVM Network Configuration: - Verified dual-stack IPv4/IPv6 reachability and latency response (`fping`) for `pikvm-studio`. - Added host entry and identity mapping in `~/.ssh/config` for direct SSH connectivity. - System Package & Media Maintenance: - Upgraded host packages via `paru -Syu` including Java runtime environment updates. - Built and installed latest `nethack-git` (5.0.0 release) from AUR. - Restarted `plexmediaserver` and monitored service journal logs. - 2026-08-18 Tuesday - ul NVIDIA Driver Upgrade & Container CDI Fix: - Diagnosed Immich Outage: Immich server (`immich_server`) and machine learning (`immich_machine_learning`) exited (127) after host NVIDIA driver upgrade (`580.173.02` → `580.178.04`). The OCI runtime failed mounting `/usr/lib/libEGL_nvidia.so.580.173.02` due to obsolete paths in `/etc/cdi/nvidia.yaml`. - Regenerated CDI Spec & Restored Services: Ran `nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml` to update the CDI specification for `580.178.04`, restarted the Immich container stack, and verified healthy container status and restored upstream proxying via Nginx (`photos.drheck.dev`). - Automated via Pacman Hook: Created `/etc/pacman.d/hooks/nvidia-cdi.hook` with a `PostTransaction` trigger on `nvidia*` and `*nvidia*` packages to automatically regenerate `/etc/cdi/nvidia.yaml` on future `paru`/`pacman` driver updates. - Opencode Wiki Skill & Knowledge Integration: - Embedded HeckWiki / BookStack management rules and helper workflows into global Opencode instructions (`~/.config/opencode/INSTRUCTIONS.md`). - Created dedicated skill `~/.config/opencode/skill/update-wiki/SKILL.md` for seamless automated wiki editing and querying. - Funkwhale Album Track Ordering Fix: - Diagnosed track numbering and sort discrepancy in Funkwhale where album and artist views displayed tracks out of sequence. - Directly updated track `position` attributes and calibrated `creation_date` timestamps in the Funkwhale database via Django management shell (`/app/api/manage.py shell`) for Album 10 so both default chronological sorting and album sequence sorting correctly align. - 2026-08-17 Monday - bmo95 Arch Linux Boot & dmesg Diagnostics & Resolution: - Resolved FireWire Audio Crash Loop & Massive dmesg Spam: - Diagnosed >50,000 repeating kernel errors (`snd_fireworks fw1.0: oPCR0: plug is already in use`) and FireWire CMP stack trace warnings (`sound/firewire/cmp.c:300` / `cmp.c:193`). - Fixed missing ALSA period count in `zita-j2a.service` and `zita-a2j.service` by adding `-n 3` (required by `snd_fireworks` driver) and fixed client naming (`-j Up` / `-j Up_In`). - Cleaned up `set-port-aliases.sh` to remove conflicting `pkill -9` invocations and background spawns that were killing services mid-handshake and wedging the FireWire CMP isochronous plug state. - Fixed hardware clock synchronization sequence: moved clock configuration into `firewire-ready.service` (`wait-for-firewire.sh`) using dynamic GUID-to-node matching (`0x00148602c583fc79` -> Internal Master, `0x001486a296a59815` -> Word Clock Slave) with a 2-second bus settling delay before starting JACK. - Verified full 24-channel capture/playback routing stably running in JACK with zero kernel errors on boot. - Udev Rules & System Tuning: - Fixed udev substitution syntax in `50-firewire-irq.rules` (piping `pgrep` into `xargs` to avoid invalid `$()` substitution error in udevd). - Fixed quote syntax and `power/control` attribute assignment in `50-usb-midi.rules` for low-latency Emagic Unitor8 USB MIDI operation. - Configured `WIRELESS_REGDOM="US"` in `/etc/conf.d/wireless-regdom`. - Wireless Regulatory Database: Installed `wireless-regdb` package to resolve `Direct firmware load for regulatory.db failed with error -2`. - PCIe Correctable AER Timeouts: Added `pci=noaer` to `GRUB_CMDLINE_LINUX_DEFAULT` in `/home/luke/git/audio_config/common/grub` and regenerated GRUB config to eliminate spurious correctable error logging from the ASMedia USB 3.1 controller. - ASUS ACPI SuperIO Hardware Monitor Conflict: Added `acpi_enforce_resources=lax` to GRUB kernel parameters to resolve ACPI OpRegion conflict on ASUS PRIME Z270-A motherboard. - 2026-08-16 Sunday - bmo95 Audio Workstation Infrastructure & Low-Latency Tuning: - Multi-Profile Audio Config Repo: Established `/home/luke/git/audio_config` git repository tracking audio profiles (`jack-alsa`, `pipewire`), metadata mappings (`filedata.json`), and custom management scripts (`apply-profile`, `consume`). - Sound Card & Clock Synchronization: Configured dual Echo AudioFire12 interfaces with forced internal clock on primary card (Node 1 / Bottom card) and Word Clock sync on secondary card (Node 2 / Top card). - ALSA + JACK2 Engine Setup: Configured JACK2 server via ALSA backend (48kHz / 512 period / 3 buffer) for low-latency audio processing. - Systemd User Service Modularization: Modularized JACK audio services into individual systemd user units (`jack.service`, `firewire-ready.service`, `zita-j2a.service`, `zita-a2j.service`, `a2jmidid.service`, `jack-aliases.service`, `patchage.service`, `ardour.service`). - Custom Port Aliases & Metadata: Scripted automatic routing and pretty-name metadata (`Down_01`..`Down_12` and `Up_01`..`Up_12`) for hardware inputs and outputs. - Realtime System & Kernel Tuning: Configured GRUB kernel parameter `threadirqs`, udev rules for FireWire IRQ thread scheduling (`50-firewire-irq.rules`), HPET/RTC audio timers, sysctl swappiness, and PAM realtime privileges. - Software & Documentation: Enabled Ardour 9 integration (`use-jack-pretty-names`) and added comprehensive architecture and routing documentation (`README.md`). - System SSH Audit Logging Setup on bmo95: - Installed `audit` package, enabled `auditd`, and configured `/etc/audit/rules.d/execve.rules` with strict `auid >= 1000` human session command tracking. - Deployed `/usr/local/bin/audit-log` execution tracking script. - 2026-08-12 Wednesday - BookStack Wiki & Nginx Reverse Proxy SSO Integration: - Reverse Proxy Routing: Configured Nginx `/login` redirect behavior. - Unauthenticated Homepage Auto-Redirect: Implemented DOMContentLoaded JavaScript trigger in BookStack's `app-custom-head` setting. - Ensured unauthenticated visitors accessing `https://wiki.drheck.dev/` are automatically forwarded to Authentik (`/login`), while keeping deep links (e.g. `/books/...`) publicly viewable. - this is for convenience, not security. I just don't want to click in "login" if I visit [wiki/](http://wiki/) - 2026-08-10 Monday - Opencode CLI Agent Configuration & Setup: - Configured Opencode settings (`opencode.json` / `opencode.jsonc`) and system instructions. - Added instruction mandate to comment bash tool command batches. - Deployed global Opencode skills under `~/.config/opencode/skill/`. - SSH Audit Logging & Command Tracking: - Configured Linux `auditd` rules on `ul` for `auid >= 1000` system call executions. - Configured OpenBSD process accounting (`acct`) and verbose SSH payload auditing on `danknasty`. - Implemented unified `audit-log` CLI helper scripts across both machines. - Documented SSH command tracking guide and created `daily-change-list` skill. - 2026-08-09 Sunday - [documenting services](https://wiki.drheck.dev/books/system-setup/page/service-list) - in this process saw that kavita permissions were messed up, fixed - saw that moebius is like halfway removed but don't know why, leading to the realization that I need to track commands. - [SSH Command Tracking](https://wiki.drheck.dev/books/system-setup/page/ssh-command-tracking-guide) - now I remember why it was removed, because it sucks. The "web service" is just a viewer for multiple drawers connecting with thier own clients. - some updates to the crowdsec blocks. realized it has crowed sourced blocking rules, not sure about implementing them though. - 2026-08-08 Saturday - setting up navidrome, and other audio related software. - put albums up on funkwhale - power outtage caused IPv6 netblock to change, the IPv6 update scripts still suck. Tried to make them better - had AI write a script to pull down missing artwork for Jellyfin - noticed collation version mismatch when poking around in posgresql. - created a short-link service - drheck.dev/$short_link - editable at drheck.dev/admin 1. Check Which Databases Have Collation Version Mismatches ```bash sudo -u postgres bash -c 'for db in $(psql -At -c "SELECT datname FROM pg_database WHERE datistemplate = false;"); do echo "=== DATABASE: $db ===" psql -d "$db" -c "SELECT 1;" 2>&1 done' ``` 2. Run amcheck to Detect Corrupted B-Tree Indexes ```bash sudo -u postgres bash -c ' for db in spacebar immich nextcloud_db anchor authentik postgres gitea nextcloud; do echo "=== AMCHECK: $db ===" psql -d "$db" -c " CREATE EXTENSION IF NOT EXISTS amcheck; SELECT c.relname, bt_index_check(c.oid, true) FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid JOIN pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind = '\\\\\\\\''i'\\\\\\\\'' AND i.indisready AND i.indisvalid AND n.nspname = '\\\\\\\\''public'\\\\\\\\'';" 2>&1 done' ``` 3. Reindex Databases and Refresh Collation Versions ```bash sudo -u postgres bash -c ' for db in spacebar immich nextcloud_db anchor authentik postgres gitea nextcloud; do echo "=== REINDEXING & REFRESHING: $db ===" psql -d "$db" -c "REINDEX DATABASE \\\\\\\\"$db\\\\\\\\";" psql -d "$db" -c "ALTER DATABASE \\\\\\\\"$db\\\\\\\\" REFRESH COLLATION VERSION;" done' ``` 4. Verify All Warnings Are Cleared ```bash sudo -u postgres bash -c ' for db in $(psql -At -c "SELECT datname FROM pg_database WHERE datistemplate = false;"); do echo "=== VERIFY: $db ===" psql -d "$db" -c "SELECT 1;" 2>&1 done' ``` - 2026-08-05 Wednesday - updates to crowdsec - parsing more SSH errors - generated a table of banned IPs but it's not very interesting. Live world view would be cooler - 2026-08-04 Tuesday - docker updates - anchor: 0.13.0 → 0.14.0 - authentik: 2026.5.5 → 2026.5.6 - bookstack: Updated to latest image - immich: v3.0.3 → v3.1.0 - infra (valkey): Updated to latest image build - kasm: Updated to latest image - nextcloud: 34.0.1-apache → 34.0.2-apache - penpot: Frontend, backend, exporter, postgres, and redis updated - romm: Updated to latest image (5.1.0) - vaultwarden: Updated to latest image - webrcade: Updated to latest image - wick: Updated to latest image - installed crowdsec - review of publicly accesible service logs (eg: wiki) shows no bot traffic (yet)! - review of nginx logs straight from the public IPs shows bot traffic looking for common vulnerabilities, exploits, and staged malware. - [documentation](https://wiki.drheck.dev/books/system-setup/page/crowdsec-openbsd-pf-perimeter-security-setup) - 2026-08-03 Monday - pull all new docker images and prepare for update - in preparation for more users, ensure unencrypted backups aren't world readable - gdrive backup credentials failed again - finally cleaned up google cloud project, renamed properly and created two separate subprojects, google sso, and rclone for ul - redid the certs for rclone, tested good - ran a backup to repare for docker update and all the online backups had no files except the instructions! - setting all the backups to root rendered the rclone script useless. - Trying all backup files owned by my user with a plan - create a dedicated backup user - create a new gpg key for backups - backup gpg key - move rclone credentials to backup user - chown all backups to 700 backup user - Profit! - 2026-08-02 Sunday - Removed old unused "web shells" like the old ttyd, whatever else. Also warpgate. - Made webshell awesome. Picked a single font, created a service to show ANSI art on login - made big improvements to ansview - first render is completely in memory, remove all jumps and relative and exact positions. This allows the final product to be rendered correctly when not placed at the first screen column. - due to this success, added an auto wide mode, if artwork can fit on the screen multiple times horizontally, do it - also truncate the artwork if viewing on a smaller screen rather than an awful wrap. This is great for the phone shell. - 2026-08-01 Saturday - Created webshell, and spent a lot time trying to make it customizable per user. This was a failure. - 2026-07-31 Friday - Set up moebius in KASM, generated a [KASM setup guide](https://wiki.drheck.dev/books/system-setup/page/kasm-container-build). - 2026-07-30 Thursday - giving up on warpgate for now, trying ttyd again with step-ca to tie it into OIDC. I really want an authentik OIDC enabled direct web -> shell. - 2026-07-28 Tuesday - got warpgate to stop crashing but I'm not sure how I can use it for a web based terminal.. we'll see - 2026-07-26 Sunday - accidentally blocked the git users with the new firewall rules, allowed it to access the gitea service and we are able to upload again - started to set up [warpgate](https://github.com/warp-tech/warpgate) as it looks like it can provide a shell that can login with OIDC - 2026-07-25 Saturday - replaced gemini-cli with opencode, it's way better - went through each docker service moving off of the default 1000 UID - removed broken web "shells" - Made a process to generate short lived ssh certs for web users to get a shell without typing a password but it felt really insecure and I think you could impersonate other users, so I killed it. - 2026-07-23 Thursday - Set nginx to cache large uploads on the ZFS array as / was filling up with video uploaded from immich - Set permanent redirects on short names in the browser for convenience, eg: http://photos -> https://photos.drheck.dev/ - 2026-07-22 Wednesday - Configured rclone for Google drive and Proton drive. [Added](https://git.drheck.dev/luke/backups/commit/c0d835fc574ddc78ae6b1d9840c42de10dbd45d1) this to the backup scripts. Now we have automated on-line backups. - created rclone client config in Google's [cloud console](https://console.cloud.google.com/) - 2026-07-21 Tuesday - transmission would timeout quickly since exposed behind authentik. After some research I modifed the nginx config to start forwarding `$host`, `$remote_addr`, `$proxy_add_x_forwarded_for`, and `$scheme`. - 2026-07-20 Monday - IPv6 netblock was wrong again. This time an IPv6 netblock change was detected then the network went out. This caused the API call to desec.io to fail. We were writing the new IPv6 netblock unconditionally though so no more updates were going to happen. Script is updated [1](https://git.drheck.dev/luke/router_config/commit/5eb5286403f3239676983bd71b6a446003775e69) [2](https://git.drheck.dev/luke/router_config/commit/ec44c1cafef9abff1db9f524d072993b47cbf6da) - also added logging to the desec.io call in change 2 ^ - 2026-07-19 Sunday - Tested LDAP users - could log in but did not have the ability to choose the window environment - [LDAP](https://wiki.drheck.dev/books/ldappamssh-ca/page/ldap-sssd-client-enrollment-guide) guide updated to fix this problem - 2026-07-17 Friday - [LDAP](https://wiki.drheck.dev/books/ldappamssh-ca/page/ldap-sssd-client-enrollment-guide) guide updated to add `systemd` to `shadow` for NSS. This was breaking the graphical boot environment on Mitsuki's laptop. - [write-up](https://wiki.drheck.dev/books/system-setup/page/nss-gdm-issue) - 2026-07-16 Thursday - [router updates](https://git.drheck.dev/luke/router_config/commit/06b21af8156cca40ed5ff8f8848896b894687b8b) - Remote IPv6 was not updated in public DNS. The router update script was failing to execute in cron - shebang was /usr/local/bin/bash but - cron executed /bin/ksh $script... (just openbsd things!) - fixed - also router IPv6 [update script updated](https://git.drheck.dev/luke/router_config/src/branch/main/update_dns.sh#L6) to use persistent storage for IPv6 address as a reboot would remove the older address - bad firewall rule forwarded all IPv6 traffic arriving on 22 to ul - fixed - 2026-07-13 Monday - created [ldap enable](https://wiki.drheck.dev/books/ssh-certificate-authority/page/ldap-enable) doc and test on desktop - works! - completed SSH [cert setup](https://wiki.drheck.dev/books/ldappamssh-ca/page/ssh-ca-setup) for deskop - works! - drafted a process to [auto generate ssh keys and sign them with authentik](https://wiki.drheck.dev/books/ldappamssh-ca/page/auto-generate-signed-ssh-keys) - 2026-07-12 Sunday - created "admin" specific firefox profile on desktop - cleared all old disparate passwords for the old individual service passwords - fixed www server for mitsuki - checked in [kanji service](https://kanji.drheck.dev) in [gitea](https://git.drheck.dev/luke/kanji) - added bounding boxes - changed the stroke order color to be more legible and not run into the background - 2026-07-11 Saturday - created this server change log - added "backup_all" script to run each service backup - finally [checked in](https://git.drheck.dev/luke/backups) backup/restore scripts - separated backup output into current and all previous to simplify disk backup - modified nginx config for immich, upload cap 30GB -> 100GB # NSS + GDM Issue ##### 1. GDM Requests a Dynamic User When GDM starts up it attempts to launch the graphical login screen. For security isolation GDM requests a session for a systemd dynamic user named `gdm-greeter` from `systemd-logind`. * Systemd dynamically allocated a UID (60578 in this case) and registered the username gdm-greeter to it. * Logind triggers the creation of the user's slice and runtime directory ``` systemd[1]: Created slice User Slice of UID 60578. systemd-logind[426]: New session '1' of user 'gdm-greeter' with class 'greeter' and type 'wayland'. ``` ##### 2. Systemd Spawns the User Manager and Invokes PAM To run the background GNOME session, systemd attempts to spawn the per-user systemd instance (user@60578.service). Before launching the service, systemd must authenticate and initialize the user's environment using the PAM stack configured in `/etc/pam.d/systemd-user` (which inherits `/etc/pam.d/system-auth`). ##### 3. pam_sss.so Gracefully Rejects the Dynamic User First, the account block in the PAM stack runs pam_sss.so: ``` account sufficient pam_sss.so ``` * pam_sss queries the SSSD daemon: "Do you have a user named gdm-greeter?" * SSSD queries your LDAP server and finds nothing. SSSD replies: "No, this user is unknown to me." * This generates the log entry: ``` 1 (systemd)[508]: pam_sss(systemd-user:account): Access denied for user gdm-greeter: 10 (User not known to the underlying authentication module) ``` * Because pam_sss is configured as sufficient, its failure is non-fatal. PAM ignores this rejection and falls through to the next module in the stack: pam_unix.so. ##### 4. pam_unix.so Spawns unix_chkpwd To verify the local account, pam_unix.so spawns the setuid helper binary /usr/bin/unix_chkpwd to check the user's password and shadow database info: ``` 1 account required pam_unix.so ``` ##### 5. The Silent NSS Database Lookup Failure Inside the C code of /usr/bin/unix_chkpwd, the program makes a standard glibc library call to look up the shadow password entry for the username: ``` 1 getspnam("gdm-greeter"); ``` To resolve this call, glibc reads your `/etc/nsswitch.conf` file to see which databases are registered under the shadow: line. It saw: ``` 1 shadow: compat sss ``` Glibc began querying the modules in order: 1. compat (local files): It searched /etc/shadow for an entry starting with gdm-greeter:. Since gdm-greeter is a dynamic user, no such line exists in /etc/shadow. This returns: ``` NSS_STATUS_NOTFOUND. ``` 2. sss (SSSD): It asked SSSD if it has a cached shadow record for gdm-greeter. SSSD does not. This returns NSS_STATUS_NOTFOUND. Because systemd was missing from that line, glibc reached the end of the databases and stopped. It never queried nss-systemd (which is the only place where GDM's dynamic users are registered). As a result, getspnam("gdm-greeter") returned NULL (No such entry). ##### 6. PAM Crashes, Locking Out the Session * unix_chkpwd realized it was completely blind to this user and threw an error: ``` unix_chkpwd[509]: could not obtain user info (gdm-greeter) ``` * Because unix_chkpwd failed, pam_unix.so returned a fatal error to PAM. * Because pam_unix.so is required, the entire PAM transaction aborted: ``` (systemd)[508]: PAM PAM failed: Authentication service cannot retrieve authentication info ``` * Systemd saw the PAM initialization failure and immediately aborted spawning the user manager: ``` (systemd)[508]: user@60578.service: Failed to set up PAM session: Operation not permitted (systemd)[508]: user@60578.service: Failed at step PAM spawning /usr/lib/systemd/systemd: Operation not permitted systemd[1]: Failed to start User Manager for UID 60578. ``` ##### 7. Crash and Lockout Without a running systemd --user manager for UID 60578, the user-level D-Bus broker (org.freedesktop.systemd1) was never launched. When the GDM Wayland session tried to initialize gnome-session, it attempted to talk to the systemd user manager over D-Bus. This lookup failed, causing the greeter worker to throw an unhandled assertion and crash (dumping core): ``` gnome-session-i[518]: Failed to start unit gnome-session@gnome.target: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.freedesktop.systemd1" does not exist systemd-coredump[527]: Process 518 (gnome-session-i) of user 60578 dumped core. ``` GDM attempted to restart this sequence six times, failing every time, until it reached its failure threshold: ``` gdm[455]: Gdm: GdmLocalDisplayFactory: maximum number of display failures reached. Giving up. ``` At this point, GDM completely gave up on opening any graphical interfaces on your screen, leaving the system stuck in the console/tty mode. By adding systemd back to the shadow: line in /etc/nsswitch.conf, the glibc call getspnam("gdm-greeter") was able to query nss-systemd, retrieve the dynamic user's shadow record, pass the PAM stack, start the user manager, and successfully launch the login screen. # Web-based Shell # Web based SSH Instant Access This is a browser-based SSH terminal service (`webshell.drheck.dev`) authenticated via Authentik Single Sign-On (SSO) and short-lived OpenSSH certificates signed by Smallstep (`step-ca`), running under rootless Podman Quadlets (`docker-data`, `UID 1100`). --- ## 1. Architectural Overview ``` [ Browser Client ] │ ▼ (HTTPS / WebSocket) [ Nginx Reverse Proxy (/etc/nginx/sites-available/webshell) ] │ ├── auth_request ──► [ Authentik Embedded Outpost ] (SSO Auth & Claims) │ ▼ (HTTP / WebSocket + X-Authentik-Username Header) [ Webshell Rootless Container (ttyd + login-wrapper.sh) on eth77 fd77:...:771d:7681 ] │ ├── Step SSH Certificate Sign ──► [ Step-CA Container on webshell_net (step-ca:9000) ] │ ▼ (SSH with Ephemeral Ed25519 Certificate over IPv6) [ Host ul OpenSSH Daemon on [fd77:7777:5010:0:1337:f1ac:d00d:337]:22 ] ``` ### Key Components 1. **`step-ca`**: Smallstep Certificate Authority issuing 5-minute ephemeral Ed25519 user certificates (`step-ca.container` on `webshell.network`). 2. **`webshell`**: Debian Bookworm container running static `ttyd` 1.7.4, `step` CLI 0.25.0, and `login-wrapper.sh` (`webshell.container` on `webshell.network`). 3. **Authentik**: Provides OIDC integration for `step-ca` and ForwardAuth single sign-on (`webshell.drheck.dev`). 4. **Nginx**: Terminates TLS and proxies HTTP/WebSocket connections while passing authenticated identity headers (`X-Authentik-Username`). 5. **OpenSSH (`sshd`)**: Configured to trust `step-ca` signed user certificates for passwordless OS login on host `ul`. --- ## 2. Network & ULA IPv6 Allocation Every web-facing service on host `ul` binds to a unique ULA IPv6 address on interface `eth77` (`fd77:7777:5010:0:1337:f1ac:d00d:.../64`). * **`step-ca` IPv6**: `fd77:7777:5010:0:1337:f1ac:d00d:ca` (Port `9000`, also used locally by `/data/minecraft/ipctl`) * **`webshell` IPv6**: `fd77:7777:5010:0:1337:f1ac:d00d:771d` (Port `7681`) * **Private Bridge Network**: `webshell.network` (`webshell_net`, serving `step-ca` container DNS via `aardvark-dns`) * **Host `ul` ULA IPv6 SSH Target**: `fd77:7777:5010:0:1337:f1ac:d00d:337` (Port `22`) --- ## 3. Step 1: Rootless Podman Quadlet Configuration (`step-ca` & `webshell`) ### A. Quadlet Unit Definitions (`/data/docker/webshell/`) Both Quadlets are symlinked into `/home/docker-data/.config/containers/systemd/` and managed by `user@1100.service`: **`/data/docker/webshell/step-ca.container`:** ```ini [Unit] Description=Smallstep SSH & X.509 CA (Rootless Podman) After=network-online.target Wants=network-online.target [Container] Pull=never ContainerName=step-ca Image=docker.io/smallstep/step-ca:latest Network=webshell.network AddHost=auth.drheck.dev:169.254.1.2 PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:ca]:9000:9000 EnvironmentFile=/data/docker/webshell/.env Volume=/data/docker/webshell/step-ca:/home/step:rw [Service] Restart=always RestartSec=5 [Install] WantedBy=default.target ``` **`/data/docker/webshell/webshell.container`:** ```ini [Unit] Description=Webshell Browser SSH Terminal (Rootless Podman) After=network-online.target step-ca.service Requires=step-ca.service [Container] Pull=never ContainerName=webshell Image=localhost/webshell:latest Network=webshell.network PublishPort=[fd77:7777:5010:0:1337:f1ac:d00d:771d]:7681:7681 Environment=STEP_CA_URL=https://step-ca:9000 Volume=/data/docker/webshell/login-wrapper.sh:/usr/local/bin/login-wrapper.sh:ro Volume=/data/docker/webshell/known_hosts:/etc/ssh/ssh_known_hosts:ro Volume=/etc/passwd:/etc/ul_passwd:ro Volume=/data/docker/webshell/step-ca/certs/root_ca.crt:/etc/step-ca/root_ca.crt:ro Volume=/data/docker/webshell/step-ca/secrets/webshell_svc_pwd:/etc/step-ca/webshell_svc_pwd:ro Volume=/data/docker/webshell/index.html:/etc/ttyd/index.html:ro Exec=-W -H X-Authentik-Username -t theme={"background":"#000000"} -I /etc/ttyd/index.html /usr/local/bin/login-wrapper.sh [Service] Restart=always RestartSec=5 [Install] WantedBy=default.target ``` ### B. CA Directory Permissions (Rootless User Namespace) Inside `docker-data`'s rootless user namespace (`docker-data:624288:65536`), the `step` user inside `smallstep/step-ca` maps cleanly to an isolated unprivileged subordinate UID on the host with zero world permissions (`o-rwx`). --- ## 4. Step 2: Host `ul` OpenSSH Daemon Trust & Web Session Banner Setup To allow `sshd` on host `ul` (`[fd77:7777:5010:0:1337:f1ac:d00d:337]:22`) to accept certificates signed by `step-ca`, enforce strict `known_hosts` checking, and serve a dedicated login banner exclusively for web sessions: 1. Extract the CA user public key: ```bash (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec step-ca step certificate inspect --format json /home/step/certs/user_ca_key.pub) ``` 2. Append the public key (`ecdsa-sha2-nistp256 ...` or `ssh-ed25519 ...`) to `/etc/ssh/drheck_ca_user_key.pub`. 3. Verify that `/etc/ssh/sshd_config.d/10-drheck-certs.conf` contains: ```text TrustedUserCAKeys /etc/ssh/drheck_ca_user_key.pub HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub Match Address fd77:7777:5010:771d::/64 Banner /etc/ssh/webshell_banner.txt ``` 4. Web Session Banner & Rotation Script: `/etc/profile.d/webshell_banner.sh` executes when a web session connects. It randomly selects an ANSI art file from `/data/nfo/ansi_list.txt` (42,000+ files), renders it via `/usr/local/bin/ansview --wide-dupe` across terminal width, right-aligns the file path, and exports `$LAST_ANSI` for the `/usr/local/bin/bad_ansi` removal command. The static fallback banner is stored at `/etc/ssh/webshell_banner.txt`. 5. Generate `/data/docker/webshell/known_hosts` from host `ul`'s public host key: ```bash echo "fd77:7777:5010:0:1337:f1ac:d00d:337 $(cat /etc/ssh/ssh_host_ed25519_key.pub)" > /data/docker/webshell/known_hosts chmod 644 /data/docker/webshell/known_hosts ``` 6. Reload OpenSSH daemon: ```bash sudo systemctl reload sshd ``` --- ## 5. Step 3: Web Terminal Backend (`ttyd` + `login-wrapper.sh`) Setup ### A. Dockerfile (`/data/docker/webshell/Dockerfile`)

Rootless Podman ttyd Listener Note: Do NOT pass -6 -i :: under rootless Podman. When rootless Podman forwards traffic from host [fd77:...:771d]:7681 into webshell.network, the forwarded socket arrives over container IPv4 (10.89.X.X:7681). Omitting -6 -i :: allows ttyd to bind all interfaces inside the container cleanly instead of rejecting IPv4 forwards with 502 Bad Gateway.

```dockerfile FROM debian:bookworm-slim ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt-get install -y --no-install-recommends \ ca-certificates curl openssh-client bash procps libnss-sss \ && sed -i 's/passwd: files/passwd: files sss/' /etc/nsswitch.conf \ && rm -rf /var/lib/apt/lists/* # Install static ttyd 1.7.4 & step CLI 0.25.0 RUN curl -sSL https://github.com/tsl0922/ttyd/releases/download/1.7.4/ttyd.x86_64 -o /usr/local/bin/ttyd \ && chmod +x /usr/local/bin/ttyd \ && curl -sSL https://dl.step.sm/gh-release/cli/docs-step-cli/v0.25.0/step_linux_0.25.0_amd64.tar.gz | tar -xz -C /tmp \ && mv /tmp/step_0.25.0/bin/step /usr/local/bin/step \ && chmod +x /usr/local/bin/step \ && rm -rf /tmp/step_0.25.0 COPY login-wrapper.sh /usr/local/bin/login-wrapper.sh RUN chmod +x /usr/local/bin/login-wrapper.sh EXPOSE 7681 ENTRYPOINT ["/usr/local/bin/ttyd"] CMD ["-W", "-H", "X-Authentik-Username", "-t", "theme={\"background\": \"#000000\"}", "-I", "/etc/ttyd/index.html", "/usr/local/bin/login-wrapper.sh"] ``` ### B. Backend Login Wrapper (`/data/docker/webshell/login-wrapper.sh`) ```bash #!/usr/bin/env bash set -euo pipefail # Resolve username across CGI-style environment variables exported by ttyd/proxy TARGET_USER="${X_AUTHENTIK_USERNAME:-${TTYD_USER:-${HTTP_X_AUTHENTIK_USERNAME:-${AUTHENTIK_USERNAME:-}}}}" if [[ -z "${TARGET_USER}" ]]; then echo "Authentication Error: Missing Authentik identity." >&2 exit 1 fi # Strict alphanumeric regex check if [[ ! "${TARGET_USER}" =~ ^[a-z_][a-z0-9_-]{0,31}$ ]]; then echo "Authentication Error: Invalid username format." >&2 exit 1 fi # Retrieve numeric UID from host passwd database TARGET_UID=$(awk -F: -v u="${TARGET_USER}" '$1 == u {print $3}' /etc/ul_passwd 2>/dev/null || true) if [[ -z "${TARGET_UID}" ]]; then echo "Authentication Error: User does not exist on target host." >&2 exit 1 fi # Reject system accounts (UID < 1000), admin, and docker* accounts if (( TARGET_UID < 1000 )) || [[ "${TARGET_USER}" == "admin" ]] || [[ "${TARGET_USER}" == docker* ]]; then echo "Authentication Error: Access to privileged or system account is forbidden." >&2 exit 1 fi KEY_DIR="$(mktemp -d -t ssh_XXXXXX)" KEY_PATH="${KEY_DIR}/id_ed25519" cleanup() { rm -rf "${KEY_DIR}" } trap cleanup EXIT INT TERM # Generate ephemeral Ed25519 keypair silently ssh-keygen -q -t ed25519 -f "${KEY_PATH}" -N "" >/dev/null 2>&1 # Sign user certificate via JWK provisioner (--sign flag is mandatory for non-interactive execution) if command -v step >/dev/null 2>&1 && [[ -n "${STEP_CA_URL:-}" ]] && [[ -f "/etc/step-ca/webshell_svc_pwd" ]]; then step ssh certificate "${TARGET_USER}" "${KEY_PATH}.pub" --sign --provisioner="webshell-svc" --provisioner-password-file="/etc/step-ca/webshell_svc_pwd" --ca-url="${STEP_CA_URL}" --root="/etc/step-ca/root_ca.crt" --not-after=5m --force >/dev/null 2>&1 || true fi # Login passwordless to host ul SSH daemon over local ULA IPv6 with strict known_hosts checking exec ssh -o StrictHostKeyChecking=yes -i "${KEY_PATH}" "${TARGET_USER}@fd77:7777:5010:0:1337:f1ac:d00d:337" "$@" ``` --- ## 6. Step 4: Authentik Single Sign-On Provisioning (`ak shell`) To configure Authentik ForwardAuth without encountering `500 Internal Server Error`, `Redirect URI Error`, or `ERR_TOO_MANY_REDIRECTS`, execute the following Python provisioning script in `authentik-server`: ```bash (cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -i authentik-server ak shell) << 'EOF' from authentik.core.models import Application from authentik.providers.proxy.models import ProxyProvider, ProxyMode from authentik.providers.oauth2.models import RedirectURI, RedirectURIMatchingMode, RedirectURIType from authentik.outposts.models import Outpost SERVICE_NAME = "webshell" EXTERNAL_HOST = f"https://{SERVICE_NAME}.drheck.dev" # 1. Inherit default scope property mappings from a working provider template template_provider = ProxyProvider.objects.get(name="cockpit") # 2. Create ProxyProvider with complete OAuth, redirect URI, and cookie settings provider, _ = ProxyProvider.objects.get_or_create( name=SERVICE_NAME, defaults={ "external_host": EXTERNAL_HOST, "mode": ProxyMode.FORWARD_SINGLE, "authorization_flow_id": "f9ad5326-f668-4d1f-861d-43b47f0a002e", "cookie_domain": "drheck.dev", "grant_types": ["authorization_code", "client_credentials", "password"], "redirect_uris": [ RedirectURI( matching_mode=RedirectURIMatchingMode.STRICT, url=f"{EXTERNAL_HOST}/outpost.goauthentik.io/callback?X-authentik-auth-callback=true", redirect_uri_type=RedirectURIType.AUTHORIZATION ), RedirectURI( matching_mode=RedirectURIMatchingMode.STRICT, url=f"{EXTERNAL_HOST}?X-authentik-auth-callback=true", redirect_uri_type=RedirectURIType.AUTHORIZATION ), ] } ) provider.property_mappings.set(template_provider.property_mappings.all()) provider.save() # 3. Create Application app, _ = Application.objects.get_or_create( slug=SERVICE_NAME, defaults={ "name": SERVICE_NAME.capitalize(), "meta_launch_url": f"{EXTERNAL_HOST}/", "open_in_new_tab": True, "provider": provider, } ) # 4. Mandatory: Assign provider to authentik Embedded Outpost outpost = Outpost.objects.get(name="authentik Embedded Outpost") if provider not in outpost.providers.all(): outpost.providers.add(provider) outpost.save() print(f"Successfully provisioned {SERVICE_NAME} and attached to Embedded Outpost.") EOF ``` --- ## 7. Step 5: Nginx Reverse Proxy Configuration (`/etc/nginx/sites-available/webshell`) ```nginx map $http_upgrade $proxy_target_connection_webshell { default upgrade; "" ""; } server { listen 443 ssl; listen [::]:443 ssl; http2 on; server_name webshell.drheck.dev; ssl_certificate /etc/nginx/certs/fullchain.pem; ssl_certificate_key /etc/nginx/certs/privkey.pem; client_max_body_size 64M; # Authentik ForwardAuth Subrequest Configuration auth_request /outpost.goauthentik.io/auth/nginx; error_page 401 = @goauthentik_proxy_signin; auth_request_set $auth_cookie $upstream_http_set_cookie; add_header Set-Cookie $auth_cookie; auth_request_set $authentik_username $upstream_http_x_authentik_username; auth_request_set $authentik_email $upstream_http_x_authentik_email; # Per-User Custom Fonts Route location /user-fonts/ { auth_request /outpost.goauthentik.io/auth/nginx; auth_request_set $authentik_username $upstream_http_x_authentik_username; alias /home/$authentik_username/.config/webshell/fonts/; types { font/woff2 woff2; font/ttf ttf; font/otf otf; } add_header Cache-Control "public, max-age=86400"; } location / { proxy_pass http://[fd77:7777:5010:0:1337:f1ac:d00d:771d]:7681; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $proxy_target_connection_webshell; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Authentik-Username $authentik_username; proxy_set_header X-Authentik-Email $authentik_email; } location /outpost.goauthentik.io { proxy_pass https://auth.drheck.dev/outpost.goauthentik.io; proxy_set_header Host auth.drheck.dev; proxy_set_header X-Original-URL $scheme://$http_host$request_uri; proxy_set_header X-Forwarded-Host $http_host; proxy_set_header X-Forwarded-Proto $scheme; add_header Set-Cookie $auth_cookie; auth_request_set $auth_cookie $upstream_http_set_cookie; proxy_pass_request_body off; proxy_set_header Content-Length ""; } location @goauthentik_proxy_signin { internal; add_header Set-Cookie $auth_cookie; return 302 /outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri; } } ``` --- ## 8. Step 6: Backup & Restore Scripts Every service on host `ul` uses `shellcheck`-verified backup and restore scripts located in `/home/luke/git/backups/ul/scripts/` with 10% `par2` redundancy: * `/home/luke/git/backups/ul/scripts/backup/webshell`: Archives `/data/docker/webshell/` (`.env`, `step-ca.container`, `webshell.container`, `webshell.network`, `step-ca/`) to `webshell_data.tar.xz`, and system configuration/scripts (`/etc/ssh/webshell_banner.txt`, `/etc/ssh/drheck_ca_user_key.pub`, `/etc/ssh/sshd_config.d/10-drheck-certs.conf`, `/etc/nginx/sites-available/webshell`, `/etc/profile.d/webshell_banner.sh`, `/usr/local/bin/bad_ansi`, `/data/nfo/ansi_list.txt`) to `webshell_system_config.tar.xz`, generating `.par2` recovery files via `par2 create -r10`. * `/home/luke/git/backups/ul/scripts/restore/webshell`: Verifies archives with `par2 verify`, restores configurations into `/data/docker/webshell/` and `/`, and restarts `step-ca.service` and `webshell.service`. --- ## 9. Initial Assumptions & Attempts vs. Actual Working Configuration | Technical Area | Initial Assumption / Attempt | Actual Working Configuration & Root Cause | Prevention Standard | | :--- | :--- | :--- | :--- | | **Outpost Assignment** (`500 Internal Server Error`) | Creating `ProxyProvider` in Authentik automatically activates ForwardAuth routing. | Creating a provider does not assign it to an Outpost. Unassigned hostnames return `404 Not Found` from `/outpost.goauthentik.io/auth/nginx`, which Nginx converts to `500 Internal Server Error`. | Explicitly attach providers to `authentik Embedded Outpost` (`outpost.providers.add(provider)`) before enabling Nginx traffic. | | **Redirect URIs** (`Redirect URI Error`) | Instantiating `ProxyProvider` sets default authorization callback URLs. | The Django ORM defaults `redirect_uris` to `[]`. Authentik rejects callback URIs against an empty database list. | Explicitly define `RedirectURI` objects for both `/outpost.goauthentik.io/callback` and root `/` in `ProxyProvider.redirect_uris`. | | **Cookie Domain & Scopes** (`ERR_TOO_MANY_REDIRECTS`) | Providers operate without defining `cookie_domain`, `grant_types`, or `property_mappings`. | Without `property_mappings`, the OAuth request omits critical scopes (`ak_proxy`). Without explicit `cookie_domain='drheck.dev'`, Authentik refuses to grant ForwardAuth proxy tokens, causing an endless redirect loop. | Clone `property_mappings` from `cockpit` and explicitly set `cookie_domain='drheck.dev'` and `grant_types`. | | **CGI Env Mapping** (`Missing Authentik identity`) | `ttyd -H X-Authentik-Username` exports `X_AUTHENTIK_USERNAME` directly to child scripts. | `ttyd` transforms HTTP headers into standard CGI environment variables (`HTTP_X_AUTHENTIK_USERNAME`). | Always check CGI-style fallback variations: `TARGET_USER="${X_AUTHENTIK_USERNAME:-${TTYD_USER:-${HTTP_X_AUTHENTIK_USERNAME:-${AUTHENTIK_USERNAME:-}}}}"`. | | **Rootless Port Forwarding vs. `-6 -i ::`** (`502 Bad Gateway`) | Keeping legacy Docker `"-6", "-i", "::"` flags works identically in rootless Podman. | Under rootless Podman, `PublishPort=[fd77:...:771d]:7681:7681` forwards traffic into `webshell_net` over IPv4 (`10.89.X.X:7681`). Passing `-6 -i ::` restricts `ttyd` to IPv6-only and rejects forwarded connections. | Omit `-6 -i ::` in `webshell.container` `Exec=` so `ttyd` listens on all interfaces inside the container namespace. | | **Container Script Updates** (`old script execution`) | Modifying `login-wrapper.sh` on host `ul` and restarting runs the updated script only if mounted. | Restarting only re-reads bind-mounted files, not `COPY`ed image layers. | Bind-mount operational scripts (`Volume=/data/docker/webshell/login-wrapper.sh:/usr/local/bin/login-wrapper.sh:ro`) in `webshell.container`. | | **CA Secret Privilege Restrictions** | Administrative or personal user accounts own the CA keys and JWK provisioner secret. | Standard unprivileged or administrative (`wheel`) user accounts should not have read access to CA private keys or provisioner passwords. | Isolate `/data/docker/webshell/step-ca` inside `docker-data`'s rootless subordinate UID namespace with `o-rwx` permissions. | | **SSH Client Host Warnings** (`Warning: Permanently added...`) | Invoking `ssh` with `/dev/null` as known_hosts prints key addition warnings on every login. | Connecting without a populated known_hosts file causes OpenSSH to log a warning to stderr whenever an ephemeral host entry is added. | Mount a static `/etc/ssh/ssh_known_hosts` file generated from `/etc/ssh/ssh_host_ed25519_key.pub` and enforce `StrictHostKeyChecking=yes`. | | **Cockpit Login MOTD Banner** (`Web console: https://ul...:9090`) | Cockpit's MOTD issue updater (`update-issue`) injects `:9090` URLs into `/etc/motd.d/`. | Cockpit's `cockpit-issue.service` inspects local interfaces and writes direct port URLs to `/run/cockpit/issue`, which `/etc/motd.d/cockpit` symlinks into login greetings. | Remove `/etc/motd.d/cockpit` and `/etc/issue.d/cockpit.issue` symlinks and mask `cockpit-issue.service` (`systemctl mask cockpit-issue.service`). | | **Dynamic Host UID Verification** | Hardcoding system username lists (`root|admin|...`) in wrapper scripts to block system accounts. | Hardcoded string checks are fragile and fail to account for newly created local or system accounts. | Mount host `/etc/passwd` as `/etc/ul_passwd:ro` and dynamically validate that `TARGET_UID >= 1000` via `awk`. | | **Web-Session-Only Login Banner** | Printing greetings via script `echo` or editing server-wide `/etc/motd`. | Script `echo` output is lost before terminal attachment, while `/etc/motd` changes affect standard SSH logins. | Configure `Match Address fd77:7777:5010:771d::/64` in `/etc/ssh/sshd_config.d/10-drheck-certs.conf` with `Banner /etc/ssh/webshell_banner.txt`. | # Customization # Webshell Configuration Guide Per-user web terminal preferences are stored in `~/.config/webshell/`. --- ## Directory Structure ```text ~/.config/webshell/ ├── config.json # Theme, font size, and cursor settings └── fonts/ # Dir for local fonts └── font.woff2 # (Optional) Local custom font file (.woff2, .ttf, or .otf) ``` --- ## `config.json` Options | Key | Type | Default | Description | | :--- | :--- | :--- | :--- | | `fontFamily` | String | `'JetBrains Mono', monospace` | Font family stack | | `fontSize` | Number | `15` | Font size in pixels | | `cursorStyle` | String | `"block"` | Cursor shape (`"block"`, `"underline"`, or `"bar"`) | | `cursorBlink` | Boolean | `true` | Enable or disable cursor blinking | | `theme` | Object | *See below* | Color palette overrides | --- ## Theme Options (`theme` object) | Key | Description | | :--- | :--- | | `background` | Terminal background color (e.g. `"#000000"`) | | `foreground` | Default text color (e.g. `"#e0e0e0"`) | | `cursor` | Cursor color (e.g. `"#ffffff"`) | | `black` – `white` | Standard 8 ANSI colors | | `brightBlack` – `brightWhite` | High-intensity 8 ANSI colors | --- ## Example `~/.config/webshell/config.json` ```json { "fontFamily": "'JetBrains Mono', 'Fira Code', monospace", "fontSize": 15, "cursorStyle": "block", "cursorBlink": true, "theme": { "background": "#000000", "foreground": "#e0e0e0", "cursor": "#00ff00", "green": "#00ff00", "cyan": "#00ffff" } } ``` ------ # Using Local Font Files ## Location Place your font file in `~/.config/webshell/` as one of the following: * `font.woff2` *(Create your own if needed! See below)* * `font.ttf` * `font.otf` The webshell will automatically encode and load your font file into the web browser upon login. ## Determining Font Family Name from a Font File fc-scan will return the font family name for you. It is possible to return multiple aliases so you can use the first one to simplify the process. ```bash $ fc-scan GoogleSansCodeNerdFontMono-SemiBoldItalic.ttf --format "%{family}\n" | cut -d, -f1 GoogleSansCode Nerd Font Mono ``` ## Use woff2 files instead of ttf or otf These are normal font files, but compressed so it will be nicer for web use: ```shell # one at a time for f in *.ttf *.otf; do woff2_compress "$f"; done # use all those cores! parallel woff2_compress ::: *.[ot]tf ``` On 181 of my fonts the size dropped from 381M to 220M. # KASM Container Build

Archived / Decommissioned (2026-10-19): Kasm Workspaces on ul has been decommissioned. This page is retained as an archival reference for container packaging and KasmVNC configuration.

This guide covers the technical workflow for packaging desktop GUI applications into custom Docker images for Kasm Workspaces (version **1.19.0+**), registering them in the database and web interface, configuring advanced features, and understanding display server architecture (X11 vs. Wayland). --- ## 1. Selecting a Base Image Kasm Workspaces runs applications inside containerized desktop environments streamed via KasmVNC. Always extend an official Kasm 1.19 base image to ensure compatibility with session management, audio forwarding, clipboard synchronization, and dynamic resizing: | Base Image | Base OS | Desktop Environment | Recommended Use Case | | :--- | :--- | :--- | :--- | | `kasmweb/terminal:1.19.0-rolling-weekly` | Ubuntu 24.04 (Noble) | XFCE4 + Terminal | GUI utilities, editors, lightweight tools | | `kasmweb/core-ubuntu-noble:1.19.0` | Ubuntu 24.04 (Noble) | Minimal XFCE4 Core | Clean custom application desktop builds | | `kasmweb/core-debian-bookworm:1.19.0` | Debian 12 (Bookworm) | Minimal XFCE4 Core | Debian-focused software packaging | | `kasmweb/desktop:1.19.0-rolling-weekly` | Ubuntu 24.04 (Noble) | Full XFCE4 + Apps | Full desktop environments | --- ## 2. Dockerfile Packaging Workflow ### Key Packaging Rules 1. **User Isolation**: Official Kasm base images run under non-root user `1000:1000` (`kasm-user`). Switch to `USER root` for package installation and explicitly switch back to `USER 1000` at the end of the Dockerfile. 2. **Ubuntu 24.04 Package Renaming (`libasound2t64`)**: In Ubuntu 24.04 (Noble), the 64-bit `time_t` transition renamed ALSA runtime packages (e.g. `libasound2` $\rightarrow$ `libasound2t64`). Use `libasound2t64` (or `libasound2*`) to prevent dependency resolution issues on 24.04 bases. 3. **Autostart Configuration**: To launch an application automatically when the workspace opens, copy an XFCE `.desktop` shortcut into `/home/kasm-user/.config/autostart/`. 4. **Custom Startup Scripts**: For applications requiring pre-launch environment variables or wrapper logic, place a script in `/dockerstartup/custom_startup.sh`. ### Example Dockerfile (`kasmweb/moebius:latest`) ```dockerfile FROM kasmweb/terminal:1.19.0-rolling-weekly USER root # Install required X11, GTK3, and audio runtime dependencies (Ubuntu 24.04 Noble) RUN apt-get update && \ apt-get install -y --no-install-recommends \ wget curl libnotify4 libnss3 libxss1 libxtst6 xdg-utils \ libsecret-1-0 libgbm1 libasound2t64 && \ wget -q -O /tmp/Moebius.deb https://github.com/blocktronics/moebius/releases/latest/download/Moebius.deb && \ apt-get install -y /tmp/Moebius.deb && \ rm -rf /var/lib/apt/lists/* /tmp/Moebius.deb && \ mkdir -p /home/kasm-user/Desktop /home/kasm-user/.config/autostart && \ cp /usr/share/applications/moebius.desktop /home/kasm-user/Desktop/moebius.desktop && \ cp /usr/share/applications/moebius.desktop /home/kasm-user/.config/autostart/moebius.desktop && \ chmod +x /home/kasm-user/Desktop/moebius.desktop && \ chown -R 1000:1000 /home/kasm-user/Desktop /home/kasm-user/.config USER 1000 ``` ### Building inside Standalone Kasm Containers When running standalone Kasm Workspaces (`lscr.io/linuxserver/kasm`), Docker images must be built inside the Kasm container engine (`kasm`): ```bash sudo docker exec -i kasm docker build -t kasmweb/moebius:latest - < Dockerfile ``` --- ## 3. Registering the Workspace in the Web Interface ### Automated Database Registration (`kasm_db`) In standalone Kasm deployments, workspaces can be registered directly in the PostgreSQL database (`kasm_db`). Each entry in the `images` table defines the container parameters, and `group_images` grants access to user groups. > [!IMPORTANT] > Unused foreign-key UUID columns (`session_banner_id`, `server_id`, `zone_id`, `filter_policy_id`, `server_pool_id`) must be explicitly set to `NULL`. Empty strings (`''`) cause PostgreSQL UUID syntax errors. ```sql -- 1. Insert Workspace definition into images table INSERT INTO images ( image_id, cores, description, docker_registry, docker_token, docker_user, image_src, enabled, available, friendly_name, hash, memory_bytes, name, session_banner_id, session_banner_force_disabled, x_res, y_res, run_config, volume_mappings, persistent_profile_path, restrict_to_network, restrict_network_names, allow_network_selection, restrict_to_server, server_id, exec_config, restrict_to_zone, zone_id, filter_policy_id, filter_policy_force_disabled, categories, require_gpu, gpu_count, gpu_graphics_acceleration, gpu_video_encoding, hidden, notes, override_egress_gateways, image_type, server_pool_id, link_url, cpu_allocation_method, uncompressed_size_bytes, launch_config, rdp_client_type, is_remote_app, remote_app_icon, remote_app_name, remote_app_program, remote_app_args, created_at, updated_at, enforce_workspace_persistence ) VALUES ( 'a1111111-60eb-4000-a000-0000000060eb', 2, 'Moebius ANSI and ASCII Art Editor full desktop client.', 'https://index.docker.io/v1/', NULL, NULL, 'img/thumbnails/terminal.png', true, true, 'Moebius', NULL, 2902458368, 'kasmweb/moebius:latest', NULL, false, 1280, 720, '{"hostname": "moebius"}', '{}', '/data/kasm_profiles/{username}/', false, '[]', false, false, NULL, '{}', false, NULL, NULL, false, '["Design"]', false, 0, '["MESA"]', '["SW"]', false, NULL, false, 'Container', NULL, NULL, 'Inherit', 1500000000, '{}', NULL, false, NULL, NULL, NULL, NULL, NOW(), NOW(), true ) ON CONFLICT (image_id) DO NOTHING; -- 2. Assign image to default All Users group (68d557ac-4cac-42cc-a9f3-1c7c853de0f3) INSERT INTO group_images (group_image_id, group_id, image_id) VALUES ( 'a2222222-60eb-4000-a000-0000000060eb', '68d557ac-4cac-42cc-a9f3-1c7c853de0f3', 'a1111111-60eb-4000-a000-0000000060eb' ) ON CONFLICT (group_image_id) DO NOTHING; ``` ### Manual UI Registration 1. Navigate to **Kasm Admin Panel** -> **Workspaces** -> **Add Workspace**. 2. **Workspace Type**: `Container` 3. **Friendly Name**: `Moebius` 4. **Docker Image**: `kasmweb/moebius:latest` 5. **Cores / Memory**: `2` / `2768` (MB) 6. **Persistent Profile Path**: `/data/kasm_profiles/{username}/` 7. Click **Save**. --- ## 4. Advanced Configuration Options When packaging complex software, evaluate these key operational options: ### 1. Persistent Profiles (`persistent_profile_path`) * **Purpose**: Retains user files, application settings, and browser profiles across session restarts. * **Configuration**: Set `persistent_profile_path` to `/data/kasm_profiles/{username}/`. * **Behavior**: When a session launches, Kasm bind-mounts the user's persistent directory to `/home/kasm-user/`. Set `enforce_workspace_persistence = true` to prevent unmanaged ephemeral sessions. ### 2. GPU Hardware Acceleration (AMD, Intel & NVIDIA) * **Purpose**: Offloads WebGL, 3D rendering, compute, and video stream encoding to host GPUs. * **Configuration for AMD & Intel GPUs**: * Pass `/dev/dri` and `/dev/kfd` device nodes in `run_config`: ```json { "devices": [ "/dev/dri:/dev/dri", "/dev/kfd:/dev/kfd" ], "group_add": ["render", "video"] } ``` * **Graphics Acceleration**: Set `gpu_graphics_acceleration: ["MESA"]` (RADV Vulkan / OpenGL). * **Hardware Video Encoding**: Set `gpu_video_encoding: ["VAAPI", "SW"]` for hardware H.264/HEVC/AV1 encoding. * Set `require_gpu = true` and `gpu_count = 1`. * **Configuration for NVIDIA GPUs**: * Ensure NVIDIA Container Toolkit is enabled on the host. * Set `gpu_graphics_acceleration: ["NVIDIA"]` and `gpu_video_encoding: ["NVENC", "SW"]`. ### 3. Bidirectional Audio & Microphone Forwarding * **Purpose**: Streams 48 kHz stereo audio output and enables low-latency WebRTC microphone input. * **Configuration**: * Kasm base images include ALSA and PulseAudio (`pulseaudio -D`) bridged to WebRTC / WebSocket audio codecs. Ensure packages install `libasound2t64` and `pulseaudio-utils`. * Enable microphone pass-through in `launch_config` if recording or voice chat is required: ```json { "allow_microphone": true } ``` ### 4. Gamepad & Controller Passthrough * **Purpose**: Passes browser HTML5 Gamepad API inputs directly into container `/dev/input/js*` virtual joystick devices. * **Configuration**: In `launch_config`, enable gamepad forwarding: ```json { "allow_gamepad": true } ``` ### 5. Network Isolation & Security (`restrict_to_network`) * **Purpose**: Restricts container egress or connects workspaces to internal backend VLANs/docker networks. * **Configuration**: Set `restrict_to_network = true` and specify target Docker network names in `restrict_network_names`: ```json ["infra_net", "isolated_workspaces"] ``` --- ## 5. Display Server Architecture: X11 vs. Wayland ### 1. Current KasmVNC Standard (X11) * **Architecture**: Official Kasm Workspaces desktop containers (`kasmweb/desktop`, `kasmweb/terminal`) operate on the **X11 display server protocol** using `Xvfb` / `Xorg` virtual framebuffers on display `:1`. * **Why X11 is Default**: * **Input Injection**: Keyboard and mouse event simulation across web browsers maps reliably to X11 input extensions (`libxtst6`). * **Dynamic Resolution Resizing**: Automatic desktop resizing when browser windows resize relies on `xrandr` / X11 Resize and Rotate extensions. * **Clipboard Synchronization**: Web-to-desktop clipboard sharing uses standard X11 selection buffers (`PRIMARY` and `CLIPBOARD`). ### 2. Wayland & Xwayland Support * **Can Kasm run Wayland?**: Yes. Modern KasmVNC releases support Wayland compositors (such as Sway or Weston) via **Xwayland** and virtual DRM/KMS backend framebuffers. * **Limitations in Wayland Mode**: * **Clipboard & Capture**: Pure Wayland restricts client-to-client screen capture and clipboard access by design. Applications running natively under Wayland require `xdg-desktop-portal-wlr` or compositor-specific protocols to share clipboards with KasmVNC. * **Application Compatibility**: Most packaged Linux GUI applications (Electron, GTK3, Qt5) run under X11 or Xwayland by default. For production packaging in Kasm Workspaces, **X11 remains the recommended display server** unless testing specialized Wayland-only compositors. # GPG Keys ### Current #### Luke https://keys.openpgp.org/search?q=lukeisgreat@gmail.com ``` pub rsa4096 2019-11-30 [SC] 9F9AC4567EDC8E2B86E5CB9475D6600BEF4E8E8F uid Luke Tidd sub rsa4096 2019-11-30 [E] ``` ``` -----BEGIN PGP PUBLIC KEY BLOCK----- Comment: 9F9A C456 7EDC 8E2B 86E5 CB94 75D6 600B EF4E 8E8F Comment: Luke Tidd xsFNBF3ifkcBEADU/riY8J3FRkZ+2qVvMQPHiwqEmZYCeSEtCsFxuXamEfRMhPEO bD2B6DXsHWXYw4g14Q0TF0A5OezSIXUt3au0d9AiSSFJtlXdUCAber5CGfQT44IU UgjBSww3FPTp9QJ7t7eUbNluablI4bt82piFJEgc0619bXmUMjZOyQps7xeFdRD0 FKNnSs7rgNgvZdTHwOvoqy0JKky/YKLgTU4jRieuVNT1mdR+IxfzbOzS7mJxa1Ur RX/FbrDp6+INMPFWoHOJaspc5jAcTtUAkjREyZmgn/Ki/KAe8A6agFseK7gKX3SS XIPhAgHfRbS9xr9vG5+QOlsksKqlEzGOxjgRGtloXBkk4H2oOJNgQfGccKl4ADel PZ3tXJGZlphZ/86kHFt1XNgr+/wsFpE5XCMiHiFo63t6jGEIQr2QgfJNZSTv1Lrr B1gt8lakaTHEOAdJafb0VAGFQPxNiiGYP6bfh9dXeKWOg0sdyBWCS0hHv29neEal rwrYTLNeEVDSxbMgw+dG53CKWZTkvRuPgGK+hcjPXOmByqFJzUJSHCHwpH073ret UgP8SGpkcC+7Sv3HP/qBe4UNAXrQnSXi2oPdHejaLXryCxEmLx+1mycGTkhkfXQ2 XFZsLWm8FSbXsnzELFPsB4gxZqOlL1k6zEk5gyW/KH+IkFulf/pRf9QQgQARAQAB zSFMdWtlIFRpZGQgPGx1a2Vpc2dyZWF0QGdtYWlsLmNvbT7CwZQEEwEIAD4CGwMF CwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQSfmsRWftyOK4bly5R11mAL706OjwUC aUVLcwUJDUQArAAKCRB11mAL706OjyQyD/9jlYzHboeOyabPqkscgGunDhBnNjrA uWSd2tyhP3/vRtv3vY8Vzp0VNdxXDD16oKmCZgj85qmpCl4ZhctZ1enyhDJKKeK/ JdwrdvZIL1atNufkzl3mko5wSISRK0dnDfOWhTOtsDFScIsQyBCOe41eU0eg3Jp/ 2SlEu3kQC6WIUXk5CeziHlPCMfoK3GS/i7caKwixelE2Ju35yDzTnrBIhN3Zykpj Fy/qn5M7TDlZZU+JzV6kKu7Q3rTnWkDJMcwv7a20P0chhZnonxKN/p5ktlgYMcWn KXkt9lGv3Kssu5CIAO9qIRTZMck0j2gxI1QCcFtRE+ZwPV2ToX12YunQAWbAhwEP scfPOYb08cNPCoZy7xvrRsY+hlxx47sc9iHvxHdqzf6B36QDyzEoJ6hSgHQYXiV5 A0dSLFDwJWxx4VfzYO9UDaKm6FwT0ocD/eZ+ZfPS4dcae0ftFBnj/S+Y7pxW3skU TywRZKsu+XQakL4chGtOi42Ii8sQMTWc50Qfc82ynnXDqlnD8s+VUcUmCYBhN4J+ QFuzIS4Dk7m8Zp6gSM8Vbr5ZYAsLkSm4ekAZcus8RBxZhODWo4l18G4EWxmLlJyz H/I5ztbnuJlUT20ZbKBjth8xT/X8SHJQPqtZ1sT1iPZGuoqEZpU81pIfLAi8Bey+ 5eon0j1/mFKnu8LBlAQTAQgAPgIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgBYh BJ+axFZ+3I4rhuXLlHXWYAvvTo6PBQJe4kecBQkC4PzVAAoJEHXWYAvvTo6PnvwP /3jg9RcHaDxnTgi/J8HaXE6AtLgZ05Qku+TW/nYE+gyL3/EuHk8TfwyqiuBv85oa oMHR9b3IP3fUZ5cdWVi/npEMm5/Tfv19DzYzVw84cFxGBDO2GnwshW9HnhBn56UG YDJ1J/8u9VutkC33JirAqd1DCAWn1KZOV6M9AFxvF5lSPXwMEXRiQr9A8rVjR8oc 6mTQKMluW6D918F6v+4MqtB7eamsMvym2ixNnyW3SuqPrvevEZRPmOv61g3e/vS8 00tBlcjAveE85rTgjZKdyXkUTD8FhPzqXW6pHx6AzhiqSS4FLwk9yLtMjC85RHvQ Texoc5WS9SwVPXhKpEc4QXIS66L5jf4mJNqdyUAeAqrbbRqcCQsSlMe3SU+9CMS0 coa4piedhO6wirFnH8sWxN2CQVCRIwrZqZDlNBEaNQinyBWmr0c+VN0EQwdv4p2c ImYpb1GhUE/FsuW2TC6EJppITz3ipTBvD1MYBdpho1TzizAQympMKmTJMrdup2FM zfVTs7q8pXDQIQCwZjYismLUIhRVLp0YJldm/uTg/UhN/R7JrQ9bmbAKjcfXcXIw 1t4ORsApEO+GkAk6YTO759CqrZw0Cg5OoaVZprg83yT+77Y9GE+OEkpzbj8n8bLo C81vgMzDkMvkk/n2klm4O+lKCd3QyK00sRhSShmIqtv9zsFNBF3ifkcBEADNeG+l 45CuegVep4ti1q6J3AAj1kDeZfNYF5IMCNmgC2mLoxQzxGUd2aPdr1zOFqo2B2TP m/FvXooZMa4Iocue+/BIpxUqJNGqCEv6B3/EzUQzomOSiy8yxm6fvMpBhS32+KKq AdhRyq32dUBdEwy20+7EcYi2sFGdJXaqf0NAM5VXp1UHx/4Q0SCdezswi8WU1a03 cDYfAc/GF1ZPucYjCa9yB3stbBTHc3VBgcbcWmpRB2Lv063e3NswLyqrKMWBxYZq HK+h/deYMRh3fmmL4OeUmzT2e1CvJxBEl9OsTo1Q1NAnByRhee+3AwLwLsA3a4jz NDz4igGhLVFS0yEWc7xQtYePfQi/0uXLezujAxrNT7ugAKyPzxtIG3Qto2O0Q+zP 5BzOFZ8XyCuRCPojd2pN6u2/Z2dQpUtwH+fbQ8c5Xui356asqBhpBCjH/nV360bd WgKNpmgnSbIGkqoUr2LjXZFA2Qt+N+HvUFl+zEm11OwFuuXvu6waaSiCkkevuA+E Ec/hsVujUkj437vF2S3hQsNcmuHtfEg4ld/d1Be4lu86tUAhsTjeyQYKhHBP5dKj qkUJwrkVP+f0zRVigedof5LG38XGxcv2KOmW2y9Pg+BIncPLHYK9pg4vJ1qvvTp2 nThVOfT7tKpiOhkbeQOvhI+DJLCedvXkcYperwARAQABwsF8BBgBCAAmAhsMFiEE n5rEVn7cjiuG5cuUddZgC+9Ojo8FAmlFS3MFCQ1EAKwACgkQddZgC+9Ojo/uiw/+ I52aAPI7S2AFX8BLD6rHCzzqNKx822e36yK3AUny6UUv51ou5YeDpO3H+O8U8bZX +L4GRT1VDUBR/B75AlBdZXtu9RTQ5AfXJYgoK8j796jH3cNTgN5iOrREjYCrFrqc V0etLOrCVXvhsBbOuznLgDoN+n3tQnme0zghB+RINoWZaOji9Ut3FaOWxzcoB6dC Swtg1kZqxhWN1VKtwSbNGX1Q9uqrtR6AoRbtG0CX8S9CDFEidDTonngspY5ICj6i Pq6Wf9DgOqT+fpwDGjV9gjM6etY6TV1vA1pLfwf7JB04faFN4KxnN7UHPvd6R2qa NJ7QfOsGHsTh5sugV2IOAuazl/xUhn3iyftSQ1F2Q+9vRhWW0AgOhWEQzVuFYq8O IB4BnzT5rt6rrsLyKHVOgc6T+3lVP8o3c2s8TXCcbYnbtiy6dlAwrLfSpxcAPXWi K4M4rS5DoxCkgt/sTdMT9BqYVCJjR7ud8Ffn76OfIM6EqYwENAPQqZPHgFE0fzOR kV9thvsvMne/a1zr6Ys1RXK3Yfv+EyxuLvH4houZ0vHfMgzrxOop5/X7u+2TcPaX XWG+U8t+SWngnYDTq93Nwg82rNazKftlYFR8TuMCYedJc+BC1+guhGkqxImA/GlZ 0wBvFkvp73MrOoPaVxHgK8f/TGZK9tggNwIo5vTOeYvCwXwEGAEIACYCGwwWIQSf msRWftyOK4bly5R11mAL706OjwUCXuJH3QUJAuD9FgAKCRB11mAL706Oj2YQD/9q Jzkn1G/palkUfAA6lFisfSY7Ej0INmxJZpI/RHOiOipcZvh2PWzAfTh1w0yLtLw+ D+vEedRpGVKvqS+CjUAKyV6Z3uP48cy93FPnLi/kiDq2eCkB+GZYTNjP2IDsLR5Z YFj/qK5olpmNff3BOMIRR55QIWT0ETgI2oG1KSJ/iqQYBxpQQ51gyWqroP8XaX0W 0Dcii4pscXL/Mj6okDO5QlN7o7pQOGShkWcsqmIbGJ3DOdArGZpi3Cm6zdBarsKg mgKE8XIVZ3XGynUh0K74TIZMGqErX3/RX6Tg8qb2T0vnSwagN8hpI5EyxGquwD0G FUyXtvW6xi2jjCQUzfE59QCpkxOL4vIsf3fRmP9UtlSgf8MhDaTR5Fjqs8C3YoHZ lAK16NmDP+rjkxdlhiPsKRS4I3AOStbtORMPB4ig4MRzRNFUtxZJgFN0yEkf42QL QOBMude+jIsi+zrwI1NqimpqTyKw248s9n+5Fir5rGYj9y3cfBFCqqfz5JqjGknf UiFXaG7MrWFz9S/Dy8for+IFUjHrs22TMGJGZL89gblbGHFqQIjFxTsj/f1ehijk G/WOWDXRMukLUBLk/R3N/sQz9lfWIGi4XdMHk2UU32OQKZU2BgB/xCNQ7U52LnlI Ga1nOpOXmBUUtXKfOgJmGU17aKNMa5SGS72XN4kWPA== =0mJb -----END PGP PUBLIC KEY BLOCK----- ``` ---- ### Obsolete #### Luke https://pgp.mit.edu/pks/lookup?op=get&search=0xAD7D030C881A4A29 ``` Type bits/keyID Date User ID pub 1024D/881A4A29 2004-12-26 *** KEY REVOKED *** [not verified] Luke Tidd Public Key Server -- Get "0xad7d030c881a4a29" ``` ``` -----BEGIN PGP PUBLIC KEY BLOCK----- Version: SKS 1.1.6 Comment: Hostname: pgp.mit.edu mQGiBEHOIAIRBACJI1tNSFOEJ0F2xPOpbtnaxqiwoseGKK6oNAFN5k8VntQVQrir5n9A8ocp uFPsB47dsHyBn7X8bFuR1Umz7tgO09lrvfulCvGRLpmSkm2E1VbOEj9IBekbng8ByzjGVLOQ RCAWoqxzdGLSxcTUGutTNRaA0RJms1CJeqQze2PYNwCggeJa5orn6FhZZcmVaaRx4iIxKkcD /1ICPOhXlO1Q0Q1zWQYjuMJIstA7pGeb+wMmN2JEHgra+ClpBU2ZafkHBz+SPR1K85F8ufZY deXcvc3gruBtYrKdlucRbelMT0gMWOtw/2f+mX5n0zvbzJQXIJN+rRJfqs9VVN5emRisvAWy 60Gja/pGBTzkwhM1+4Qt7S1iyQ4qA/9diGL6Ft0oph0zMYZSKAYIArkyEB19UrwX1mhenxsw f7WXrOoW/Vq3y6TBldtfGTclBqpMwgpbfOr5HtYoer007izKIQVzyI2UfEPkCdSc7mlvy3RY rejVMFwgwZ4rUy5miAqmcPYoz8b2ECE4KCt0M0eW0vfk+vMp7Jb4f4N0xYhJBCARAgAJBQJB 7yRNAh0AAAoJEK19AwyIGkopkNEAmgKSZ+rREOPVofe3aNP3KTrxEEh8AJ0UnNPfYJUY8Xk8 GspO0SfenryshrQdTHVrZSBUaWRkIDxjNjBAYmVsbHNvdXRoLm5ldD6IXgQTEQIAHgUCQc4g AgIbAwYLCQgHAwIDFQIDAxYCAQIeAQIXgAAKCRCtfQMMiBpKKbYhAJ9kqM6c2GGCosftce/Y nNDyfkqYegCfa/7irE2R2MKtm8x9O/v5qSMALC65BA0EQc4hbBAQAIHmEr8VvBJ838gTAv++ 8snvnrckpTfLenSUdG5IbbcJrJyWEwMC3V3i669RMpsDO0t8Z9mSa8B12A8pN3g1DqHJAJv6 WRnH3FOOkcI54ypBTUJBYhp8FYOxFEEi7B0a8c4w3/Zic3mXdnvbkHa3n6OnJ11ugFrIVJUz wmynfk69QmaHnY9st0NC/TqRLmcq2iWEbdk76GBA0stiitjAIYwSEkRXZzMYyq3QwJM0FLfg U2xuves+PjfnXbxiEi9l13FSXerhgZWxVkdGIp+3fmuBqa1dUrJLsMLM8IalEKiAue9vFqgH cF/ZVBKpCQnyAZLih9DBzvRF8kiqTgayyjIN+oF7kYIZuwbIBjP1PwFF/ILwKUMIG5uTdyJ1 C0yQj3HC/R7qjB1QRP7pQVvUrjgI6W7oT0t0Cy8thchW5PJmR48zQIepOQodc/EKsLL2l2ii BdTcXdTBsJbOzelMIMnOJaedOqLEZvNFFCIGmoPumm+mOVXAzzOY2R48n4oTE1xwkk3clZ6G 2Lhk0yvdGYsW6+TrhXx53O+JSzOqz4C96SKclIhrVmzJAi+gmIX1giiWnW3jwgrAFFvEoLFa 4egF//beOPVaBFMHe6bXabMLsG4dst+f3O6h1tJk79VTurTHLb4JVrSg7nOJ+Ye03BXFpGzG bftEIwJR5cHn6lurAAMFD/wMQsd/3DgkrlO5diVnBvVuLVrCewKj49aP9j69zsc18BLFcLKu I7/JJBOhiyvAeYiWXF1Z2rdzkHA32eolkNoRyN2zBh+17U9BZxXtiLlqf3y+YmpqzJexn4t1 yXLh23EcHIdSwcQtAH2VRFf+MQHlQY9Z7omQRGILyS6dvBLz0iHHR/XdtMk+6dxlQ+7a/r6s rPAsct5nKa0D8UhYeUzE7xSp7RvH9fBQBM+XRb5Zk/gF7acj5lklO0a4CdLHcDXlxGyLrdyg EZo4HdvK4E2xYm/bnLRMh8uFACWlKOwyi+dpd20R2l6V+dSEvgsRR+WR9PGCel+QlL1tjV6p WKnGvK3WbGPNvNanto1wn0HlbtMBwKUgy3+DzAkpjdjrrJy1tgSnfy+XBw5PkvICx16Za6hT cTcyTxmpwK13UZ9M6+oXEDOaFCF33U+HrR5EL9VMkRpkwJ1qzIwjCjsdlkvEQmN9eZNnakr6 9SZXDVHcQDE3OqmzycutBMmyVRJupcnE+GEHe4eWvEcf+4C9lEcyMxSdI/Pn71UWJvODqOVL wma+4Md83L2JBvUHRSgLuI8AY9cfx9R7EoBa8gEtlRP+CqWpStxl597ZR2qJYFk/gtL4hh3N t0dotxone2fozW1xn7cJnz8qhpchABW1GtpWntawuzwNpCVAT87TeUbbz4hJBBgRAgAJBQJB ziFsAhsMAAoJEK19AwyIGkopb4wAniDf4xRYStRvlozVTmUnqYl2hj+PAJ4tXfl2456EOvqp 5/XNFKNEpAmuaA== =z7x2 -----END PGP PUBLIC KEY BLOCK----- ``` # CrowdSec & OpenBSD PF Perimeter Security Setup ## 1. Overview & Architectural Concept This document outlines the distributed intrusion detection and perimeter enforcement setup across the local infrastructure (`drheck.dev`). ``` [ Public Internet (WAN) ] │ ▼ ┌─────────────────────────────────┐ │ danknasty (OpenBSD Router) │ │ ─────────────────────────────── │ │ • OpenBSD PF Kernel Firewall │ │ • Kernel SSH Rate Limiter │ │ • Syslog SSH Forwarder (UDP 514)│ │ • CrowdSec PF Firewall Bouncer │ └────────────────┬────────────────┘ │ Drops banned IPs at WAN edge │ Stream bans via IPv6 LAN ([fd77:...:c70d]:8080) ▼ ┌─────────────────────────────────┐ │ ul (Arch Linux Server) │ │ ─────────────────────────────── │ │ • Dedicated IPv6: ...:c70d/64 │ │ • Nginx Web Reverse Proxy │ │ • Remote Syslog Ingestion (514) │ │ • CrowdSec LAPI Decision Engine │ │ • 53-Hour Ban Policy Engine │ │ • Public Active Enriched JSON │ │ • Append-Only NDJSON Stream │ └─────────────────────────────────┘ ``` ### Design Philosophy & Conventions 1. **Dedicated Service IPv6 Address (`ul`):** * Per local network convention (`20-wired.network`), CrowdSec LAPI listens on a dedicated IPv6 ULA: `fd77:7777:5010:0:1337:f1ac:d00d:c70d` on port `8080` (LAPI) and UDP port `514` (Syslog Ingestion). 2. **Offload All Heavy Detection to Powerful Hardware (`ul`):** * All heavy log analysis, regex pattern matching, SQLite/LAPI database operations, and scenario evaluations run on the Arch Linux server (`ul`). * **Nginx Web Probes & Direct IP Hits:** Parses Nginx access logs to detect scanner bots (e.g., `.env` harvesting, WordPress probes, webshell attempts, open proxy CONNECT attempts, and direct IP connection attempts). * **Router SSH Probes, Invalid Users & Root Attempts:** Receives OpenBSD SSH authentication logs (`/var/log/authlog`) forwarded over UDP IPv6 syslog from `danknasty` to detect SSH brute-force attempts, instant invalid username bans, and instant root attempt bans. * **53-Hour Ban Policy:** Issues a global **53-hour ban** for any detected web probe, direct IP scan, or SSH brute-force/invalid user/root attempt. 3. **Perimeter Enforcement at Gateway (`danknasty`):** * The OpenBSD router (`danknasty`) runs only the lightweight CrowdSec PF Bouncer (`crowdsec_firewall_bouncer`) and native `syslogd`. * Banned IPs are pulled from `ul` and injected into OpenBSD PF's kernel `` table. Banned packets are dropped at the network interface before entering the local network. 4. **Zero-Overhead Edge SSH Protection (`danknasty`):** * OpenBSD PF tracks TCP connection rates directly in kernel state space (`max-src-conn-rate 3/30`). * Any bot opening more than 3 SSH connections in 30 seconds is added to the OpenBSD PF `` table and dropped with **0% CPU and ~0 MB RAM usage**. 5. **LAN Subnet & Dynamic IPv6 Whitelisting:** * Private subnets (`10.0.0.0/24`, `fd77:7777:5010::/64`), loopbacks (`127.0.0.1/32`, `::1/128`), and the dynamic public ISP IPv6 prefix (auto-detected via `/usr/local/bin/crowdsec-update-ipv6-whitelist`) are whitelisted on `ul` so internal LAN users and SSH typos can never trigger bans. 6. **Rich Context & OSINT Enriched Export (`ul`):** * Every banned IP is automatically enriched with Reverse DNS (PTR), Country, Region, City, Coordinates (Lat/Lon), Timezone, ISP, Organization, and Autonomous System (ASN) details. * **Active Array File:** `/var/log/crowdsec-bans-public.json` (`chmod 644`). * **Append-Only NDJSON Stream:** `/var/log/crowdsec-bans.json` (`chmod 644`). --- ### Detection Latency & Pipeline Timeline | Protection Layer | Target Threat | Latency | Mechanism / Action | | :--- | :--- | :--- | :--- | | **Layer 1: OpenBSD PF Kernel Limiter** | SSH Connection Spikes | **Instant (Nanoseconds / ~0s)** | In-kernel TCP state tracking (`max-src-conn-rate 3/30`) instantly moves abusive IPs into `` table. | | **Layer 2: CrowdSec Distributed Ban** | Web Probes, SSH Invalid Users & Root Attempts | **~5 seconds (Max 10s)** | Syslog/Nginx logs stream to `ul`. `ul` triggers 53h ban. `danknasty` bouncer pulls decision and injects IP into OpenBSD PF ``. | #### CrowdSec Distributed Pipeline Timeline: 1. **Event Generation:** `sshd-session` or Nginx emits log event (`< 1 ms`). 2. **Log Forwarding:** UDP IPv6 syslog or Nginx log tailer streams log to `ul` (`< 1 ms`). 3. **Scenario Trigger & Ban Creation:** CrowdSec parses line, leaks scenario bucket, and writes 53h ban to LAPI database (`~20–50 ms`). 4. **Router Bouncer Polling:** `danknasty` bouncer polls `ul` LAPI over IPv6 LAN (`update_frequency: 10s`, average `~5 seconds`). 5. **Kernel Table Injection:** Bouncer executes `pfctl -t crowdsec-blacklists -T add ` (`~10 ms`). --- ## 2. Implementation & Configuration Details ### A. Central Decision Engine on `ul` * **Interface Configuration (`/etc/systemd/network/20-wired.network`):** ```ini # crowdsec Address=fd77:7777:5010::1337:f1ac:d00d:c70d/64 ``` * **Listen URI (`/etc/crowdsec/config.yaml`):** Bound to dedicated IPv6 ULA `[fd77:7777:5010:0:1337:f1ac:d00d:c70d]:8080`: ```yaml api: server: listen_uri: "[fd77:7777:5010:0:1337:f1ac:d00d:c70d]:8080" ``` * **Nginx Log Acquisition (`/etc/crowdsec/acquis.d/nginx.yaml`):** ```yaml filenames: - /var/log/nginx/*.log - /var/log/nginx/access.log labels: type: nginx ``` * **Router SSH Syslog Ingestion (`/etc/crowdsec/acquis.d/danknasty-ssh.yaml`):** ```yaml source: syslog listen_addr: "fd77:7777:5010:0:1337:f1ac:d00d:c70d" listen_port: 514 labels: type: syslog ``` * **LAN & Dynamic IPv6 Whitelist Override (`/etc/crowdsec/parsers/s02-override/00-lan-whitelist.yaml`):** ```yaml name: custom/lan-whitelist description: "Whitelist internal LAN subnets and dynamic public IPv6 prefix" filter: "evt.Meta.source_ip != nil" whitelist: reason: "Exempt internal LAN subnets and dynamic public IPv6 prefix" cidr: - "10.0.0.0/24" - "fd77:7777:5010::/64" - "127.0.0.1/32" - "::1/128" - "2603:3001:626:9d80::/64" # Auto-updated by crowdsec-update-ipv6-whitelist ``` * **Custom Honeypot Scenario (`/etc/crowdsec/scenarios/honeypot-endpoints.yaml`):** ```yaml type: leaky name: custom/honeypot-endpoints description: "Instant trigger on bot/scanner endpoints" filter: | evt.Parsed.request in ["/.env", "/cgi-bin/authLogin.cgi", "/IndoSec.php", "/alfa.php", "/luxe.php", "/BDKR28WP.php"] || evt.Parsed.request matches "(?i)^/(wp-admin|wp-login|wp-includes|\\.env|cgi-bin|alfa\\.php|1\\.php|222\\.php|8\\.php|shell\\.php)" capacity: 1 leakspeed: 1h groupby: evt.Meta.source_ip blackhole: 1m labels: service: nginx type: honeypot_scan remediation: true ``` * **Custom SSH Invalid User Scenario (`/etc/crowdsec/scenarios/ssh-invalid-user.yaml`):** Instant 53-hour ban for any attempt to log in using an invalid username: ```yaml type: leaky name: custom/ssh-invalid-user description: "Instant 53h ban for any SSH login attempt using an invalid user" filter: | evt.Meta.log_type == "ssh_failed-auth" && ( evt.Parsed.sshd_invalid_user != "" || evt.Meta.target_user != "" ) capacity: 1 leakspeed: 1h groupby: evt.Meta.source_ip blackhole: 1m labels: service: ssh type: bruteforce remediation: true ``` * **Custom SSH Root Attempt Scenario (`/etc/crowdsec/scenarios/ssh-root-login-attempt.yaml`):** Instant 53-hour ban for any attempt targeting the `root` account: ```yaml type: leaky name: custom/ssh-root-login-attempt description: "Instant 53h ban for any SSH login attempt targeting user root" filter: | evt.Meta.log_type == "ssh_failed-auth" && ( evt.Meta.target_user == "root" || evt.Parsed.sshd_invalid_user == "root" || evt.Parsed.sshd_user == "root" ) capacity: 1 leakspeed: 1h groupby: evt.Meta.source_ip blackhole: 1m labels: service: ssh type: bruteforce remediation: true ``` * **Direct IP Access Scenario (`/etc/crowdsec/scenarios/direct-ip-access.yaml`):** Instant 53-hour ban for any HTTP/HTTPS connection made directly by IP address or invalid host header: ```yaml type: leaky name: custom/direct-ip-access description: "Instant 53h ban for any HTTP/HTTPS connection made directly to IP address or non-domain host" filter: | evt.Meta.log_type == "http_access-log" && ( evt.Meta.http_status in ["444", "400", "403"] || evt.Parsed.status in ["444", "400", "403"] || evt.Parsed.target_fqdn matches "^([0-9]{1,3}\\.){3}[0-9]{1,3}$" || evt.Parsed.http_host matches "^([0-9]{1,3}\\.){3}[0-9]{1,3}$" ) capacity: 1 leakspeed: 1h groupby: evt.Meta.source_ip blackhole: 1m labels: service: nginx type: direct_ip_scan remediation: true ``` * **53-Hour Ban Policy (`/etc/crowdsec/profiles.yaml`):** ```yaml name: default_ip_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 53h notifications: - file_default on_success: break --- name: default_range_remediation filters: - Alert.Remediation == true && Alert.GetScope() == "Range" decisions: - type: ban duration: 53h notifications: - file_default on_success: break ``` * **File Notification Plugin (`/etc/crowdsec/notifications/file.yaml`):** Appends real-time events upon trigger to `/var/log/crowdsec-notifications.log`: ```yaml type: file name: file_default log_level: info group_wait: 0s group_threshold: 1 format: | {{range . -}} {{$alert := . -}} {{range .Decisions -}} {"timestamp":"{{$alert.CreatedAt}}","ip":"{{.Value}}","action":"{{.Action}}","duration":"{{.Duration}}","reason":"{{.Scenario}}","country":"{{$alert.Source.Cn}}","as_name":"{{$alert.Source.AsName}}"} {{end -}} {{end -}} log_path: "/var/log/crowdsec-notifications.log" rotate: enabled: false ``` * **Host Firewall Rules (`/etc/nftables.conf`):** ```nftables ip6 saddr fd77:7777:5010::/64 ip6 daddr fd77:7777:5010:0:1337:f1ac:d00d:c70d tcp dport 8080 accept comment "Allow CrowdSec LAPI from local IPv6 LAN" ip6 saddr fd77:7777:5010::/64 ip6 daddr fd77:7777:5010:0:1337:f1ac:d00d:c70d udp dport 514 accept comment "Allow CrowdSec Syslog Ingestion from local IPv6 LAN" ``` * **Automated OSINT & Threat Pipeline (`/usr/local/bin/crowdsec-export-bans`):** Runs every 60 seconds (`crowdsec-export-bans.timer`): 1. Updates dynamic public IPv6 whitelist (`/usr/local/bin/crowdsec-update-ipv6-whitelist`). 2. Ingests all router SSH auth failures in real time via native UDP IPv6 syslog (`auth.info;authpriv.info` -> port 514), triggering instant sub-second bans (`custom/ssh-invalid-user` via `type: trigger`). 3. Updates `/var/log/crowdsec-bans-public.json` (active JSON array) and appends newly discovered ban events with full GeoIP/ASN/rDNS metadata to `/var/log/crowdsec-bans.json` (`chmod 644`). --- ### B. Perimeter Enforcement & Syslog Forwarding on `danknasty` (Router) * **Syslog Forwarding Configuration (`/etc/syslog.conf`):** ```syslog auth.info;authpriv.info @[fd77:7777:5010:0:1337:f1ac:d00d:c70d]:514 ``` *Note:* OpenBSD `syslogd(8)` requires the `:514` port suffix when enclosing IPv6 addresses in square brackets `@[...]`. Without `:514`, `syslogd` silently skips UDP forwarding. * **Bouncer Configuration (`/etc/crowdsec-firewall-bouncer/crowdsec-firewall-bouncer.yaml`):** ```yaml mode: pf retry_initial_connect: true update_frequency: 10s api_url: http://[fd77:7777:5010:0:1337:f1ac:d00d:c70d]:8080/ api_key: deny_action: DROP blacklists_ipv4: crowdsec-blacklists blacklists_ipv6: crowdsec6-blacklists pf: anchor_name: "" ``` * **Router Firewall Rules (`/etc/pf.conf`):** ```pf # CrowdSec Blacklist Tables table persist table persist block drop quick from block drop quick from # Table for SSH Brute-force Auto-Blocking table persist block drop quick from # SSH rules with stateful connection tracking pass in quick on $ext proto tcp to self port ssh \ flags S/SA keep state \ (max-src-conn 10, max-src-conn-rate 3/30, overload flush global) ``` --- ## 3. Interaction, Operations & Data Format ### A. Enriched Public JSON Data Schema The exported JSON files (`/var/log/crowdsec-bans-public.json` and `/var/log/crowdsec-bans.json`) contain rich OSINT, network, and threat context for each banned IP: ```json { "decision_id": 4, "alert_id": 4, "ip": "119.30.119.3", "ip_version": "IPv4", "origin": "crowdsec", "reason": "crowdsecurity/netgear_rce", "action": "ban", "duration": "52h59m18s", "country": "Pakistan", "country_code": "PK", "region": "Punjab", "city": "Sialkot", "latitude": 32.426, "longitude": 74.3644, "timezone": "Asia/Karachi", "isp": "Mobilink GSM, Pakistan Mobile Communication Ltd.", "org": "Linkdotnet Telecom Limited", "as_name": "AS58470 IX Peering for Mobilink and Link Direct International.", "rdns": "static-host119-30-119-3.link.net.pk", "target_uri": "/setup.cgi?next_file=netgear.cfg&todo=syscmd&cmd=rm+-rf+/tmp/*...", "user_agent": "-", "method": "GET", "status_code": "301", "created_at": "2026-08-05T02:44:21Z" } ``` --- ### B. Viewing Active System State & OpenBSD PF Kernel Tables #### 1. View Active Bans on `ul` (CrowdSec LAPI) ```bash # View all active decisions (untruncated): sudo cscli decisions list -a # Query if a specific IP is banned in CrowdSec: sudo cscli decisions list --ip ``` #### 2. View Active Enriched JSON Array ```bash cat /var/log/crowdsec-bans-public.json ``` #### 3. Tail the Append-Only Ban Stream (NDJSON) ```bash tail -f /var/log/crowdsec-bans.json ``` #### 4. View Active Bouncers & Connection Status ```bash sudo cscli bouncers list ``` #### 5. View Detection Metrics & Statistics Shows lines read via Nginx logs and remote router syslog: ```bash sudo cscli metrics ``` #### 6. View Active OpenBSD PF Kernel Tables on Router (`danknasty`) ```bash # List all blocked IPs in CrowdSec's PF Radix Tree table: SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t crowdsec-blacklists -T show" # View packet & byte drop statistics for the CrowdSec PF table: SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -v -t crowdsec-blacklists -T show" # Test if a specific IP is currently blocked in the OpenBSD PF kernel table: SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t crowdsec-blacklists -T test " # View OpenBSD SSH kernel brute-force bans: SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t bruteforce -T show" ``` --- ### C. Manual Interactivity & Ban Management #### 1. Manually Ban an IP for 53 Hours ```bash sudo cscli decisions add --ip --duration 53h --reason "Manual administrative ban" ``` #### 2. Remove a Ban (Unban an IP) ```bash sudo cscli decisions delete --ip ``` #### 3. Force Immediate Synchronization & OSINT Enrichment ```bash sudo /usr/local/bin/crowdsec-export-bans ``` #### 4. Inspect Recent Detection Alerts ```bash sudo cscli alerts list sudo cscli alerts inspect ``` #### 5. Flush Banned IP Tables on the Router (Emergency Reset) ```bash SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t crowdsec-blacklists -T flush" ``` --- ### D. Whitelisting Local Trusted Networks Internal LAN subnets and your dynamic public IPv6 prefix are protected from accidental bans via `/etc/crowdsec/parsers/s02-override/00-lan-whitelist.yaml`: ```yaml name: custom/lan-whitelist description: "Whitelist internal LAN subnets from all CrowdSec scenarios" filter: "evt.Meta.source_ip != nil" whitelist: reason: "Exempt internal LAN subnets" cidr: - "10.0.0.0/24" - "fd77:7777:5010::/64" - "127.0.0.1/32" - "::1/128" - "2603:3001:626:9d80::/64" ``` After modifying whitelists or scenarios, reload CrowdSec: ```bash sudo systemctl reload crowdsec ``` ### E. Adding More Scanner Endpoints `/etc/crowdsec/scenarios/honeypot-endpoints.yaml` Current Configuration: ``` type: leaky name: custom/honeypot-endpoints description: "Instant trigger on bot/scanner endpoints" filter: | evt.Parsed.request in ["/.env", "/cgi-bin/authLogin.cgi", "/IndoSec.php", "/alfa.php", "/luxe.php", "/BDKR28WP.php"] || evt.Parsed.request matches "(?i)^/(wp-admin|wp-login|wp-includes|\\.env|cgi-bin|alfa\\.php|1\\.php|222\\.php|8\\.php|shell\\.php)" capacity: 1 leakspeed: 1h groupby: evt.Meta.source_ip blackhole: 1m labels: service: nginx type: honeypot_scan remediation: true ``` How to Add More Scanner Endpoints 1. Add new exact paths to the evt 1. Parsed.request in [...] list (e.g. "/setup.php", "/config.json"), or append regex patterns to evt.Parsed.request matches "(?i)...". 1. Reload Crowd ``` $ sudo cscli scenarios reload $ sudo systemctl reload crowdsec ``` # Service List

Runtime Architecture Note: All 30 containerized applications on ul run under rootless Podman Quadlets (docker-data, UID 1100) fronted by Nginx TLS + Authentik SSO on dedicated eth77 ULA IPv6 addresses. See podman for full operational reference.

--- ### [Anchor](https://keep.drheck.dev) Anchor is a fast, self-hosted collaborative note-taking and checklist application integrated with Authentik OIDC Single Sign-On and backed by PostgreSQL on `ul`. --- ### [AudioMass](https://audiomass.drheck.dev) AudioMass is a full-featured, open-source web-based audio waveform editor and recorder. Running entirely inside the browser using Web Audio and WebAssembly, it provides users with a desktop-class digital audio editing workspace without requiring local software installation. Users can load audio files, record microphone input, view real-time visual spectrums, cut, copy, paste, trim, and mix audio tracks. The tool includes an extensive suite of built-in DSP effects and processing tools, such as equalizers, gain adjusters, compressors, pitch shifters, delay, reverb, and noise gates. It allows quick audio editing, podcast clips creation, or sound snippet modifications with instant rendering and export in WAV or MP3 formats directly from the browser. [![audiomass main view](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260808-19h14m13s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260808-19h14m13s-grim.png) --- ### [Authentik Identity Provider](https://auth.drheck.dev) Authentik is the central OpenID Connect (OIDC), ForwardAuth proxy, and LDAPS (`ldap.drheck.dev:636`) Single Sign-On identity provider powering unified authentication across web apps and Linux workstations on `drheck.dev`. --- ### [BentoPDF](https://bentopdf.drheck.dev) BentoPDF is a privacy-preserving, browser-based PDF manipulation toolkit gated behind Authentik ForwardAuth for merging, splitting, compressing, rotating, watermarking, and converting PDF documents with zero external cloud uploads. --- ### [BookStack](https://wiki.drheck.dev) BookStack is a wiki platform designed to make storing and organizing information simple and structured. BookStack breaks content down into an intuitive real-world book hierarchy: Shelves, Books, Chapters, and Pages. This visual structure ensures that technical documentation, standard operating procedures, personal notes, and knowledge bases remain easy to search, navigate, and maintain. BookStack offers both WYSIWYG rich text and Markdown editors, built-in version control with page revisions, image management, cross-linking, and granular role-based access controls. Single Sign-On support: yes, with SSO can visit the site and be signed in directly. [![bookstack](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260808-19h19m36s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260808-19h19m36s-grim.png) --- ### [Camera Hub (`cam`)](https://cam.drheck.dev) DrHeck Camera Hub provides live low-latency video and MP3 audio streaming, two-way RTSP ONVIF backchannel talk, YOLOv8n + hardware AI event capture timelines, 5X optical zoom controls, floodlight/IR controls, and a Google TV 10-foot UI for the PoE Reolink `RLC-811A` cameras, restricted to the `camera_access` group. --- ### [Console Games (Arcade Launcher)](https://arcade.drheck.dev/content/launcher.html) Console Games is a web-based retro gaming launcher and front-end interface built on top of webЯcade. It provides end users with an interactive, TV- and controller-friendly library for browsing and launching classic console video games directly in modern web browsers using WebAssembly emulation cores. Users can connect gamepad controllers via USB or Bluetooth, save game states, customize controller mappings, and stream classic retro titles across desktop computers, mobile devices, and smart TVs without installing native emulators or software dependencies. [![arcade](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260808-19h21m12s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260808-19h21m12s-grim.png) --- ### [Degoog Search](https://search.drheck.dev) Degoog is a self-hosted privacy metasearch engine at `https://search.drheck.dev` (`https://s.drheck.dev`) backed by host Valkey caching without tracking or ad profiling. --- ### [Demucs Stem Splitter](https://demucs.drheck.dev) Demucs is a GPU-accelerated music source separation studio running on the AMD Radeon AI PRO R9700 (`ROCm`), splitting songs into isolated vocals, drums, bass, guitar, piano, and accompaniment stems. --- ### [Drawpile](https://drawpile.drheck.dev) Drawpile is a free, open-source collaborative drawing program that allows multiple artists to draw on the same canvas simultaneously over the internet. The server component hosted on drheck.dev manages shared session state, brush strokes, layer ordering, and user permissions for real-time multiplayer digital art sessions. Users can host private or public drawing rooms, sketch together in real time, record and playback drawing sessions, lock specific layers to prevent accidental edits, and chat with co-creators. Authentik integration ensures authenticated users can easily jump into ongoing canvas sessions or manage drawing servers. This is just the server, create a room and connect with your client. --- ### [Excalidraw](https://excalidraw.drheck.dev) Excalidraw is a virtual, hand-drawn style collaborative whiteboard application designed for quick sketching, architecture diagrams, wireframes, and visual brainstorming. Its distinctive hand-drawn aesthetic removes the pressure of high-fidelity graphics, encouraging rapid ideation and clear visual communication. Featuring real-time end-to-end encryption, infinite canvas navigation, rich shape libraries, customizable arrows, text blocks, and exported PNG/SVG diagrams, Excalidraw allows end users to draft system architectures, mind maps, and workflow diagrams quickly within their browser. [![excalidraw](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260808-19h25m09s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260808-19h25m09s-grim.png) --- ### [Funkwhale](https://funkwhale.drheck.dev/) Funkwhale is a self-hosted, decentralized audio and music streaming platform built on open web protocols (ActivityPub). It allows end users to import, organize, stream, and share their personal music collections and podcasts across devices while connecting with audio libraries on other federated instances. End users can create custom playlists, follow public audio channels, listen to high-quality audio streams in the web application or via Subsonic-compatible mobile apps, and share music tracks with friends without depending on centralized commercial streaming services. [![funkwhale](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260808-20h54m34s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260808-20h54m34s-grim.png) --- ### [Forgejo](https://git.drheck.dev/) Forgejo is a self-hosted, community-governed lightweight software forge that provides complete software development workflows similar to GitHub. It offers source code hosting, Git repository management, pull requests, code reviews, issue tracking, wikis, and organization management in a fast, low-footprint application. Integrated with Authentik via Single Sign-On (OIDC), users can log into Forgejo with their unified credentials to push and pull Git repositories, manage SSH keys, configure webhook integrations, run automated CI/CD pipelines, and collaborate on software projects. [![gitea](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-12h04m49s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-12h04m49s-grim.png) --- ### [Grill Monitor (`FireBoard 2`)](https://grill.drheck.dev) Local cloud-free BLE-to-Wi-Fi telemetry and multi-channel temperature graphing dashboard (`grill.drheck.dev` / `fireboard.drheck.dev`) for the hardware-switched Seeed XIAO ESP32-C3 + FireBoard 2 6-probe smoker thermometer. --- ### [Immich](https://photos.drheck.dev) Immich is a high-performance, self-hosted photo and video management solution designed as a privacy-respecting alternative to Google Photos or Apple Photos. It provides seamless background photo backup from mobile devices (iOS and Android) as well as an intuitive web interface for browsing, searching, and sharing media libraries. Immich includes machine-learning features such as face recognition, object detection, spatial map views based on EXIF GPS data, album sharing, and multi-user support with quota management. SSO authentication via Authentik allows smooth, secure onboarding across all web and mobile client applications. [![immich](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-12h05m52s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-12h05m52s-grim.png) --- ### [Jellyfin](https://jellyfin.drheck.dev/) Jellyfin is a free, open-source media system that puts users in control of managing and streaming their personal movies, TV shows, music, and live TV broadcasts. Unlike commercial alternatives, Jellyfin contains no tracking, premium paywalls, or central server dependencies, running entirely self-hosted. Jellyfin automatically fetches metadata, artwork, and subtitle tracks for media libraries, transcoding video streams on-the-fly for smooth playback across web browsers, smart TVs, Roku, Fire TV, and mobile devices. Authentik LDAP integration allows unified credentials for all household members. [![jellyfin](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-12h08m08s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-12h08m08s-grim.png) --- ### [Kanji Practice](https://kanji.drheck.dev/) Kanji Practice is a specialized web application built for Japanese language learners to practice stroke order. There are two modes. In the first, each stroke will be animated in the correct order in a single pane, or there is an view showing each character being assembled one stroke at a time in successive panes. [![kanji](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-12h09m06s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-12h09m06s-grim.png) --- ### [Kavita](https://read.drheck.dev) Kavita digital library designed specifically for reading and managing digital comics, manga, light novels, and traditional e-books. It automatically organizes files (EPUB, CBZ, CBR, PDF) into series, volumes, and chapters with cover art and metadata tagging. Kavita includes built-in responsive web readers tailored for mobile devices, tablets, and desktop screens, offering custom reading modes (continuous scroll, dual page, manga right-to-left mode), bookmark sync, reading progress tracking, and user library permissions. [![kavita](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-12h26m50s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-12h26m50s-grim.png) --- ### [LibreSprite](https://libresprite.drheck.dev) LibreSprite is an open-source animated sprite editor and pixel art tool compiled to WebAssembly. Running 100% client-side in the browser, it provides a full-featured digital drawing and animation workspace for creating 2D game assets, animated GIFs, and pixel graphics without requiring local software installation. Users can draw pixel artwork, construct frame-by-frame sprite animations, manage custom color palettes, and export finished graphics as sprite sheets or animated images, bringing classic pixel art editing into a cloud-hosted web environment. --- ### [Minecraft Control Plane](https://mc.drheck.dev) Minecraft Control Plane (`mc-manager`) is the unified web administration and world provisioning studio for all Minecraft servers hosted on `ul`. Protected behind Authentik ForwardAuth, it provides real-time server lifecycle management (`On-Demand` wake-on-join via HAProxy, `Always On`, and `Manual`), a live WebSocket PTY server console (`mc-runner`), interactive `cubiomes` 2D biome & structure seed previews, and a built-in **Minecraft Text, MOTD, Color Gradient & 8-Bit Pixel Icon Studio** with live Multiplayer Server List ping updates. It also integrates with automated network, HAProxy SNI, and DNS provisioning (`ipctl` using short-lived `step-ca` SSH certificates and external config files) as well as 3D world maps such as [PokeLovers BlueMap](https://pokelovers.drheck.dev). --- ### [Navidrome](https://navidrome.drheck.dev/) Navidrome (`navidrome.drheck.dev` / `music.drheck.dev`) is a modern, lightweight, open-source music server and streamer that gives users complete freedom to listen to their personal music collection from anywhere. Built with high performance in mind, Navidrome indexes large music libraries effortlessly with low CPU and memory overhead. Navidrome features a responsive web UI and full compatibility with Subsonic/Airsonic mobile clients (such as DSub, amposer, and Symfonium). Users can stream music in original quality or on-the-fly transcoded formats, create playlists, mark favorites, and share music with family members. [![navidrome](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-13h46m51s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-13h46m51s-grim.png) --- ### [Nextcloud & Collabora Online](https://cloud.drheck.dev) Nextcloud is a productivity and cloud storage platform that provides file storage and access, documents, calendars, contacts, and communication tools. It serves as a privacy-focused alternative to proprietary cloud services like Google Drive and Microsoft 365. End users can sync files across desktop and mobile clients, edit office documents collaboratively in real time via Collabora Online (`office.drheck.dev`), schedule events, manage task lists, and share files securely via public password-protected web links. Authentik OIDC Single Sign-On grants secure access across all Nextcloud modules. [![nextcloud](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-13h48m01s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-13h48m01s-grim.png) --- ### [Penpot](https://penpot.drheck.dev) Penpot is the first open-source, web-based design and prototyping tool built natively on open web standards (SVG). Created as an alternative to proprietary platforms like Figma, Penpot allows design and development teams to craft UI/UX wireframes, interactive component prototypes, and vector illustrations. Because Penpot uses SVG as its native domain model, vector designs render with absolute fidelity for web Developers. It supports real-time multiplayer collaboration, shared design systems, component libraries, layout grids, and CSS code generation. Single Sign-On via Authentik enables instant user authentication. [![penpot](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-13h50m01s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-13h50m01s-grim.png) --- ### [Pixelorama](https://pixelorama.drheck.dev) Pixelorama is a free, open-source pixel art editor built with the Godot engine and deployed to the web for creating 2D sprites, tilesets, pixel animations, and retro game art directly in the browser. It features a rich toolkit tailored specifically for pixel artists, including custom pencil and eraser brushes, color palettes, onion skinning for frame-by-frame animation, layer management, pixel-perfect lines, tile mode previewing, and export options to sprite sheets or animated GIFs. [![pixelorama](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-14h17m33s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-14h17m33s-grim.png) --- ### [Plex](https://plex.drheck.dev) Plex is a comprehensive media streaming application that organizes personal video, audio, and photo collections into a clean, beautifully formatted media streaming service. Plex automatically matches files against online databases to attach movie posters, plot summaries, cast lists, trailers, and themes. End users can stream content seamlessly across a vast ecosystem of devices, including smart TVs, streaming sticks, mobile phones, game consoles, and web browsers, with intelligent server-side video transcoding to match client device capabilities and network bandwidth. [![plex](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-14h21m35s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-14h21m35s-grim.png) --- ### [RomM (ROM Manager)](https://romm.drheck.dev) RomM (ROM Manager) is a modern web application designed for organizing, scanning, and managing retro video game ROM libraries. RomM connects with online databases (such as IGDB and SteamGridDB) to automatically enrich game titles with high-resolution box art, screenshots, release dates, genre tags, and game descriptions. End users can browse their game collection by console platform, search titles, manage save states and ROM variants, and launch games directly in the browser using integrated web emulators or send game files to local emulation devices. --- ### [Sopro Text-to-Speech](https://sopro.drheck.dev) Sopro is a self-hosted neural text-to-speech synthesis web studio gated behind Authentik ForwardAuth on `ul`. --- ### [Studio Control Gateway (`scontrol`)](https://scontrol.drheck.dev) Studio Control (`scontrol.drheck.dev` / `studio-kvm-web.drheck.dev`) orchestrates one-touch RF power sequencing, ATX motherboard ignition, and Echo AudioFire 12 / Ardour DAW readiness across the recording studio rack. --- ### [SubOCR Japanese Subtitle Study Platform](https://subocr.drheck.dev) SubOCR extracts burned-in Japanese video subtitles via ROCm `PP-OCRv5` GPU inference on the AMD Radeon AI PRO R9700, refines dialogue via local `qwen2.5:32b`, and generates Kanji-only furigana, JLPT tags, conjugation tables, and interactive video study decks. --- ### [Sunshine Game Stream Host](https://sunshine.drheck.dev) Bare-metal KMS/DRM Gamescope + Sunshine low-latency streaming server running directly on the AMD Radeon AI PRO R9700 with Steam libraries hosted on ZFS `/data/games/steam`. --- ### [tldraw](https://tldraw.drheck.dev) tldraw is a collaborative, infinite-canvas drawing tool and visual whiteboarding app designed for instant sketching, diagramming, and visual note-taking. Built around speed and simplicity, tldraw provides a minimal canvas interface with immediate tool accessibility. End users can create vector shapes, freehand ink lines, sticky notes, connective arrows, and text labels. Supporting multiplayer real-time collaboration and SVG/PNG exports, tldraw is ideal for quick technical diagrams, architecture brainstorming, and informal remote collaboration. [![20260809_14h27m42s_grim.png](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-14h27m42s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-14h27m42s-grim.png) --- ### [Transmission](https://bt.drheck.dev) Transmission is a fast, lightweight BitTorrent client equipped with a clean web interface for managing peer-to-peer file transfers remotely. The web UI allows users to monitor active downloads and uploads, add new torrent files or magnet links, adjust bandwidth rate limits, and organize downloads into destination folders. Protected behind Authentik ForwardAuth reverse-proxy authentication, users can safely monitor long-running torrent transfers and manage media downloads on drheck.dev from any web browser without exposing unauthenticated torrent control endpoints to the public internet. [![20260812_21h29m26s_grim.png](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260812-21h29m26s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260812-21h29m26s-grim.png) --- ### [Vaultwarden](https://pw.drheck.dev) Vaultwarden (`pw.drheck.dev` / `vault.drheck.dev`) is a lightweight, self-hosted implementation of the Bitwarden secrets and password management server API written in Rust. It provides full compatibility with official Bitwarden desktop applications, browser extensions, mobile apps, and CLI tools while consuming minimal memory resources. End users can securely store end-to-end encrypted login credentials, credit cards, secure notes, personal identities, and two-factor authentication (TOTP) seed keys. Enforced with Authentik OIDC Single Sign-On and strict password policies, Vaultwarden ensures seamless, secure password management across all user devices. (secrets) --- ### [Vikunja Task Manager](https://vikunja.drheck.dev) Vikunja is a self-hosted project, Kanban board, Gantt chart, and task management platform backed by host PostgreSQL, host Valkey caching, and Authentik OIDC Single Sign-On. --- ### [Wavacity](https://wavacity.drheck.dev) Wavacity is an open-source web browser port of the popular Audacity audio editing application. Compiled to WebAssembly, it brings Audacity's iconic multi-track audio editing workspace directly into modern web browsers without requiring local software installation or native binaries. Users can import multi-track audio files, perform non-destructive editing, apply audio filters, cut and align tracks, inspect frequency spectrums, and export finished master tracks in standard audio formats, carrying the familiar Audacity interface into a cloud-hosted web environment. [![wavacity](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-14h29m23s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-14h29m23s-grim.png) --- ### [webЯcade Setup](https://arcade.drheck.dev) webЯcade is a cloud-based retro gaming ecosystem that allows users to play classic console and arcade games directly inside modern web browsers across desktop computers, mobile phones, tablets, and smart TVs. The webЯcade setup application allows users and administrators to configure game feeds, set up cloud save game synchronization, map custom controller inputs, and manage retro emulator settings for seamless web-based retro gaming. [![arcade](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-14h30m13s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-14h30m13s-grim.png) --- ### [Webshell](https://webshell.drheck.dev/) Webshell is a web-based terminal interface (powered by ttyd and short-lived 5-minute `step-ca` SSH certificates) that provides authenticated users with direct, browser-based command-line shell access to host environment tools. Protected behind Authentik Single Sign-On, Webshell enables administrators and authorized users to safely perform terminal administration tasks, check system diagnostics, run scripts, and manage files from mobile devices or workstations without needing native SSH client software. [![20260812_21h31m43s_grim.png](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260812-21h31m43s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260812-21h31m43s-grim.png) --- ### [Wick Editor](https://wick.drheck.dev) Wick Editor is an open-source, web-based tool for creating animations, games, and interactive code projects. Combining the accessibility of classic Flash software with modern web standards (HTML5/JavaScript), Wick Editor makes interactive media creation approachable directly in the browser. Users can draw vector artwork, build timeline-based frame animations, program interactivity with beginner-friendly JavaScript code snippets, attach sound effects, and export projects as interactive HTML5 games, standalone web apps, animated GIFs, or video files. [![wick](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-14h32m31s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-14h32m31s-grim.png) --- ### [WIPStudio](https://wipstudio.drheck.dev/) WIPStudio (Work-In-Progress Studio) is a self-hosted creative media and web development studio platform designed for managing ongoing multimedia projects, interactive web prototypes, and creative assets. Integrated with Authentik via OpenID Connect (OIDC), WIPStudio provides authenticated creators with a unified workspace to draft visual concepts, organize project files, review media builds, and collaborate on experimental digital projects. [![wip studio](https://wiki.drheck.dev/uploads/images/gallery/2026-08/scaled-1680-/20260809-14h52m18s-grim.png)](https://wiki.drheck.dev/uploads/images/gallery/2026-08/20260809-14h52m18s-grim.png) --- ### [YOURLS Shortener (`drheck.dev`)](https://drheck.dev/admin/) YOURLS powers custom short links on `https://drheck.dev/`, with public HTTP `301` resolution and private admin/API management protected behind Authentik ForwardAuth (`/yourls-api.php` and `/admin/`). # SSH Command Tracking Guide Detail of command execution auditing and tracking across human SSH sessions and AI agent sessions (`amux`). --- ## Part 1: Arch Linux (`ul`, `bmo95`, `studio-kvm-ssh`) The Linux Audit Framework (`auditd`) captures `execve` system calls triggered by human user sessions (`auid >= 1000`), recording all interactive and SSH payload commands while strictly filtering out background system and rootless Podman container noise. ### 1. Installation & Service Setup Install the `audit` package and enable the `auditd` daemon: ```shell $ sudo pacman -S --noconfirm audit $ sudo systemctl enable --now auditd ``` ### 2. Configuration Rules & Interactive User Filtering Create `/etc/audit/rules.d/execve.rules` with Audit User ID filtering (`auid >= 1000`): > **CRITICAL:** Do NOT include catch-all `execve` rules without `-F auid>=1000`. Background system daemons and rootless Podman containers started by `user@1100.service` (e.g. MinIO healthchecks, Authentik workers) have `auid = 4294967295` (`unset`). An unfiltered rule generates ~8 MB of log noise every 7 minutes, rotating out entire log histories within 30 minutes. ```shell $ sudo tee /etc/audit/rules.d/execve.rules > /dev/null << 'EOF' # Delete pre-existing rules -D # Set kernel buffer size -b 8192 # Log all command executions strictly for human/login users (auid >= 1000) -a always,exit -F arch=b64 -S execve -F auid>=1000 -k remote_commands -a always,exit -F arch=b32 -S execve -F auid>=1000 -k remote_commands EOF ``` Load rules into the active kernel: ```shell $ sudo auditctl -D && sudo augenrules --load $ sudo auditctl -e 1 ``` Verify active rules: ```shell $ sudo auditctl -l ``` #### Process UID (`uid`) vs Audit User ID (`auid`) * **Process UID (`uid=1100` `docker-data` & subordinate `624288`–`689823`):** Controls unprivileged host kernel UIDs for rootless Podman containers. * **Audit User ID (`auid` / `loginuid`):** Assigned exclusively by PAM when a human authenticates via SSH or TTY (`auid = 1000`). Background containers started via systemd lingering (`user@1100.service`) have `auid = 4294967295` (`unset`). * **Result:** Filter by `auid >= 1000`. --- ## Part 2: Agent Execution Auditing (`amux` / `systemd journal`) When AI agents or automated scripts run multi-line code via `write-pane -f ` or `write-pane -b ` in **amux**, those commands can bypass standard `auditd` `execve` logging (because they are streamed directly into an already-running shell's PTY or an interactive Python/Node REPL). To ensure complete, tamper-resistant auditing and forensic recovery: ### 1. Structured Journal Logging Every call to `write-pane` in `amux` automatically logs directly into the **systemd journal** with structured fields: - `SYSLOG_IDENTIFIER=amux` - `AMUX_PANE_ID`: Target pane (`%0`, `%1`, etc.) - `AMUX_WINDOW_NAME`: Hierarchical task window (e.g. `mc-pokelovers-bluemap`) - `AMUX_SOURCE`: Original script file path or `base64` - `AMUX_ARCHIVE_PATH`: Permanent snapshot location in `/data/amux/scripts/` - `AMUX_PAYLOAD_SIZE`: Injected byte count Querying agent execution logs: ```shell # Stream live agent injections $ journalctl -t amux -f # Inspect detailed metadata for recent agent commands $ journalctl -t amux -n 20 -o verbose ``` ### 2. Permanent Script Archival (`/data/amux/scripts/`) Before any byte is written to the terminal PTY, `amux` automatically saves a bit-for-bit snapshot to: ``` /data/amux/scripts/YYYY-MM-DD_HHMM.SS_.sh ``` - **Durability:** Stored on `/data` with ownership `luke:luke`. - **Zero Loss:** Even if an agent immediately removes a temporary file from `/dev/shm` after execution, the exact script executed is preserved permanently. - **Disaster Recovery:** `/data/amux/scripts/` is automatically backed up during the daily backup run via `/home/luke/git/backups/ul/scripts/backup/amux`. --- ## Part 3: Python Command Inspection & Summarization (`audit-log`) The `/usr/local/bin/audit-log` Python utility parses, decodes, and summarizes `EXECVE` records from `/var/log/audit/audit.log.*`. ### CLI Command Options & Usage ```shell # 1. Default view: last 20 executed commands $ audit-log # 2. View specific number of entries (e.g. last 50 commands) $ audit-log 50 $ audit-log -n 50 # 3. Stream all recorded command history $ audit-log all $ audit-log --all # 4. Filter by specific day (YYYY-MM-DD) $ audit-log --day 2026-08-29 # 5. Filter date range (since date / until date) $ audit-log --since 2026-08-25 --clean # 6. Filter repetitive background noise (tmux status bars, pollers) $ audit-log --clean -n 30 # 7. Regex / substring search across command history $ audit-log --search "funkwhale|podman" --clean # 8. Deduplicate identical commands in output $ audit-log --unique --since 2026-08-28 # 9. Generate automated daily change log summary outline $ audit-log --daily --since 2026-08-25 --clean ``` --- ## Part 4: OpenBSD (Router `danknasty`) OpenBSD uses native kernel process accounting (`acct`) to track binary executions, alongside syslog-based `sshd` verbose logging and `doas` auditing. ### 1. Kernel Process Accounting (`acct`) #### Enable Accounting: ```shell # Create log file $ doas touch /var/account/acct # Enable accounting at boot $ echo "accounting=YES" | doas tee -a /etc/rc.conf.local # Start accounting daemon immediately $ doas /usr/sbin/accton /var/account/acct ``` ### 2. Real-Time Command Inspection (`audit-log`) Query process execution history using the matching `audit-log` helper script: ```shell # View the last 20 executed commands (Default) $ doas audit-log # View a specific number of recent commands (e.g. 50 entries) $ doas audit-log 50 # View ALL recorded command history $ doas audit-log all ``` ### 3. SSH Verbose Payload & `doas` Logging To log non-interactive SSH connections and payload commands to `/var/log/authlog`: 1. Edit `/etc/ssh/sshd_config`: ```text LogLevel VERBOSE ``` 2. Reload `sshd`: ```shell $ doas rcctl reload sshd ``` 3. Monitor logs: ```shell $ doas tail -f /var/log/authlog ``` # Agent's Guide to Wiki ## Agent Guide: Updating BookStack Wiki Pages This guide outlines how AI agents can directly query, read, edit, and update pages in the self-hosted **BookStack Wiki** (https://wiki.drheck.dev/). --- ## 1. Golden Rules for Agents > ### 🚨 MANDATORY RULE: READ LIVE CONTENT FIRST BEFORE EDITING > **NEVER generate or overwrite page content from memory or scratch without first reading the current live page and its latest revisions.** > Humans frequently make live edits in the BookStack browser interface between agent turns. Overwriting a page without reading its latest `markdown` and `page_revisions` destroys user contributions, custom links, code blocks, and embedded images. > > **Every page update MUST follow this sequence:** > 1. Fetch live markdown to a file via `/home/luke/bin/read-bookstack-page `. > 2. Query attached images from the `images` table. > 3. Incorporate existing content, user edits, and image tags into the updated document. > 4. Deploy changes with `/home/luke/bin/update-bookstack-page `. > 5. Pull down the live rendered page and verify HTML and KaTeX formula output via `/home/luke/bin/verify-bookstack-page `. > ### 🚨 MANDATORY RULE: PULL DOWN & VERIFY THE RENDERED PAGE & FORMULAS AFTER EVERY EDIT > **Mathematical formulas (`$$...$$` and `$...$`), signal chains (`\longrightarrow`), matrices, and TeX output are desired and encouraged across the wiki—but once an edit is made, you MUST pull down the live rendered page and check the output.** > > Never assume a formula rendered properly just because the Markdown source looks right or `LENGTH(html)` is non-zero in MariaDB. BookStack's server-side CommonMark parser transforms backslash-punctuation sequences before client-side KaTeX runs, and unquoted shell expansions can corrupt either the page source or the global `app-custom-head` KaTeX loader. > > **After every wiki edit, you MUST:** > 1. Run `/home/luke/bin/verify-bookstack-page ` (also executed automatically by `update-bookstack-page` and `create-bookstack-page`), which pulls down the live rendered page over HTTPS (`https://wiki.drheck.dev/books//page/`) and validates: > - Every inline `