Web based SSH Instant Access

Webshell (`webshell.drheck.dev`) Setup & Rollout Guide

Setup for webshell.drheck.dev, a browser-based SSH terminal service authenticated via Authentik Single Sign-On (SSO) and short-lived OpenSSH certificates signed by Smallstep (step-ca).

1. Architectural Overview

[ Browser Client ]
        │
        ▼ (HTTPS / WebSocket)
[ Nginx Reverse Proxy (/etc/nginx/sites-available/webshell) ]
        │
        ├── auth_request ──► [ Authentik Embedded Outpost ] (SSO Auth & Claims)
        │
        ▼ (HTTP / WebSocket + X-Authentik-Username Header)
[ Webshell Container (ttyd + login-wrapper.sh) on eth77 fd77:...:771d:7681 ]
        │
        ├── Step SSH Certificate Sign ──► [ Step-CA Container on fd77:...:ca:9000 ]
        │
        ▼ (SSH with Ephemeral Ed25519 Certificate over IPv6)
[ Host ul OpenSSH Daemon on [fd77:7777:5010:0:1337:f1ac:d00d:337]:22 ]

Key Components

step-ca: Smallstep Certificate Authority issuing 5-minute ephemeral Ed25519 user certificates, running as unprivileged system user stepca (945:945).
webshell: Debian Bookworm container running static ttyd 1.7.4, step CLI 0.25.0, and login-wrapper.sh.
Authentik: Provides OIDC integration for step-ca and ForwardAuth single sign-on (webshell.drheck.dev).
Nginx: Terminates TLS and proxies HTTP/WebSocket connections while passing authenticated identity headers (X-Authentik-Username).
OpenSSH (sshd): Configured to trust step-ca signed user certificates for passwordless OS login on host ul.

2. Network & ULA IPv6 Allocation

Every container on host ul binds to a unique ULA IPv6 address on interface eth77 (fd77:7777:5010:0:1337:f1ac:d00d:.../64).

step-ca IPv6: fd77:7777:5010:0:1337:f1ac:d00d:ca (Port 9000)
webshell IPv6: fd77:7777:5010:0:1337:f1ac:d00d:771d (Port 7681)
Host ul ULA IPv6 SSH Target: fd77:7777:5010:0:1337:f1ac:d00d:337 (Port 22)

3. Step 1: Step-CA Configuration & Provisioners

A. Docker Compose & Environment

In /data/docker/webshell/docker-compose.yml, define step-ca (running as unprivileged system user 945:945) and webshell with dual-network connectivity, explicit ssh_known_hosts mounting, /etc/ttyd/index.html mounting, and /etc/ul_passwd validation mounting:

services:
  step-ca:
    image: smallstep/step-ca:latest
    container_name: step-ca
    restart: unless-stopped
    user: "945:945"
    env_file: .env
    volumes:
      - ./step-ca:/home/step:rw
    ports:
      - "[fd77:7777:5010:0:1337:f1ac:d00d:ca]:9000:9000"
    networks:
      - infra_net

  webshell:
    build: .
    container_name: webshell
    restart: unless-stopped
    environment:
      - STEP_CA_URL=https://step-ca:9000
    volumes:
      - ./login-wrapper.sh:/usr/local/bin/login-wrapper.sh:ro
      - ./known_hosts:/etc/ssh/ssh_known_hosts:ro
      - /etc/passwd:/etc/ul_passwd:ro
      - ./step-ca/certs/root_ca.crt:/etc/step-ca/root_ca.crt:ro
      - ./step-ca/secrets/webshell_svc_pwd:/etc/step-ca/webshell_svc_pwd:ro
      - ./index.html:/etc/ttyd/index.html:ro
    ports:
      - "[fd77:7777:5010:0:1337:f1ac:d00d:771d]:7681:7681"
    networks:
      - infra_net
      - webshell_net
    depends_on:
      - step-ca

networks:
  infra_net:
    external: true
  webshell_net:
    enable_ipv6: true
    ipam:
      config:
        - subnet: fd77:7777:5010:771d::/64

B. CA Directory Permissions

On host ul, restrict access to all CA keys and provisioner passwords to unprivileged system user stepca:stepca (945:945, /usr/bin/nologin) with mode 700:

sudo chown -R 945:945 /data/docker/webshell/step-ca
sudo chmod -R o-rwx /data/docker/webshell/step-ca
sudo chmod 700 /data/docker/webshell/step-ca/secrets /data/docker/webshell/step-ca/secrets/*

4. Step 2: Host `ul` OpenSSH Daemon Trust & Web Session Banner Setup

To allow sshd on host ul ([fd77:7777:5010:0:1337:f1ac:d00d:337]:22) to accept certificates signed by step-ca, enforce strict known_hosts checking, and serve a dedicated login banner exclusively for web sessions:

Extract the CA user public key:

docker exec step-ca step certificate inspect --format json /home/step/certs/user_ca_key.pub

Append the public key (ecdsa-sha2-nistp256 ... or ssh-ed25519 ...) to /etc/ssh/drheck_ca_user_key.pub.

Verify that /etc/ssh/sshd_config.d/10-drheck-certs.conf contains:

TrustedUserCAKeys /etc/ssh/drheck_ca_user_key.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

Match Address fd77:7777:5010:771d::/64
    Banner /etc/ssh/webshell_banner.txt

~~Create~~
Web Session Banner & Rotation Script: /etc/profile.d/webshell_banner.sh executes when a web session connects (SSH_CLIENT matches fd77:7777:5010:771d::2). It randomly selects an ANSI art file from /data/nfo/ansi_list.txt (42,000+ files), renders it via /usr/local/bin/ansview --wide-dupe across terminal width, right-aligns the file path, and exports $LAST_ANSI for the /usr/local/bin/bad_ansi removal command.

The static fallback banner is stored at /etc/ssh/webshell_banner.txt ~~on host~~ ul:
```
sudo touch /etc/ssh/webshell_banner.txt
sudo chmod 644 /etc/ssh/webshell_banner.txt
```
.

Generate /data/docker/webshell/known_hosts from host ul's public host key:

echo "fd77:7777:5010:0:1337:f1ac:d00d:337 $(cat /etc/ssh/ssh_host_ed25519_key.pub)" > /data/docker/webshell/known_hosts
chmod 644 /data/docker/webshell/known_hosts

Reload OpenSSH daemon:
```
sudo systemctl reload sshd
```

5. Step 3: Web Terminal Backend (`ttyd` + `login-wrapper.sh`) Setup

A. Dockerfile (`/data/docker/webshell/Dockerfile`)

Mandatory: Include -6 and -i :: in CMD so ttyd binds to IPv6 inside the ~~container~~container, ~~when~~-I connected/etc/ttyd/index.html to anload ~~IPv6~~custom ~~Docker~~HTML ~~bridge~~template, ~~network.~~and -t theme='{"background": "#000000"}' for pure black background:

FROM debian:bookworm-slim

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates curl openssh-client bash procps libnss-sss \
    && sed -i 's/passwd: files/passwd: files sss/' /etc/nsswitch.conf \
    && rm -rf /var/lib/apt/lists/*

# Install static ttyd 1.7.4 & step CLI 0.25.0
RUN curl -sSL https://github.com/tsl0922/ttyd/releases/download/1.7.4/ttyd.x86_64 -o /usr/local/bin/ttyd \
    && chmod +x /usr/local/bin/ttyd \
    && curl -sSL https://dl.step.sm/gh-release/cli/docs-step-cli/v0.25.0/step_linux_0.25.0_amd64.tar.gz | tar -xz -C /tmp \
    && mv /tmp/step_0.25.0/bin/step /usr/local/bin/step \
    && chmod +x /usr/local/bin/step \
    && rm -rf /tmp/step_0.25.0

COPY login-wrapper.sh /usr/local/bin/login-wrapper.sh
RUN chmod +x /usr/local/bin/login-wrapper.sh

EXPOSE 7681
ENTRYPOINT ["/usr/local/bin/ttyd"]
CMD ["-6", "-i", "::", "-W", "-H", "X-Authentik-Username", "-t", "theme={\"background\": \"#000000\"}", "-I", "/etc/ttyd/index.html", "/usr/local/bin/login-wrapper.sh"]

B. Backend Login Wrapper (`/data/docker/webshell/login-wrapper.sh`)

#!/usr/bin/env bash
set -euo pipefail

# Resolve username across CGI-style environment variables exported by ttyd/proxy
TARGET_USER="${X_AUTHENTIK_USERNAME:-${TTYD_USER:-${HTTP_X_AUTHENTIK_USERNAME:-${AUTHENTIK_USERNAME:-}}}}"

if [[ -z "${TARGET_USER}" ]]; then
  echo "Authentication Error: Missing Authentik identity." >&2
  exit 1
fi

# Strict alphanumeric regex check
if [[ ! "${TARGET_USER}" =~ ^[a-z_][a-z0-9_-]{0,31}$ ]]; then
  echo "Authentication Error: Invalid username format." >&2
  exit 1
fi

# Retrieve numeric UID from host passwd database
TARGET_UID=$(awk -F: -v u="${TARGET_USER}" '$1 == u {print $3}' /etc/ul_passwd 2>/dev/null || true)
if [[ -z "${TARGET_UID}" ]]; then
  echo "Authentication Error: User does not exist on target host." >&2
  exit 1
fi

# Reject system accounts (UID < 1000), admin, and docker* accounts
if (( TARGET_UID < 1000 )) || [[ "${TARGET_USER}" == "admin" ]] || [[ "${TARGET_USER}" == docker* ]]; then
  echo "Authentication Error: Access to privileged or system account is forbidden." >&2
  exit 1
fi

KEY_DIR="$(mktemp -d -t ssh_XXXXXX)"
KEY_PATH="${KEY_DIR}/id_ed25519"

cleanup() {
  rm -rf "${KEY_DIR}"
}
trap cleanup EXIT INT TERM

# Generate ephemeral Ed25519 keypair silently
ssh-keygen -q -t ed25519 -f "${KEY_PATH}" -N "" >/dev/null 2>&1

# Sign user certificate via JWK provisioner (--sign flag is mandatory for non-interactive execution)
if command -v step >/dev/null 2>&1 && [[ -n "${STEP_CA_URL:-}" ]] && [[ -f "/etc/step-ca/webshell_svc_pwd" ]]; then
  step ssh certificate "${TARGET_USER}" "${KEY_PATH}.pub"     --sign     --provisioner="webshell-svc"     --provisioner-password-file="/etc/step-ca/webshell_svc_pwd"     --ca-url="${STEP_CA_URL}"     --root="/etc/step-ca/root_ca.crt"     --not-after=5m     --force >/dev/null 2>&1 || true
fi

# Login passwordless to host ul SSH daemon over local ULA IPv6 with strict known_hosts checking
exec ssh -o StrictHostKeyChecking=yes          -i "${KEY_PATH}"          "${TARGET_USER}@fd77:7777:5010:0:1337:f1ac:d00d:337" "$@"

6. Step 4: Authentik Single Sign-On Provisioning (`ak shell`)

To configure Authentik ForwardAuth without encountering 500 Internal Server Error, Redirect URI Error, or ERR_TOO_MANY_REDIRECTS, execute the following Python provisioning script in authentik-server:

docker exec -i authentik-server ak shell << 'EOF'
from authentik.core.models import Application
from authentik.providers.proxy.models import ProxyProvider, ProxyMode
from authentik.providers.oauth2.models import RedirectURI, RedirectURIMatchingMode, RedirectURIType
from authentik.outposts.models import Outpost

SERVICE_NAME = "webshell"
EXTERNAL_HOST = f"https://{SERVICE_NAME}.drheck.dev"

# 1. Inherit default scope property mappings from a working provider template
template_provider = ProxyProvider.objects.get(name="cockpit")

# 2. Create ProxyProvider with complete OAuth, redirect URI, and cookie settings
provider, _ = ProxyProvider.objects.get_or_create(
    name=SERVICE_NAME,
    defaults={
        "external_host": EXTERNAL_HOST,
        "mode": ProxyMode.FORWARD_SINGLE,
        "authorization_flow_id": "f9ad5326-f668-4d1f-861d-43b47f0a002e",
        "cookie_domain": "drheck.dev",
        "grant_types": ["authorization_code", "client_credentials", "password"],
        "redirect_uris": [
            RedirectURI(
                matching_mode=RedirectURIMatchingMode.STRICT,
                url=f"{EXTERNAL_HOST}/outpost.goauthentik.io/callback?X-authentik-auth-callback=true",
                redirect_uri_type=RedirectURIType.AUTHORIZATION
            ),
            RedirectURI(
                matching_mode=RedirectURIMatchingMode.STRICT,
                url=f"{EXTERNAL_HOST}?X-authentik-auth-callback=true",
                redirect_uri_type=RedirectURIType.AUTHORIZATION
            ),
        ]
    }
)

provider.property_mappings.set(template_provider.property_mappings.all())
provider.save()

# 3. Create Application
app, _ = Application.objects.get_or_create(
    slug=SERVICE_NAME,
    defaults={
        "name": SERVICE_NAME.capitalize(),
        "meta_launch_url": f"{EXTERNAL_HOST}/",
        "open_in_new_tab": True,
        "provider": provider,
    }
)

# 4. Mandatory: Assign provider to authentik Embedded Outpost
outpost = Outpost.objects.get(name="authentik Embedded Outpost")
if provider not in outpost.providers.all():
    outpost.providers.add(provider)
    outpost.save()

print(f"Successfully provisioned {SERVICE_NAME} and attached to Embedded Outpost.")
EOF

7. Step 5: Nginx Reverse Proxy Configuration (`/etc/nginx/sites-available/webshell`)

map $http_upgrade $proxy_target_connection_webshell {
    default upgrade;
    ""      "";
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    server_name webshell.drheck.dev;

    ssl_certificate     /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/privkey.pem;

    client_max_body_size 64M;

    # Authentik ForwardAuth Subrequest Configuration
    auth_request     /outpost.goauthentik.io/auth/nginx;
    error_page       401 = @goauthentik_proxy_signin;
    auth_request_set $auth_cookie $upstream_http_set_cookie;
    add_header       Set-Cookie $auth_cookie;
    auth_request_set $authentik_username $upstream_http_x_authentik_username;
    auth_request_set $authentik_email $upstream_http_x_authentik_email;

    # Per-User Custom Fonts Route
    location /user-fonts/ {
        auth_request     /outpost.goauthentik.io/auth/nginx;
        auth_request_set $authentik_username $upstream_http_x_authentik_username;
        alias /home/$authentik_username/.config/webshell/fonts/;
        types {
            font/woff2 woff2;
            font/ttf ttf;
            font/otf otf;
        }
        add_header Cache-Control "public, max-age=86400";
    }

    location / {
        proxy_pass http://[fd77:7777:5010:0:1337:f1ac:d00d:771d]:7681;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $proxy_target_connection_webshell;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Authentik-Username $authentik_username;
        proxy_set_header X-Authentik-Email $authentik_email;
    }

    location /outpost.goauthentik.io {
        proxy_pass              https://auth.drheck.dev/outpost.goauthentik.io;
        proxy_set_header        Host auth.drheck.dev;
        proxy_set_header        X-Original-URL $scheme://$http_host$request_uri;
        proxy_set_header        X-Forwarded-Host $http_host;
        proxy_set_header        X-Forwarded-Proto $scheme;
        add_header              Set-Cookie $auth_cookie;
        auth_request_set        $auth_cookie $upstream_http_set_cookie;
        proxy_pass_request_body off;
        proxy_set_header        Content-Length "";
    }

    location @goauthentik_proxy_signin {
        internal;
        add_header Set-Cookie $auth_cookie;
        return 302 /outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
    }
}

8. Step 6: Backup & Restore Scripts

Every service on host ul requires shellcheck-verified <service>_backup and <service>_restore scripts located in /home/luke/git/backups/ul/scripts/ with 10% par2 redundancy.

/home/luke/git/backups/ul/scripts/webshell_backup: Archives /data/docker/webshell/ (.env, docker-compose.yml, step-ca/) to /data/backups/services/webshell/YYYY-MM-DD_HHMM.SS/webshell_config.webshell_data.tar.xz, and ~~generates~~system configuration/scripts (/etc/ssh/webshell_banner.txt, /etc/ssh/drheck_ca_user_key.pub, /etc/ssh/sshd_config.d/10-drheck-certs.conf, /etc/nginx/sites-available/webshell, /etc/profile.d/webshell_banner.sh, /usr/local/bin/bad_ansi, /data/nfo/ansi_list.txt) to webshell_system_config.tar.xz, generating .par2 recovery files via par2 create -r10.
/home/luke/git/backups/ul/scripts/webshell_restore: Verifies archives with par2 verify and ~~extracts~~restores configurations ~~back~~ into /data/docker/webshell/ and /.

9. Initial Assumptions & Attempts vs. Actual Working Configuration

Technical Area	Initial Assumption / Attempt	Actual Working Configuration & Root Cause	Prevention Standard
Outpost Assignment (`500 Internal Server Error`)	Creating `ProxyProvider` in Authentik automatically activates ForwardAuth routing.	Creating a provider does not assign it to an Outpost. Unassigned hostnames return `404 Not Found` from `/outpost.goauthentik.io/auth/nginx`, which Nginx converts to `500 Internal Server Error`.	Explicitly attach providers to `authentik Embedded Outpost` (`outpost.providers.add(provider)`) before enabling Nginx traffic.
Redirect URIs (`Redirect URI Error`)	Instantiating `ProxyProvider` sets default authorization callback URLs.	The Django ORM defaults `redirect_uris` to `[]`. Authentik rejects callback URIs against an empty database list.	Explicitly define `RedirectURI` objects for both `/outpost.goauthentik.io/callback` and root `/` in `ProxyProvider.redirect_uris`.
Cookie Domain & Scopes (`ERR_TOO_MANY_REDIRECTS`)	Providers operate without defining `cookie_domain`, `grant_types`, or `property_mappings`.	Without `property_mappings`, the OAuth request omits critical scopes (`ak_proxy`). Without explicit `cookie_domain='drheck.dev'`, Authentik refuses to grant ForwardAuth proxy tokens, causing an endless redirect loop.	Clone `property_mappings` from `cockpit` and explicitly set `cookie_domain='drheck.dev'` and `grant_types`.
CGI Env Mapping (`Missing Authentik identity`)	`ttyd -H X-Authentik-Username` exports `X_AUTHENTIK_USERNAME` directly to child scripts.	`ttyd` transforms HTTP headers into standard CGI environment variables (`HTTP_X_AUTHENTIK_USERNAME`).	Always check CGI-style fallback variations: `TARGET_USER="${X_AUTHENTIK_USERNAME:-${TTYD_USER:-${HTTP_X_AUTHENTIK_USERNAME:-${AUTHENTIK_USERNAME:-}}}}"`.
IPv6 Dual-Stack Listening (`502 Bad Gateway`)	Default `ttyd` invocation listens on all interfaces when connected to an IPv6 Docker network.	Without `-6 -i ::`, `ttyd` binds only to IPv4 `0.0.0.0`. When Docker Engine forwards published IPv6 ports to the container's IPv6 address, Nginx connections are rejected (`Connection refused`).	Always pass `"-6", "-i", "::"` to `ttyd` in `Dockerfile` so it listens in dual-stack mode.
Container Script Updates (`old script execution`)	Modifying `login-wrapper.sh` on host `ul` and running `docker compose restart` runs the updated script.	`docker compose restart` only restarts containers without re-reading `COPY`ed image files.	Always bind-mount operational scripts (`./login-wrapper.sh:/usr/local/bin/login-wrapper.sh:ro`) in `docker-compose.yml`.
CA Secret Privilege Restrictions	Administrative or personal user accounts own the CA keys and JWK provisioner secret.	Standard unprivileged or administrative (`wheel`) user accounts should not have read access to CA private keys or provisioner passwords.	Assign ownership of `/data/docker/webshell/step-ca` to unprivileged system user `stepca:stepca` (`945:945`, `/usr/bin/nologin`) and run `step-ca` container under user `945:945`.
SSH Client Host Warnings (`Warning: Permanently added...`)	Invoking `ssh` with `/dev/null` as known_hosts prints key addition warnings on every login.	Connecting without a populated known_hosts file causes OpenSSH to log a warning to stderr whenever an ephemeral host entry is added.	Mount a static `/etc/ssh/ssh_known_hosts` file generated from `/etc/ssh/ssh_host_ed25519_key.pub` and enforce `StrictHostKeyChecking=yes`.
Cockpit Login MOTD Banner (`Web console: https://ul...:9090`)	Cockpit's MOTD issue updater (`update-issue`) injects `:9090` URLs into `/etc/motd.d/`.	Cockpit's `cockpit-issue.service` inspects local interfaces and writes direct port URLs to `/run/cockpit/issue`, which `/etc/motd.d/cockpit` symlinks into login greetings.	Remove `/etc/motd.d/cockpit` and `/etc/issue.d/cockpit.issue` symlinks and mask `cockpit-issue.service` (`systemctl mask cockpit-issue.service`).
Dynamic Host UID Verification	Hardcoding system username lists (`root	admin	...`) in wrapper scripts to block system accounts.
Web-Session-Only Login Banner	Printing greetings via script `echo` or editing server-wide `/etc/motd`.	Script `echo` output is lost before terminal attachment, while `/etc/motd` changes affect standard SSH logins.	Configure `Match Address fd77:7777:5010:771d::/64` in `/etc/ssh/sshd_config.d/10-drheck-certs.conf` with `Banner /etc/ssh/webshell_banner.txt`.