SSH CA setup

OVERVIEW

Authentication Flow: Incoming SSH clients authenticate using an OpenSSH user certificate signed by the server's User Certificate Authority (CA).
LDAP & Authentik Integration: Host ul derives user accounts from Authentik via LDAP (managed via SSSD). When a client presents an SSH certificate, sshd validates the certificate signature against the CA public key and checks that one of the certificate's valid principals matches the requested LDAP login username. No per-user ~/.ssh/authorized_keys file is required.
Key Storage & Backup:
- The User CA private key (drheck_ca_user_key) and Host CA private key (drheck_ca_host_key) must never reside directly on client machines or internet-facing servers (such as ul).
- Store the offline CA private keys and passphrases securely in Vaultwarden.
- Deploy only the CA public keys (drheck_ca_user_key.pub and drheck_ca_host_key.pub) to host ul and client machines.
Key Inventory & Revocation:
- Every signed public key and certificate is automatically archived in /etc/ssh/ca_inventory/ with a descriptive label and fingerprint.
- Any compromised key can be revoked immediately using sudo revoke-key <label>.

PHASE 1: CA GENERATION & VAULTWARDEN BACKUP

Generate CA Keypairs on a secure offline or admin workstation:

When prompted by ssh-keygen, enter a strong, unique passphrase to encrypt each CA private key.

# Generate the User CA keypair
ssh-keygen -t ed25519 -f drheck_ca_user_key -C "drheck-user-ca"

# Generate the Host CA keypair
ssh-keygen -t ed25519 -f drheck_ca_host_key -C "drheck-host-ca"

Secure Storage in Vaultwarden:

Create a secure note or attachment in Vaultwarden containing the private keys (drheck_ca_user_key, drheck_ca_host_key) and their passphrases.
Remove the private CA keys from the unencrypted local storage of your workstation after verifying the backup.

PHASE 2: MAIN SERVER CONFIGURATION (ul)

1. Deploy Trusted User CA Public Key

Copy the public CA file drheck_ca_user_key.pub to host ul at /etc/ssh/drheck_ca_user_key.pub.

Set secure permissions:

sudo chmod 644 /etc/ssh/drheck_ca_user_key.pub
sudo chown root:root /etc/ssh/drheck_ca_user_key.pub

2. Configure Host Certificate (Signed by Host CA)

To keep the private drheck_ca_host_key offline, the host key must be signed on your secure workstation or via the signing script:

Copy ul's public host key /etc/ssh/ssh_host_ed25519_key.pub to your secure workstation.
On your workstation, sign the host key:

ssh-keygen -s drheck_ca_host_key -I "drheck-ul-host-cert" -h -n ul,ul.drheck.dev -V +52w ssh_host_ed25519_key.pub

Copy the resulting certificate ssh_host_ed25519_key-cert.pub back to ul at /etc/ssh/ssh_host_ed25519_key-cert.pub.
Set secure permissions on ul:

sudo chmod 644 /etc/ssh/ssh_host_ed25519_key-cert.pub
sudo chown root:root /etc/ssh/ssh_host_ed25519_key-cert.pub

3. Configure OpenSSH Daemon on ul

Create the config file /etc/ssh/sshd_config.d/10-drheck-certs.conf:
```
sudo vim /etc/ssh/sshd_config.d/10-drheck-certs.conf
```

Add the following directives to trust incoming User CA certificates and deploy the host certificate:

TrustedUserCAKeys /etc/ssh/drheck_ca_user_key.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

4. Validate and Reload SSHD on ul

Validate the configuration syntax before reloading:
```
sudo sshd -t
```
If no errors are returned, reload the daemon:
```
sudo systemctl reload sshd
```

PHASE 3: STREAMLINED REMOTE SIGNING (One-Liner Pipelines)

To eliminate manual scp, sudo mv, and context switching, helper utilities on host ul (/usr/local/bin/sign-host-key and /usr/local/bin/sign-user-key) accept public keys on stdin and stream the signed certificate directly to stdout.

1. Unlock Vaultwarden (Run Once per Onboarding Session on `ul`)

Log into ul and unlock the akadmin Vaultwarden vault (keeps credentials in RAM for 1 hour):

sudo -iu akadmin vw unlock

2. Stream & Sign New Host Key (Run on New Host)

From the machine being onboarded, pipe the host's public key directly over SSH into sign-host-key:

cat /etc/ssh/ssh_host_ed25519_key.pub | ssh -t ul "sudo sign-host-key <hostname> [principals]" | sudo tee /etc/ssh/ssh_host_ed25519_key-cert.pub

Interactive Label Prompt: If run with ssh -t, the script prompts on your terminal: Enter inventory label for this host [<hostname>]:

Automated Label Flag: You can also pass --label <name> or -l <name> to skip the prompt:

cat /etc/ssh/ssh_host_ed25519_key.pub | ssh ul "sudo sign-host-key bmo95 -l studio-pc" | sudo tee /etc/ssh/ssh_host_ed25519_key-cert.pub

Set secure permissions on the new host:

sudo chmod 644 /etc/ssh/ssh_host_ed25519_key-cert.pub

3. Stream & Sign User Key (Run on Client Workstation)

From your client workstation, pipe your user public key directly into sign-user-key:

cat ~/.ssh/id_ed25519.pub | ssh -t ul "sudo sign-user-key <ldap_username>" > ~/.ssh/id_ed25519-cert.pub
chmod 644 ~/.ssh/id_ed25519-cert.pub

Security & Lifecycle Details:

The public key is piped directly into a temporary /dev/shm RAM filesystem on ul.
The CA private key and passphrase are decrypted from Vaultwarden into /dev/shm, piped via SSH_ASKPASS, and securely shredded upon command exit (trap ... shred -u).
The private keys never touch physical disk storage on ul.
A copy of the public key and signed certificate are automatically preserved in /etc/ssh/ca_inventory/ for auditing and revocation.

PHASE 4: AUDITING & KEY REVOCATION (KRL)

1. View Inventory of Active & Revoked Keys

To see all signed hosts and user keys:

revoke-key --list

Example Output:

==========================================================================================
LABEL                TYPE     FINGERPRINT               STATUS      
------------------------------------------------------------------------------------------
danknasty            host     SHA256:oj796ZOeYmi6hAm+pG [ACTIVE]     
ul                   host     SHA256:1a4QDcX1p/QT4qbFdc [ACTIVE]     
borrowbook           user     SHA256:HPu6XQUlARHnLWgMWo [ACTIVE]     
metropolis           user     SHA256:22OChYHL9QxAe57Noe [ACTIVE]     
==========================================================================================

2. Revoke a Compromised Key

To immediately invalidate any key or certificate:

sudo revoke-key <label>

What this does:

Locates the archived public key in /etc/ssh/ca_inventory/.
Adds the key to OpenSSH's Key Revocation List (KRL) at /etc/ssh/revoked_keys.
Ensures RevokedKeys /etc/ssh/revoked_keys is configured in /etc/ssh/sshd_config.d/10-drheck-certs.conf.
Automatically reloads sshd. Any future connection attempts with that key or certificate are instantly rejected.

PHASE 5: VERIFICATION & MIGRATION

1. Test Certificate Authentication

Connect from the client machine with verbose debug output:
```
ssh -v ul
```
Confirm that the client authenticates using the certificate signature:
```
Server accepts key: pkalg ssh-ed25519-cert-v01@openssh.com
```

2. Client Host Verification (~/.ssh/known_hosts)

Add the Host CA public key to /etc/ssh/ssh_known_hosts (system-wide) or ~/.ssh/known_hosts on all client machines:
```
@cert-authority * ssh-ed25519 <drheck_ca_host_key.pub_content>
```
Clients will now connect to any host bearing a valid DrHeck host certificate without TOFU ("Trust On First Use") prompts or host key verification warnings.

3. Migration Strategy (Disabling Raw Keys)

Keep PubkeyAuthentication yes active on ul to allow standard .ssh/authorized_keys logins during client onboarding.

To enforce certificates from the external internet while keeping authorized_keys bootstrapping available on your trusted LAN:

# /etc/ssh/sshd_config.d/10-drheck-certs.conf
AuthorizedKeysFile /dev/null

Match Address 10.0.0.0/24,fd77:7777:5010::/64
    AuthorizedKeysFile .ssh/authorized_keys