# LDAP+PAM+SSH CA
# LDAP + SSSD Client Enrollment Guide
How to enroll Linux machines into the `drheck.dev` Authentik LDAP directory service using SSSD, with offline credential caching, dual-stack LAN failover, and GDM graphical login integration.
---
## 1. Architecture & Authentication Pipeline
$$
\text{Client PAM / NSS} \quad \longrightarrow \quad \text{SSSD Backend (\texttt{sssd})} \quad \longrightarrow \quad \text{HAProxy LDAPS (\texttt{:636})} \quad \longrightarrow \quad \text{Rootless Outpost (\texttt{authentik-ldap :6636})}
$$
| Parameter | Value | Notes |
| :--- | :--- | :--- |
| **Primary LDAPS URI (IPv6 ULA)** | `ldaps://ldap.drheck.dev:636` | Resolves to `[fd77:7777:5010:0:1337:f1ac:d00d:aaaa]:636` on `ul` (`eth77`, relayed via HAProxy TCP passthrough `/etc/haproxy/conf.d/ldap.cfg` to rootless `authentik-ldap` on `:6636`) |
| **Backup LDAPS URI (Dual-Stack IPv4/IPv6)** | `ldaps://ul.drheck.dev:636` | Resolves to `10.0.0.3:636` (IPv4) and `[fd77:7777:5010:0:1337:f1ac:d00d:337]:636` (relayed via HAProxy TCP passthrough) |
| **Base DN** | `dc=ldap,dc=goauthentik,dc=io` | Authentik LDAP Provider (`bind_mode=cached`, `search_mode=cached`) |
| **Linux User Search Base & Filter** | `ou=users,dc=ldap,dc=goauthentik,dc=io?sub?(memberOf=cn=locallogin,ou=groups,dc=ldap,dc=goauthentik,dc=io)` | Restricts enumeration & login to members of `locallogin` (`luke`, `haruka`, `ikura`), excluding internal service accounts |
| **Bind Service Account DN** | `cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io` | Requires `search_full_directory` object permission on the Authentik `ldap` provider |
| **Local SSSD Cache DB** | `/var/lib/sss/db/cache_goauthentik.io.ldb` | Persists enumerated users & password hashes (`cache_credentials = true`) for offline login |
Offline & LAN Resilience: SSSD is configured with enumerate = true, ldap_enumeration_refresh_offset = 0, and cache_credentials = true. Immediately upon enrollment on the local network, SSSD enumerates and caches all members of locallogin into /var/lib/sss/db/cache_goauthentik.io.ldb and pre-populates /var/lib/AccountsService/users/ so graphical GDM logins and getent passwd work even when the internet is offline.
---
## 2. Enrollment Steps
### Step 0: Automated One-Liner (`ldap_deploy.py`)
Run the enrollment script from a local administrator account with `sudo`:
```shell
git clone -b main https://git.drheck.dev/luke/ldap_deploy.git /tmp/ldap_deploy && sudo python3 /tmp/ldap_deploy/ldap_deploy.py && rm -rf /tmp/ldap_deploy
```
The automated installer performs all steps below, verifies live LDAPS TLS connectivity (`ldap.drheck.dev:636` / `ul.drheck.dev:636`), confirms `sssctl domain-status goauthentik.io` is `Online`, verifies that LDAP users resolve via `getent passwd -s sss`, and pre-provisions GDM `AccountsService` profiles.
---
### Step 1: Install SSSD & OpenLDAP Client Tools
Install both `sssd` and the OpenLDAP client utilities (`openldap` on Arch Linux, `ldap-utils` on Debian/Ubuntu) so `ldapsearch` and `sssctl` are available for offline diagnostics:
* **Arch Linux:**
```bash
sudo pacman -S --needed sssd openldap
```
* **Debian / Ubuntu:**
```bash
sudo apt install sssd sssd-tools sssd-ldap libnss-sss libpam-sss ldap-utils
```
Installing When Internet Is Down (Arch Linux): On Arch Linux, the sssd package depends on libldap (shared libraries) rather than openldap (CLI tools like ldapsearch). If the internet is down and a package is already in the local pacman cache, install it directly with sudo pacman -U /var/cache/pacman/pkg/sssd-*.pkg.tar.zst. If ldapsearch is not installed, use the built-in openssl s_client and sssctl diagnostic commands in Section 4 below.
Automatic Home Directory Creation: The pam_mkhomedir.so module in the session block automatically creates /home/<username> on first login, copying skeleton files from /etc/skel/ with umask=0022 (0755 permissions).
```text
#%PAM-1.0
auth required pam_faillock.so preauth
auth sufficient pam_sss.so forward_pass
auth [success=1 default=bad] pam_unix.so try_first_pass nullok
auth [default=die] pam_faillock.so authfail
auth optional pam_permit.so
auth required pam_env.so
auth required pam_faillock.so authsucc
account sufficient pam_sss.so
account required pam_unix.so
account optional pam_permit.so
account required pam_time.so
password sufficient pam_sss.so
password required pam_unix.so try_first_pass nullok sha512 shadow
password optional pam_permit.so
session required pam_limits.so
session required pam_unix.so
session required pam_mkhomedir.so skel=/etc/skel/ umask=0022
session optional pam_exec.so seteuid /usr/local/bin/gdm-ldap-user-session.sh
session optional pam_sss.so
session optional pam_permit.so
```
Ensure `/etc/pam.d/su` and `/etc/pam.d/su-l` include `system-auth` so `su - ` works from a terminal:
```text
#%PAM-1.0
auth sufficient pam_rootok.so
auth include system-auth
account include system-auth
session include system-auth
password include system-auth
```
---
### Step 5: Unlock GDM Login Screen & Session Selector (`AccountsService`)
Arch Linux compiles `accountsservice` without native SSSD enumeration support (see [NSS + GDM Issue](https://wiki.drheck.dev/books/system-setup/page/nss-gdm-issue)). Without a profile in `/var/lib/AccountsService/users/`, GDM may hide remote LDAP users from the login list or hide the desktop session selector (gear icon).
#### 1. Create the PAM Auto-Provisioning Script (`/usr/local/bin/gdm-ldap-user-session.sh`)
```shell
sudo tee /usr/local/bin/gdm-ldap-user-session.sh >/dev/null << 'EOS'
#!/bin/bash
# Automatically provision AccountsService files for remote SSSD/LDAP users
# on their first login, unlocking GDM session selection on Arch Linux.
USER="$PAM_USER"
if [ -z "$USER" ]; then
exit 0
fi
# 1. Skip local users (already in /etc/passwd)
if ! grep -q "^$USER:" /etc/passwd && id "$USER" &>/dev/null; then
UID_NUM=$(id -u "$USER")
# 2. Only process standard users (UID >= 1000)
if [ "$UID_NUM" -ge 1000 ]; then
FILE="/var/lib/AccountsService/users/$USER"
if [ ! -f "$FILE" ]; then
cat < "$FILE"
[User]
Session=gnome
Icon=/home/$USER/.face
SystemAccount=false
EOF
chown root:root "$FILE"
chmod 600 "$FILE"
fi
fi
fi
EOS
sudo chmod 755 /usr/local/bin/gdm-ldap-user-session.sh
```
#### 2. Pre-Provision Existing LDAP Users Immediately (Before First Login)
Once SSSD is started in Step 6, run the following command (handled automatically by `ldap_deploy.py`) to pre-populate `/var/lib/AccountsService/users/` for all enumerated LDAP users so they appear on the GDM login screen immediately without needing a prior terminal login:
```shell
if [ -d /var/lib/AccountsService/users ]; then
for u in $(getent passwd -s sss | awk -F: '$3 >= 2000 {print $1}'); do
if ! grep -q "^${u}:" /etc/passwd && [ ! -f "/var/lib/AccountsService/users/${u}" ]; then
sudo tee "/var/lib/AccountsService/users/${u}" >/dev/null <No Internet Required: All troubleshooting steps below work completely offline on the local LAN using standard system binaries (getent, sssctl, openssl, python3, journalctl) even if ldapsearch (openldap) is not installed.
### Issue A: `getent passwd` Only Shows `root` and Local User (`luke`), Nothing for LDAP Users
1. **Test Direct Username Lookup vs. Enumeration:**
```bash
getent passwd -s sss ikura
getent passwd -s sss haruka
```
* **If `getent passwd -s sss ikura` returns a line (`ikura:*:2022:...`) while bare `getent passwd` does not:**
`/etc/sssd/sssd.conf` has `enumerate = false` (or the initial 10-second enumeration delay has not finished). Set `enumerate = true` and `ldap_enumeration_refresh_offset = 0` in `/etc/sssd/sssd.conf` and restart `sssd`.
* **If `getent passwd -s sss ikura` returns nothing (exit code `2`):**
SSSD is failing to reach or bind to `ldap.drheck.dev:636`. Proceed to **Issue B** below.
---
### Issue B: SSSD Domain Is `Offline` or Cannot Resolve Any LDAP Users
Check SSSD's domain status and backend log:
```bash
sudo sssctl domain-status goauthentik.io
sudo tail -n 35 /var/log/sssd/sssd_goauthentik.io.log
```
#### Step 1: Verify DNS Resolution (`ldap.drheck.dev` and `ul.drheck.dev`)
`ldap.drheck.dev` resolves to `ul`'s ULA IPv6 address (`fd77:7777:5010:0:1337:f1ac:d00d:aaaa`), while `ul.drheck.dev` resolves to both IPv4 (`10.0.0.3`) and IPv6 (`fd77:7777:5010:0:1337:f1ac:d00d:337`):
```bash
getent ahosts ldap.drheck.dev
getent ahosts ul.drheck.dev
```
* If the client machine does not have a local ULA IPv6 address (`ip -6 addr show`), ensure `ldap_backup_uri = ldaps://ul.drheck.dev:636` is present in `/etc/sssd/sssd.conf` so SSSD connects over IPv4 (`10.0.0.3:636`).
#### Step 2: Test LDAPS TCP & TLS Certificate Handshake (Without `ldapsearch`)
Use `openssl` (installed on all base systems) to verify that port `636` on `ul` is reachable and presenting a valid Let's Encrypt certificate for `*.drheck.dev`:
```bash
openssl s_client -connect ldap.drheck.dev:636 -servername ldap.drheck.dev &1 | grep -E "CONNECTED|subject=|issuer=|Verify return code"
```
Or test over IPv4 via `ul.drheck.dev:636`:
```bash
openssl s_client -connect 10.0.0.3:636 -servername ul.drheck.dev &1 | grep -E "CONNECTED|subject=|issuer=|Verify return code"
```
Expected output:
```text
CONNECTED(00000003)
subject=CN=drheck.dev
issuer=C=US, O=Let's Encrypt, CN=YE1
Verify return code: 0 (ok)
```
* **If `Verify return code` is non-zero or clock is skewed:** Check `timedatectl status` on the client laptop (if the CMOS clock is wrong when offline, TLS certificate validation against Let's Encrypt will fail).
#### Step 3: Check for `Invalid credentials (49)` / Bind Password Mismatch
If `openssl s_client` succeeds with `Verify return code: 0 (ok)`, check `/var/log/sssd/sssd_goauthentik.io.log` (with debug level 6 if needed):
```bash
sudo sssctl debug-level 6
sudo systemctl restart sssd
sleep 3
sudo grep -i -E "bind|credential|tls|error|fail|offline" /var/log/sssd/sssd_goauthentik.io.log | tail -n 30
sudo sssctl debug-level 1
```
If `openldap` (`ldapsearch`) is installed, test the `ldapservice` bind directly:
```bash
LDAPTLS_REQCERT=demand ldapsearch -LLL -x \
-H ldaps://ldap.drheck.dev:636 \
-D "cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io" \
-w "hgNbhl3t1hzyV4jdfsiQuAeYS" \
-b "ou=users,dc=ldap,dc=goauthentik,dc=io" "(cn=ikura)" cn uidNumber gidNumber
```
* **If `ldapsearch` returns `ldap_bind: Invalid credentials (49)`** (or `sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u authentik-ldap.service` on `ul` shows `"error":"flow error password: Invalid password"` for `cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io`):
The `ldapservice` password in Authentik on `ul` does not match `ldap_default_authtok` in `/etc/sssd/sssd.conf` (or another service overwrote `ldapservice`'s password instead of using its own dedicated service account).
To verify and restore `ldapservice` on `ul`:
```bash
ssh ul '(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -i authentik-server ak shell -c "
from authentik.core.models import User
from authentik.providers.ldap.models import LDAPProvider
p = LDAPProvider.objects.first()
u = User.objects.get(username=\"ldapservice\")
print(\"Password match:\", u.check_password(\"hgNbhl3t1hzyV4jdfsiQuAeYS\"))
print(\"LDAPProvider perms:\", u.get_all_permissions(p))
" 2>/dev/null)'
```
If `Password match:` is `False`, restore it on `ul` and restart `authentik-ldap.service` so its cached bind session is flushed:
```bash
ssh ul '(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -i authentik-server ak shell -c "
from authentik.core.models import User
u = User.objects.get(username=\"ldapservice\")
u.set_password(\"hgNbhl3t1hzyV4jdfsiQuAeYS\")
u.save()
" 2>/dev/null) && sudo systemctl --user -M docker-data@ restart authentik-ldap.service'
```
---
### Issue C: Flushing Stale SSSD Cache & Forcing SSSD Online
If SSSD marked the domain offline during a temporary network glitch, force it back online and clear the cache:
```bash
sudo systemctl stop sssd
sudo rm -f /var/lib/sss/db/cache_goauthentik.io.ldb
sudo systemctl start sssd
sudo sssctl domain-status goauthentik.io
getent passwd -s sss
```
---
### Issue D: User Resolves in `getent passwd` But Cannot Log In on GDM / Laptop
1. **Check if the user appears on the GDM login screen:**
Verify `/var/lib/AccountsService/users/` exists and has `SystemAccount=false`:
```bash
sudo cat /var/lib/AccountsService/users/ikura
```
If missing, run the pre-provisioning snippet in **Step 5.2** above and restart `accounts-daemon`:
```bash
sudo systemctl restart accounts-daemon
```
2. **Test PAM authentication from a terminal (`su` / `login`):**
```bash
su - ikura
```
If `su - ikura` fails while `sudo su - ikura` works, inspect `/var/log/sssd/sssd_pam.log` and `journalctl -n 40 --no-pager`:
* Ensure `/etc/pam.d/system-auth` contains `pam_sss.so` in `auth`, `account`, `password`, and `session`.
* Ensure the user is active in Authentik and their password was set in Authentik (`pwdChangedTime` present on their LDAP entry).
# SSH CA setup
### OVERVIEW & ARCHITECTURE
* **Authentication Flow:** Incoming SSH clients authenticate using an OpenSSH user certificate signed by the server's User Certificate Authority (CA). Similarly, clients verify host identity using an OpenSSH host certificate signed by the Host CA.
* **LDAP & Authentik Integration:** Host **ul** derives user accounts from Authentik via LDAP (managed via SSSD). When a client presents an SSH certificate, `sshd` validates the certificate signature against the trusted CA public key and checks that one of the certificate's valid principals matches the requested LDAP login username. No per-user `~/.ssh/authorized_keys` file is required.
* **Key Storage & Backup:**
* The User CA private key (`drheck_ca_user_key`) and Host CA private key (`drheck_ca_host_key`) must **never** reside on unencrypted client machines or internet-facing servers.
* Store the offline CA private keys and passphrases securely in Vaultwarden under account `akadmin` (`admin@drheck.dev`).
* Deploy only the CA public keys (`drheck_ca_user_key.pub` and `drheck_ca_host_key.pub`) to host **ul** and client machines.
* **Operational Separation:**
* **Architecture & Foundational CA Setup:** Documented on this page.
* **Day-to-Day Client & Host Onboarding:** See the dedicated operational runbook: [SSH Client and Host Onboarding Guide](https://wiki.drheck.dev/books/ldappamssh-ca/page/ssh-client-and-host-onboarding-guide).
---
### PHASE 1: CA GENERATION & VAULTWARDEN BACKUP
Generate the standard Ed25519 CA Keypairs on a secure offline or administrative workstation:
*When prompted by `ssh-keygen`, enter a strong, unique passphrase to encrypt each CA private key.*
```bash
# Generate the User CA keypair
ssh-keygen -t ed25519 -f drheck_ca_user_key -C "drheck-user-ca"
# Generate the Host CA keypair
ssh-keygen -t ed25519 -f drheck_ca_host_key -C "drheck-host-ca"
```
#### Storage in Vaultwarden:
1. Create a secure item in Vaultwarden under `akadmin` (`admin@drheck.dev`):
- `drheck_ca_user_key` (multi-line private key in Notes field)
- `drheck_ca_user_key_pw` (passphrase)
- `drheck_ca_host_key` (multi-line private key in Notes field)
- `drheck_ca_host_key_pw` (passphrase)
2. Remove the private CA keys from the unencrypted local disk of your workstation after verifying the Vaultwarden backup.
---
### PHASE 2: MAIN SERVER CONFIGURATION (ul)
#### 1. Deploy Trusted User CA Public Key
* Copy the public CA file `drheck_ca_user_key.pub` to host **ul** at `/etc/ssh/drheck_ca_user_key.pub`.
* Set secure permissions:
```bash
sudo chmod 644 /etc/ssh/drheck_ca_user_key.pub
sudo chown root:root /etc/ssh/drheck_ca_user_key.pub
```
#### 2. Configure Host Certificate (Signed by Host CA)
1. Copy **ul**'s public host key `/etc/ssh/ssh_host_ed25519_key.pub` to your signing environment.
2. Sign the host key using the vaulted `drheck_ca_host_key` (defining all valid FQDNs and IP addresses as principals):
```bash
ssh-keygen -s drheck_ca_host_key -I "drheck-ul-host-cert" -h -n ul,ul.drheck.dev,10.0.0.3,fd77:7777:5010:0:1337:f1ac:d00d:337 -V +52w /etc/ssh/ssh_host_ed25519_key.pub
```
3. Place the resulting certificate at `/etc/ssh/ssh_host_ed25519_key-cert.pub`.
4. Set secure permissions:
```bash
sudo chmod 644 /etc/ssh/ssh_host_ed25519_key-cert.pub
sudo chown root:root /etc/ssh/ssh_host_ed25519_key-cert.pub
```
#### 3. Configure OpenSSH Daemon on ul
* Create or edit `/etc/ssh/sshd_config.d/10-drheck-certs.conf`:
```text
TrustedUserCAKeys /etc/ssh/drheck_ca_user_key.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
```
#### 4. Validate and Reload SSHD on ul
* Validate the configuration syntax before reloading:
```bash
sudo sshd -t
```
* Reload the daemon:
```bash
sudo systemctl reload sshd
```
---
### PHASE 3: AUDITING & KEY REVOCATION (KRL)
OpenSSH natively supports Key Revocation Lists (KRL) to block compromised keys or expired certificates before their expiration timestamp.
#### 1. Key Inventory Architecture
Every key signed by the signing utilities is automatically archived under:
* `/etc/ssh/ca_inventory/hosts/