LDAP+PAM+SSH CA LDAP + SSSD Client Enrollment Guide How to enroll Linux machines into the drheck.dev Authentik LDAP directory service using SSSD, with offline credential caching, dual-stack LAN failover, and GDM graphical login integration. 1. Architecture & Authentication Pipeline $$ \text{Client PAM / NSS} \quad \longrightarrow \quad \text{SSSD Backend (\texttt{sssd})} \quad \longrightarrow \quad \text{HAProxy LDAPS (\texttt{:636})} \quad \longrightarrow \quad \text{Rootless Outpost (\texttt{authentik-ldap :6636})} $$ Parameter Value Notes Primary LDAPS URI (IPv6 ULA) ldaps://ldap.drheck.dev:636 Resolves to [fd77:7777:5010:0:1337:f1ac:d00d:aaaa]:636 on ul ( eth77, relayed via HAProxy TCP passthrough /etc/haproxy/conf.d/ldap.cfg to rootless authentik-ldap on :6636) Backup LDAPS URI (Dual-Stack IPv4/IPv6) ldaps://ul.drheck.dev:636 Resolves to 10.0.0.3:636 (IPv4) and [fd77:7777:5010:0:1337:f1ac:d00d:337]:636 (relayed via HAProxy TCP passthrough) Base DN dc=ldap,dc=goauthentik,dc=io Authentik LDAP Provider ( bind_mode=cached, search_mode=cached) Linux User Search Base & Filter ou=users,dc=ldap,dc=goauthentik,dc=io?sub?(memberOf=cn=locallogin,ou=groups,dc=ldap,dc=goauthentik,dc=io) Restricts enumeration & login to members of locallogin ( luke, haruka, ikura), excluding internal service accounts Bind Service Account DN cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io Requires search_full_directory object permission on the Authentik ldap provider Local SSSD Cache DB /var/lib/sss/db/cache_goauthentik.io.ldb Persists enumerated users & password hashes ( cache_credentials = true) for offline login Offline & LAN Resilience: SSSD is configured with enumerate = true, ldap_enumeration_refresh_offset = 0, and cache_credentials = true. Immediately upon enrollment on the local network, SSSD enumerates and caches all members of locallogin into /var/lib/sss/db/cache_goauthentik.io.ldb and pre-populates /var/lib/AccountsService/users/ so graphical GDM logins and getent passwd work even when the internet is offline. 2. Enrollment Steps Step 0: Automated One-Liner ( ldap_deploy.py) Run the enrollment script from a local administrator account with sudo: git clone -b main https://git.drheck.dev/luke/ldap_deploy.git /tmp/ldap_deploy && sudo python3 /tmp/ldap_deploy/ldap_deploy.py && rm -rf /tmp/ldap_deploy The automated installer performs all steps below, verifies live LDAPS TLS connectivity ( ldap.drheck.dev:636 / ul.drheck.dev:636), confirms sssctl domain-status goauthentik.io is Online, verifies that LDAP users resolve via getent passwd -s sss, and pre-provisions GDM AccountsService profiles. Step 1: Install SSSD & OpenLDAP Client Tools Install both sssd and the OpenLDAP client utilities ( openldap on Arch Linux, ldap-utils on Debian/Ubuntu) so ldapsearch and sssctl are available for offline diagnostics: Arch Linux: sudo pacman -S --needed sssd openldap Debian / Ubuntu: sudo apt install sssd sssd-tools sssd-ldap libnss-sss libpam-sss ldap-utils Installing When Internet Is Down (Arch Linux): On Arch Linux, the sssd package depends on libldap (shared libraries) rather than openldap (CLI tools like ldapsearch). If the internet is down and a package is already in the local pacman cache, install it directly with sudo pacman -U /var/cache/pacman/pkg/sssd-*.pkg.tar.zst. If ldapsearch is not installed, use the built-in openssl s_client and sssctl diagnostic commands in Section 4 below. Step 2: Configure SSSD ( /etc/sssd/sssd.conf) Create backups of existing authentication configs: sudo cp /etc/nsswitch.conf /etc/nsswitch.conf.bak sudo cp /etc/pam.d/system-auth /etc/pam.d/system-auth.bak Create /etc/sssd/sssd.conf: sudo vim /etc/sssd/sssd.conf Paste the following configuration: [sssd] services = nss, pam config_file_version = 2 domains = goauthentik.io debug_level = 1 [nss] debug_level = 1 [pam] debug_level = 1 [domain/goauthentik.io] debug_level = 1 id_provider = ldap auth_provider = ldap sudo_provider = none ldap_uri = ldaps://ldap.drheck.dev:636 ldap_backup_uri = ldaps://ul.drheck.dev:636 ldap_id_use_start_tls = false ldap_search_base = dc=ldap,dc=goauthentik,dc=io ldap_user_search_base = ou=users,dc=ldap,dc=goauthentik,dc=io?sub?(memberOf=cn=locallogin,ou=groups,dc=ldap,dc=goauthentik,dc=io) ldap_default_bind_dn = cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io ldap_default_authtok = hgNbhl3t1hzyV4jdfsiQuAeYS # Secure Production TLS Configuration (Let's Encrypt) ldap_tls_reqcert = demand ldap_tls_cacertdir = /etc/ssl/certs ldap_schema = rfc2307bis ldap_user_object_class = posixAccount ldap_user_name = sAMAccountName ldap_user_uid_number = uidNumber ldap_user_gid_number = gidNumber ldap_user_home_directory = homeDirectory ldap_user_shell = loginShell ldap_group_object_class = posixGroup ldap_group_name = cn ldap_group_gid_number = gidNumber ldap_group_member = member ldap_ignore_unreadable_references = true default_shell = /bin/bash fallback_homedir = /home/%u enumerate = true ldap_enumeration_refresh_offset = 0 ldap_entry_usn = entryUSN cache_credentials = true Set strict ownership and 0600 permissions (SSSD refuses to start if /etc/sssd/sssd.conf is group- or world-readable): sudo chmod 600 /etc/sssd/sssd.conf sudo chown root:root /etc/sssd/sssd.conf Step 3: Configure NSS ( /etc/nsswitch.conf) Open /etc/nsswitch.conf: sudo vim /etc/nsswitch.conf Ensure sss is appended to passwd, group, and shadow: passwd: compat mymachines systemd sss group: compat mymachines systemd sss shadow: compat systemd sss Step 4: Configure PAM ( /etc/pam.d/system-auth, su, su-l) Open /etc/pam.d/system-auth: sudo vim /etc/pam.d/system-auth Replace the contents with the SSSD + pam_mkhomedir.so PAM stack: Automatic Home Directory Creation: The pam_mkhomedir.so module in the session block automatically creates /home/ on first login, copying skeleton files from /etc/skel/ with umask=0022 ( 0755 permissions). #%PAM-1.0 auth required pam_faillock.so preauth auth sufficient pam_sss.so forward_pass auth [success=1 default=bad] pam_unix.so try_first_pass nullok auth [default=die] pam_faillock.so authfail auth optional pam_permit.so auth required pam_env.so auth required pam_faillock.so authsucc account sufficient pam_sss.so account required pam_unix.so account optional pam_permit.so account required pam_time.so password sufficient pam_sss.so password required pam_unix.so try_first_pass nullok sha512 shadow password optional pam_permit.so session required pam_limits.so session required pam_unix.so session required pam_mkhomedir.so skel=/etc/skel/ umask=0022 session optional pam_exec.so seteuid /usr/local/bin/gdm-ldap-user-session.sh session optional pam_sss.so session optional pam_permit.so Ensure /etc/pam.d/su and /etc/pam.d/su-l include system-auth so su - works from a terminal: #%PAM-1.0 auth sufficient pam_rootok.so auth include system-auth account include system-auth session include system-auth password include system-auth Step 5: Unlock GDM Login Screen & Session Selector ( AccountsService) Arch Linux compiles accountsservice without native SSSD enumeration support (see NSS + GDM Issue). Without a profile in /var/lib/AccountsService/users/, GDM may hide remote LDAP users from the login list or hide the desktop session selector (gear icon). 1. Create the PAM Auto-Provisioning Script ( /usr/local/bin/gdm-ldap-user-session.sh) sudo tee /usr/local/bin/gdm-ldap-user-session.sh >/dev/null << 'EOS' #!/bin/bash # Automatically provision AccountsService files for remote SSSD/LDAP users # on their first login, unlocking GDM session selection on Arch Linux. USER="$PAM_USER" if [ -z "$USER" ]; then exit 0 fi # 1. Skip local users (already in /etc/passwd) if ! grep -q "^$USER:" /etc/passwd && id "$USER" &>/dev/null; then UID_NUM=$(id -u "$USER") # 2. Only process standard users (UID >= 1000) if [ "$UID_NUM" -ge 1000 ]; then FILE="/var/lib/AccountsService/users/$USER" if [ ! -f "$FILE" ]; then cat < "$FILE" [User] Session=gnome Icon=/home/$USER/.face SystemAccount=false EOF chown root:root "$FILE" chmod 600 "$FILE" fi fi fi EOS sudo chmod 755 /usr/local/bin/gdm-ldap-user-session.sh 2. Pre-Provision Existing LDAP Users Immediately (Before First Login) Once SSSD is started in Step 6, run the following command (handled automatically by ldap_deploy.py) to pre-populate /var/lib/AccountsService/users/ for all enumerated LDAP users so they appear on the GDM login screen immediately without needing a prior terminal login: if [ -d /var/lib/AccountsService/users ]; then for u in $(getent passwd -s sss | awk -F: '$3 >= 2000 {print $1}'); do if ! grep -q "^${u}:" /etc/passwd && [ ! -f "/var/lib/AccountsService/users/${u}" ]; then sudo tee "/var/lib/AccountsService/users/${u}" >/dev/null <&1 | grep -E "CONNECTED|subject=|issuer=|Verify return code" Or test over IPv4 via ul.drheck.dev:636: openssl s_client -connect 10.0.0.3:636 -servername ul.drheck.dev &1 | grep -E "CONNECTED|subject=|issuer=|Verify return code" Expected output: CONNECTED(00000003) subject=CN=drheck.dev issuer=C=US, O=Let's Encrypt, CN=YE1 Verify return code: 0 (ok) If Verify return code is non-zero or clock is skewed: Check timedatectl status on the client laptop (if the CMOS clock is wrong when offline, TLS certificate validation against Let's Encrypt will fail). Step 3: Check for Invalid credentials (49) / Bind Password Mismatch If openssl s_client succeeds with Verify return code: 0 (ok), check /var/log/sssd/sssd_goauthentik.io.log (with debug level 6 if needed): sudo sssctl debug-level 6 sudo systemctl restart sssd sleep 3 sudo grep -i -E "bind|credential|tls|error|fail|offline" /var/log/sssd/sssd_goauthentik.io.log | tail -n 30 sudo sssctl debug-level 1 If openldap ( ldapsearch) is installed, test the ldapservice bind directly: LDAPTLS_REQCERT=demand ldapsearch -LLL -x \ -H ldaps://ldap.drheck.dev:636 \ -D "cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io" \ -w "hgNbhl3t1hzyV4jdfsiQuAeYS" \ -b "ou=users,dc=ldap,dc=goauthentik,dc=io" "(cn=ikura)" cn uidNumber gidNumber If ldapsearch returns ldap_bind: Invalid credentials (49) (or sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u authentik-ldap.service on ul shows "error":"flow error password: Invalid password" for cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io): The ldapservice password in Authentik on ul does not match ldap_default_authtok in /etc/sssd/sssd.conf (or another service overwrote ldapservice's password instead of using its own dedicated service account). To verify and restore ldapservice on ul: ssh ul '(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -i authentik-server ak shell -c " from authentik.core.models import User from authentik.providers.ldap.models import LDAPProvider p = LDAPProvider.objects.first() u = User.objects.get(username=\"ldapservice\") print(\"Password match:\", u.check_password(\"hgNbhl3t1hzyV4jdfsiQuAeYS\")) print(\"LDAPProvider perms:\", u.get_all_permissions(p)) " 2>/dev/null)' If Password match: is False, restore it on ul and restart authentik-ldap.service so its cached bind session is flushed: ssh ul '(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -i authentik-server ak shell -c " from authentik.core.models import User u = User.objects.get(username=\"ldapservice\") u.set_password(\"hgNbhl3t1hzyV4jdfsiQuAeYS\") u.save() " 2>/dev/null) && sudo systemctl --user -M docker-data@ restart authentik-ldap.service' Issue C: Flushing Stale SSSD Cache & Forcing SSSD Online If SSSD marked the domain offline during a temporary network glitch, force it back online and clear the cache: sudo systemctl stop sssd sudo rm -f /var/lib/sss/db/cache_goauthentik.io.ldb sudo systemctl start sssd sudo sssctl domain-status goauthentik.io getent passwd -s sss Issue D: User Resolves in getent passwd But Cannot Log In on GDM / Laptop Check if the user appears on the GDM login screen: Verify /var/lib/AccountsService/users/ exists and has SystemAccount=false: sudo cat /var/lib/AccountsService/users/ikura If missing, run the pre-provisioning snippet in Step 5.2 above and restart accounts-daemon: sudo systemctl restart accounts-daemon Test PAM authentication from a terminal ( su / login): su - ikura If su - ikura fails while sudo su - ikura works, inspect /var/log/sssd/sssd_pam.log and journalctl -n 40 --no-pager: Ensure /etc/pam.d/system-auth contains pam_sss.so in auth, account, password, and session. Ensure the user is active in Authentik and their password was set in Authentik ( pwdChangedTime present on their LDAP entry). SSH CA setup OVERVIEW & ARCHITECTURE Authentication Flow: Incoming SSH clients authenticate using an OpenSSH user certificate signed by the server's User Certificate Authority (CA). Similarly, clients verify host identity using an OpenSSH host certificate signed by the Host CA. LDAP & Authentik Integration: Host ul derives user accounts from Authentik via LDAP (managed via SSSD). When a client presents an SSH certificate, sshd validates the certificate signature against the trusted CA public key and checks that one of the certificate's valid principals matches the requested LDAP login username. No per-user ~/.ssh/authorized_keys file is required. Key Storage & Backup: The User CA private key ( drheck_ca_user_key) and Host CA private key ( drheck_ca_host_key) must never reside on unencrypted client machines or internet-facing servers. Store the offline CA private keys and passphrases securely in Vaultwarden under account akadmin ( admin@drheck.dev). Deploy only the CA public keys ( drheck_ca_user_key.pub and drheck_ca_host_key.pub) to host ul and client machines. Operational Separation: Architecture & Foundational CA Setup: Documented on this page. Day-to-Day Client & Host Onboarding: See the dedicated operational runbook: SSH Client and Host Onboarding Guide. PHASE 1: CA GENERATION & VAULTWARDEN BACKUP Generate the standard Ed25519 CA Keypairs on a secure offline or administrative workstation: When prompted by ssh-keygen, enter a strong, unique passphrase to encrypt each CA private key. # Generate the User CA keypair ssh-keygen -t ed25519 -f drheck_ca_user_key -C "drheck-user-ca" # Generate the Host CA keypair ssh-keygen -t ed25519 -f drheck_ca_host_key -C "drheck-host-ca" Storage in Vaultwarden: Create a secure item in Vaultwarden under akadmin ( admin@drheck.dev): drheck_ca_user_key (multi-line private key in Notes field) drheck_ca_user_key_pw (passphrase) drheck_ca_host_key (multi-line private key in Notes field) drheck_ca_host_key_pw (passphrase) Remove the private CA keys from the unencrypted local disk of your workstation after verifying the Vaultwarden backup. PHASE 2: MAIN SERVER CONFIGURATION (ul) 1. Deploy Trusted User CA Public Key Copy the public CA file drheck_ca_user_key.pub to host ul at /etc/ssh/drheck_ca_user_key.pub. Set secure permissions: sudo chmod 644 /etc/ssh/drheck_ca_user_key.pub sudo chown root:root /etc/ssh/drheck_ca_user_key.pub 2. Configure Host Certificate (Signed by Host CA) Copy ul's public host key /etc/ssh/ssh_host_ed25519_key.pub to your signing environment. Sign the host key using the vaulted drheck_ca_host_key (defining all valid FQDNs and IP addresses as principals): ssh-keygen -s drheck_ca_host_key -I "drheck-ul-host-cert" -h -n ul,ul.drheck.dev,10.0.0.3,fd77:7777:5010:0:1337:f1ac:d00d:337 -V +52w /etc/ssh/ssh_host_ed25519_key.pub Place the resulting certificate at /etc/ssh/ssh_host_ed25519_key-cert.pub. Set secure permissions: sudo chmod 644 /etc/ssh/ssh_host_ed25519_key-cert.pub sudo chown root:root /etc/ssh/ssh_host_ed25519_key-cert.pub 3. Configure OpenSSH Daemon on ul Create or edit /etc/ssh/sshd_config.d/10-drheck-certs.conf: TrustedUserCAKeys /etc/ssh/drheck_ca_user_key.pub HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub 4. Validate and Reload SSHD on ul Validate the configuration syntax before reloading: sudo sshd -t Reload the daemon: sudo systemctl reload sshd PHASE 3: AUDITING & KEY REVOCATION (KRL) OpenSSH natively supports Key Revocation Lists (KRL) to block compromised keys or expired certificates before their expiration timestamp. 1. Key Inventory Architecture Every key signed by the signing utilities is automatically archived under: /etc/ssh/ca_inventory/hosts/