LDAP + SSSD Client Enrollment Guide

How to enroll Linux machines into the drheck.dev Authentik LDAP directory service using SSSD, with offline credential caching, dual-stack LAN failover, and GDM graphical login integration.

1. Architecture & Authentication Pipeline

$$ \text{Client PAM / NSS} \quad \longrightarrow \quad \text{SSSD Backend (\texttt{sssd})} \quad \longrightarrow \quad \text{HAProxy LDAPS (\texttt{:636})} \quad \longrightarrow \quad \text{~~Authentik~~Rootless Outpost (\texttt{authentik-~~ldap}~~ldap :6636})} $$

Parameter	Value	Notes
Primary LDAPS URI (IPv6 ULA)	`ldaps://ldap.drheck.dev:636`	Resolves to `[fd77:7777:5010:0:1337:f1ac:d00d:aaaa]:636` on `ul` (`eth77`, relayed via HAProxy TCP passthrough `/etc/haproxy/conf.d/ldap.cfg` to rootless `authentik-ldap` on `:6636`)
Backup LDAPS URI (Dual-Stack IPv4/IPv6)	`ldaps://ul.drheck.dev:636`	Resolves to `10.0.0.3:636` (IPv4) and `[fd77:7777:5010:0:1337:f1ac:d00d:337]:636` (relayed via HAProxy TCP passthrough)
Base DN	`dc=ldap,dc=goauthentik,dc=io`	Authentik LDAP Provider (`bind_mode=cached`, `search_mode=cached`)
Linux User Search Base & Filter	`ou=users,dc=ldap,dc=goauthentik,dc=io?sub?(memberOf=cn=locallogin,ou=groups,dc=ldap,dc=goauthentik,dc=io)`	Restricts enumeration & login to members of `locallogin` (`luke`, `haruka`, `ikura`), excluding internal service accounts
Bind Service Account DN	`cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io`	Requires `search_full_directory` object permission on the Authentik `ldap` provider
Local SSSD Cache DB	`/var/lib/sss/db/cache_goauthentik.io.ldb`	Persists enumerated users & password hashes (`cache_credentials = true`) for offline login

Offline & LAN Resilience: SSSD is configured with enumerate = true, ldap_enumeration_refresh_offset = 0, and cache_credentials = true. Immediately upon enrollment on the local network, SSSD enumerates and caches all members of locallogin into /var/lib/sss/db/cache_goauthentik.io.ldb and pre-populates /var/lib/AccountsService/users/ so graphical GDM logins and getent passwd work even when the internet is offline.

2. Enrollment Steps

Step 0: Automated One-Liner (`ldap_deploy.py`)

Run the enrollment script from a local administrator account with sudo:

git clone -b main https://git.drheck.dev/luke/ldap_deploy.git /tmp/ldap_deploy && sudo python3 /tmp/ldap_deploy/ldap_deploy.py && rm -rf /tmp/ldap_deploy

The automated installer performs all steps below, verifies live LDAPS TLS connectivity (ldap.drheck.dev:636 / ul.drheck.dev:636), confirms sssctl domain-status goauthentik.io is Online, verifies that LDAP users resolve via getent passwd -s sss, and pre-provisions GDM AccountsService profiles.

Step 1: Install SSSD & OpenLDAP Client Tools

Install both sssd and the OpenLDAP client utilities (openldap on Arch Linux, ldap-utils on Debian/Ubuntu) so ldapsearch and sssctl are available for offline diagnostics:

Arch Linux:
```
sudo pacman -S --needed sssd openldap
```

Debian / Ubuntu:

sudo apt install sssd sssd-tools sssd-ldap libnss-sss libpam-sss ldap-utils

Installing When Internet Is Down (Arch Linux): On Arch Linux, the sssd package depends on libldap (shared libraries) rather than openldap (CLI tools like ldapsearch). If the internet is down and a package is already in the local pacman cache, install it directly with sudo pacman -U /var/cache/pacman/pkg/sssd-*.pkg.tar.zst. If ldapsearch is not installed, use the built-in openssl s_client and sssctl diagnostic commands in Section 4 below.

Step 2: Configure SSSD (`/etc/sssd/sssd.conf`)

Create backups of existing authentication configs:

sudo cp /etc/nsswitch.conf /etc/nsswitch.conf.bak
sudo cp /etc/pam.d/system-auth /etc/pam.d/system-auth.bak

Create /etc/sssd/sssd.conf:

sudo vim /etc/sssd/sssd.conf

Paste the following configuration:

[sssd]
services = nss, pam
config_file_version = 2
domains = goauthentik.io
debug_level = 1

[nss]
debug_level = 1

[pam]
debug_level = 1

[domain/goauthentik.io]
debug_level = 1
id_provider = ldap
auth_provider = ldap
sudo_provider = none
ldap_uri = ldaps://ldap.drheck.dev:636
ldap_backup_uri = ldaps://ul.drheck.dev:636
ldap_id_use_start_tls = false
ldap_search_base = dc=ldap,dc=goauthentik,dc=io
ldap_user_search_base = ou=users,dc=ldap,dc=goauthentik,dc=io?sub?(memberOf=cn=locallogin,ou=groups,dc=ldap,dc=goauthentik,dc=io)
ldap_default_bind_dn = cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io
ldap_default_authtok = hgNbhl3t1hzyV4jdfsiQuAeYS

# Secure Production TLS Configuration (Let's Encrypt)
ldap_tls_reqcert = demand
ldap_tls_cacertdir = /etc/ssl/certs

ldap_schema = rfc2307bis
ldap_user_object_class = posixAccount
ldap_user_name = sAMAccountName
ldap_user_uid_number = uidNumber
ldap_user_gid_number = gidNumber
ldap_user_home_directory = homeDirectory
ldap_user_shell = loginShell
ldap_group_object_class = posixGroup
ldap_group_name = cn
ldap_group_gid_number = gidNumber
ldap_group_member = member
ldap_ignore_unreadable_references = true
default_shell = /bin/bash
fallback_homedir = /home/%u
enumerate = true
ldap_enumeration_refresh_offset = 0
ldap_entry_usn = entryUSN
cache_credentials = true

Set strict ownership and 0600 permissions (SSSD refuses to start if /etc/sssd/sssd.conf is group- or world-readable):

sudo chmod 600 /etc/sssd/sssd.conf
sudo chown root:root /etc/sssd/sssd.conf

Step 3: Configure NSS (`/etc/nsswitch.conf`)

Open /etc/nsswitch.conf:

sudo vim /etc/nsswitch.conf

Ensure sss is appended to passwd, group, and shadow:

passwd: compat mymachines systemd sss
group: compat mymachines systemd sss
shadow: compat systemd sss

Step 4: Configure PAM (`/etc/pam.d/system-auth`, `su`, `su-l`)

Open /etc/pam.d/system-auth:

sudo vim /etc/pam.d/system-auth

Replace the contents with the SSSD + pam_mkhomedir.so PAM stack:

Automatic Home Directory Creation: The pam_mkhomedir.so module in the session block automatically creates /home/<username> on first login, copying skeleton files from /etc/skel/ with umask=0022 (0755 permissions).

#%PAM-1.0

auth       required                    pam_faillock.so      preauth
auth       sufficient                  pam_sss.so           forward_pass
auth       [success=1 default=bad]     pam_unix.so          try_first_pass nullok
auth       [default=die]               pam_faillock.so      authfail
auth       optional                    pam_permit.so
auth       required                    pam_env.so
auth       required                    pam_faillock.so      authsucc

account    sufficient                  pam_sss.so
account    required                    pam_unix.so
account    optional                    pam_permit.so
account    required                    pam_time.so

password   sufficient                  pam_sss.so
password   required                    pam_unix.so          try_first_pass nullok sha512 shadow
password   optional                    pam_permit.so

session    required                    pam_limits.so
session    required                    pam_unix.so
session    required                    pam_mkhomedir.so     skel=/etc/skel/ umask=0022
session    optional                    pam_exec.so          seteuid /usr/local/bin/gdm-ldap-user-session.sh
session    optional                    pam_sss.so
session    optional                    pam_permit.so

Ensure /etc/pam.d/su and /etc/pam.d/su-l include system-auth so su - <ldap-user> works from a terminal:

#%PAM-1.0
auth            sufficient      pam_rootok.so
auth            include         system-auth
account         include         system-auth
session         include         system-auth
password        include         system-auth

Step 5: Unlock GDM Login Screen & Session Selector (`AccountsService`)

Arch Linux compiles accountsservice without native SSSD enumeration support (see NSS + GDM Issue). Without a profile in /var/lib/AccountsService/users/<username>, GDM may hide remote LDAP users from the login list or hide the desktop session selector (gear icon).

1. Create the PAM Auto-Provisioning Script (`/usr/local/bin/gdm-ldap-user-session.sh`)

sudo tee /usr/local/bin/gdm-ldap-user-session.sh >/dev/null << 'EOS'
#!/bin/bash
# Automatically provision AccountsService files for remote SSSD/LDAP users
# on their first login, unlocking GDM session selection on Arch Linux.

USER="$PAM_USER"
if [ -z "$USER" ]; then
    exit 0
fi

# 1. Skip local users (already in /etc/passwd)
if ! grep -q "^$USER:" /etc/passwd && id "$USER" &>/dev/null; then
    UID_NUM=$(id -u "$USER")

    # 2. Only process standard users (UID >= 1000)
    if [ "$UID_NUM" -ge 1000 ]; then
        FILE="/var/lib/AccountsService/users/$USER"
        if [ ! -f "$FILE" ]; then
            cat <<EOF > "$FILE"
[User]
Session=gnome
Icon=/home/$USER/.face
SystemAccount=false
EOF
            chown root:root "$FILE"
            chmod 600 "$FILE"
        fi
    fi
fi
EOS
sudo chmod 755 /usr/local/bin/gdm-ldap-user-session.sh

2. Pre-Provision Existing LDAP Users Immediately (Before First Login)

Once SSSD is started in Step 6, run the following command (handled automatically by ldap_deploy.py) to pre-populate /var/lib/AccountsService/users/<user> for all enumerated LDAP users so they appear on the GDM login screen immediately without needing a prior terminal login:

if [ -d /var/lib/AccountsService/users ]; then
    for u in $(getent passwd -s sss | awk -F: '$3 >= 2000 {print $1}'); do
        if ! grep -q "^${u}:" /etc/passwd && [ ! -f "/var/lib/AccountsService/users/${u}" ]; then
            sudo tee "/var/lib/AccountsService/users/${u}" >/dev/null <<EOF
[User]
Session=gnome
Icon=/home/${u}/.face
SystemAccount=false
EOF
            sudo chmod 600 "/var/lib/AccountsService/users/${u}"
        fi
    done
fi

Step 6: Start SSSD & Verify Before Logging Out

Clear any stale SSSD cache from previous attempts, then enable and start sssd:

sudo systemctl stop sssd
sudo rm -f /var/lib/sss/db/cache_goauthentik.io.ldb
sudo systemctl enable --now sssd

3. Verification Checklist (Run Before Logging Out)

Verify SSSD Backend Online Status:

sudo sssctl domain-status goauthentik.io

Expected output:

Online status: Online

Active servers:
LDAP: ldap.drheck.dev

Discovered LDAP servers:
- ldap.drheck.dev
- ul.drheck.dev

Verify SSSD LDAP User Enumeration: Query only the sss NSS module to list all LDAP POSIX users resolved from Authentik:
```
getent passwd -s sss
```
Expected output:
```
akadmin:*:2004:2004:akadmin:/home/akadmin:/bin/bash
brett:*:2024:2024:brett:/home/brett:/bin/bash
ikura:*:2022:2022:ikura:/home/ikura:/bin/bash
haruka:*:2021:2021:haruka:/home/haruka:/bin/bash
luke:*:2013:2013:luke:/home/luke:/bin/bash
```
(With enumerate = true, bare getent passwd also includes all five lines above at the bottom of the output. If enumerate = false was used on an older config, bare getent passwd only shows local /etc/passwd users and you must pass a username explicitly, e.g. getent passwd ikura.)
Verify UID/GID and Group Memberships:
```
id ikura
id haruka
```
Verify PAM Authentication & Automatic Home Directory Creation:
```
sudo su - ikura -c "pwd && id"
```
Or test full password authentication via SSH/PAM:
```
ssh ikura@localhost
```

4. Offline & LAN Troubleshooting Runbook

No Internet Required: All troubleshooting steps below work completely offline on the local LAN using standard system binaries (getent, sssctl, openssl, python3, journalctl) even if ldapsearch (openldap) is not installed.

Issue A: `getent passwd` Only Shows `root` and Local User (`luke`), Nothing for LDAP Users

Test Direct Username Lookup vs. Enumeration:
```
getent passwd -s sss ikura
getent passwd -s sss haruka
```
- If getent passwd -s sss ikura returns a line (ikura:*:2022:...) while bare getent passwd does not: /etc/sssd/sssd.conf has enumerate = false (or the initial 10-second enumeration delay has not finished). Set enumerate = true and ldap_enumeration_refresh_offset = 0 in /etc/sssd/sssd.conf and restart sssd.
- If getent passwd -s sss ikura returns nothing (exit code 2): SSSD is failing to reach or bind to ldap.drheck.dev:636. Proceed to Issue B below.

Issue B: SSSD Domain Is `Offline` or Cannot Resolve Any LDAP Users

Check SSSD's domain status and backend log:

sudo sssctl domain-status goauthentik.io
sudo tail -n 35 /var/log/sssd/sssd_goauthentik.io.log

Step 1: Verify DNS Resolution (`ldap.drheck.dev` and `ul.drheck.dev`)

ldap.drheck.dev resolves to ul's ULA IPv6 address (fd77:7777:5010:0:1337:f1ac:d00d:aaaa), while ul.drheck.dev resolves to both IPv4 (10.0.0.3) and IPv6 (fd77:7777:5010:0:1337:f1ac:d00d:337):

getent ahosts ldap.drheck.dev
getent ahosts ul.drheck.dev

If the client machine does not have a local ULA IPv6 address (ip -6 addr show), ensure ldap_backup_uri = ldaps://ul.drheck.dev:636 is present in /etc/sssd/sssd.conf so SSSD connects over IPv4 (10.0.0.3:636).

Step 2: Test LDAPS TCP & TLS Certificate Handshake (Without `ldapsearch`)

Use openssl (installed on all base systems) to verify that port 636 on ul is reachable and presenting a valid Let's Encrypt certificate for *.drheck.dev:

openssl s_client -connect ldap.drheck.dev:636 -servername ldap.drheck.dev </dev/null 2>&1 | grep -E "CONNECTED|subject=|issuer=|Verify return code"

Or test over IPv4 via ul.drheck.dev:636:

openssl s_client -connect 10.0.0.3:636 -servername ul.drheck.dev </dev/null 2>&1 | grep -E "CONNECTED|subject=|issuer=|Verify return code"

Expected output:

CONNECTED(00000003)
subject=CN=drheck.dev
issuer=C=US, O=Let's Encrypt, CN=YE1
    Verify return code: 0 (ok)

If Verify return code is non-zero or clock is skewed: Check timedatectl status on the client laptop (if the CMOS clock is wrong when offline, TLS certificate validation against Let's Encrypt will fail).

Step 3: Check for `Invalid credentials (49)` / Bind Password Mismatch

If openssl s_client succeeds with Verify return code: 0 (ok), check /var/log/sssd/sssd_goauthentik.io.log (with debug level 6 if needed):

sudo sssctl debug-level 6
sudo systemctl restart sssd
sleep 3
sudo grep -i -E "bind|credential|tls|error|fail|offline" /var/log/sssd/sssd_goauthentik.io.log | tail -n 30
sudo sssctl debug-level 1

If openldap (ldapsearch) is installed, test the ldapservice bind directly:

LDAPTLS_REQCERT=demand ldapsearch -LLL -x \
  -H ldaps://ldap.drheck.dev:636 \
  -D "cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io" \
  -w "hgNbhl3t1hzyV4jdfsiQuAeYS" \
  -b "ou=users,dc=ldap,dc=goauthentik,dc=io" "(cn=ikura)" cn uidNumber gidNumber

If ldapsearch returns ldap_bind: Invalid credentials (49) (or dockersudo logs-u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u authentik-ldapldap.service on ul shows "error":"flow error password: Invalid password" for cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io): The ldapservice password in Authentik on ul does not match ldap_default_authtok in /etc/sssd/sssd.conf (or another service overwrote ldapservice's password instead of using its own dedicated service account). To verify and restore ldapservice on ul:

ssh ul 'docker(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -i authentik-server ak shell -c "
from authentik.core.models import User
from authentik.providers.ldap.models import LDAPProvider
p = LDAPProvider.objects.first()
u = User.objects.get(username=\"ldapservice\")
print(\"Password match:\", u.check_password(\"hgNbhl3t1hzyV4jdfsiQuAeYS\"))
print(\"LDAPProvider perms:\", u.get_all_permissions(p))
" 2>/dev/null'null)'

If Password match: is False, restore it on ul and restart authentik-ldapldap.service so its cached bind session is flushed:

ssh ul 'docker(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -i authentik-server ak shell -c "
from authentik.core.models import User
u = User.objects.get(username=\"ldapservice\")
u.set_password(\"hgNbhl3t1hzyV4jdfsiQuAeYS\")
u.save()
" 2>/dev/nullnull) && dockersudo systemctl --user -M docker-data@ restart authentik-ldap'ldap.service'

Issue C: Flushing Stale SSSD Cache & Forcing SSSD Online

If SSSD marked the domain offline during a temporary network glitch, force it back online and clear the cache:

sudo systemctl stop sssd
sudo rm -f /var/lib/sss/db/cache_goauthentik.io.ldb
sudo systemctl start sssd
sudo sssctl domain-status goauthentik.io
getent passwd -s sss

Issue D: User Resolves in `getent passwd` But Cannot Log In on GDM / Laptop

Check if the user appears on the GDM login screen: Verify /var/lib/AccountsService/users/<username> exists and has SystemAccount=false:
```
sudo cat /var/lib/AccountsService/users/ikura
```
If missing, run the pre-provisioning snippet in Step 5.2 above and restart accounts-daemon:
```
sudo systemctl restart accounts-daemon
```
Test PAM authentication from a terminal (su / login):
```
su - ikura
```
If su - ikura fails while sudo su - ikura works, inspect /var/log/sssd/sssd_pam.log and journalctl -n 40 --no-pager:
- Ensure /etc/pam.d/system-auth contains pam_sss.so in auth, account, password, and session.
- Ensure the user is active in Authentik and their password was set in Authentik (pwdChangedTime present on their LDAP entry).

LDAP + SSSD Client Enrollment Guide

1. Architecture & Authentication Pipeline

2. Enrollment Steps

Step 0: Automated One-Liner (ldap_deploy.py)

Step 1: Install SSSD & OpenLDAP Client Tools

Step 2: Configure SSSD (/etc/sssd/sssd.conf)

Step 3: Configure NSS (/etc/nsswitch.conf)

Step 4: Configure PAM (/etc/pam.d/system-auth, su, su-l)

Step 5: Unlock GDM Login Screen & Session Selector (AccountsService)

1. Create the PAM Auto-Provisioning Script (/usr/local/bin/gdm-ldap-user-session.sh)

2. Pre-Provision Existing LDAP Users Immediately (Before First Login)

Step 6: Start SSSD & Verify Before Logging Out

3. Verification Checklist (Run Before Logging Out)

4. Offline & LAN Troubleshooting Runbook

Issue A: getent passwd Only Shows root and Local User (luke), Nothing for LDAP Users

Issue B: SSSD Domain Is Offline or Cannot Resolve Any LDAP Users

Step 1: Verify DNS Resolution (ldap.drheck.dev and ul.drheck.dev)

Step 2: Test LDAPS TCP & TLS Certificate Handshake (Without ldapsearch)

Step 3: Check for Invalid credentials (49) / Bind Password Mismatch

Issue C: Flushing Stale SSSD Cache & Forcing SSSD Online

Issue D: User Resolves in getent passwd But Cannot Log In on GDM / Laptop

Step 0: Automated One-Liner (`ldap_deploy.py`)

Step 2: Configure SSSD (`/etc/sssd/sssd.conf`)

Step 3: Configure NSS (`/etc/nsswitch.conf`)

Step 4: Configure PAM (`/etc/pam.d/system-auth`, `su`, `su-l`)

Step 5: Unlock GDM Login Screen & Session Selector (`AccountsService`)

1. Create the PAM Auto-Provisioning Script (`/usr/local/bin/gdm-ldap-user-session.sh`)

Issue A: `getent passwd` Only Shows `root` and Local User (`luke`), Nothing for LDAP Users

Issue B: SSSD Domain Is `Offline` or Cannot Resolve Any LDAP Users

Step 1: Verify DNS Resolution (`ldap.drheck.dev` and `ul.drheck.dev`)

Step 2: Test LDAPS TCP & TLS Certificate Handshake (Without `ldapsearch`)

Step 3: Check for `Invalid credentials (49)` / Bind Password Mismatch

Issue D: User Resolves in `getent passwd` But Cannot Log In on GDM / Laptop