# LDAP + SSSD Client Enrollment Guide

How to enroll Linux machines into the `drheck.dev` Authentik LDAP directory service using SSSD, with offline credential caching, dual-stack LAN failover, and GDM graphical login integration.

---

## 1. Architecture & Authentication Pipeline

$$
\text{Client PAM / NSS} \quad \longrightarrow \quad \text{SSSD Backend (\texttt{sssd})} \quad \longrightarrow \quad \text{HAProxy LDAPS (\texttt{:636})} \quad \longrightarrow \quad \text{Rootless Outpost (\texttt{authentik-ldap :6636})}
$$

| Parameter | Value | Notes |
| :--- | :--- | :--- |
| **Primary LDAPS URI (IPv6 ULA)** | `ldaps://ldap.drheck.dev:636` | Resolves to `[fd77:7777:5010:0:1337:f1ac:d00d:aaaa]:636` on `ul` (`eth77`, relayed via HAProxy TCP passthrough `/etc/haproxy/conf.d/ldap.cfg` to rootless `authentik-ldap` on `:6636`) |
| **Backup LDAPS URI (Dual-Stack IPv4/IPv6)** | `ldaps://ul.drheck.dev:636` | Resolves to `10.0.0.3:636` (IPv4) and `[fd77:7777:5010:0:1337:f1ac:d00d:337]:636` (relayed via HAProxy TCP passthrough) |
| **Base DN** | `dc=ldap,dc=goauthentik,dc=io` | Authentik LDAP Provider (`bind_mode=cached`, `search_mode=cached`) |
| **Linux User Search Base & Filter** | `ou=users,dc=ldap,dc=goauthentik,dc=io?sub?(memberOf=cn=locallogin,ou=groups,dc=ldap,dc=goauthentik,dc=io)` | Restricts enumeration & login to members of `locallogin` (`luke`, `haruka`, `ikura`), excluding internal service accounts |
| **Bind Service Account DN** | `cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io` | Requires `search_full_directory` object permission on the Authentik `ldap` provider |
| **Local SSSD Cache DB** | `/var/lib/sss/db/cache_goauthentik.io.ldb` | Persists enumerated users & password hashes (`cache_credentials = true`) for offline login |

<p class="callout info"><strong>Offline &amp; LAN Resilience:</strong> SSSD is configured with <code>enumerate = true</code>, <code>ldap_enumeration_refresh_offset = 0</code>, and <code>cache_credentials = true</code>. Immediately upon enrollment on the local network, SSSD enumerates and caches all members of <code>locallogin</code> into <code>/var/lib/sss/db/cache_goauthentik.io.ldb</code> and pre-populates <code>/var/lib/AccountsService/users/</code> so graphical GDM logins and <code>getent passwd</code> work even when the internet is offline.</p>

---

## 2. Enrollment Steps

### Step 0: Automated One-Liner (`ldap_deploy.py`)

Run the enrollment script from a local administrator account with `sudo`:

```shell
git clone -b main https://git.drheck.dev/luke/ldap_deploy.git /tmp/ldap_deploy && sudo python3 /tmp/ldap_deploy/ldap_deploy.py && rm -rf /tmp/ldap_deploy
```

The automated installer performs all steps below, verifies live LDAPS TLS connectivity (`ldap.drheck.dev:636` / `ul.drheck.dev:636`), confirms `sssctl domain-status goauthentik.io` is `Online`, verifies that LDAP users resolve via `getent passwd -s sss`, and pre-provisions GDM `AccountsService` profiles.

---

### Step 1: Install SSSD & OpenLDAP Client Tools

Install both `sssd` and the OpenLDAP client utilities (`openldap` on Arch Linux, `ldap-utils` on Debian/Ubuntu) so `ldapsearch` and `sssctl` are available for offline diagnostics:

* **Arch Linux:**
  ```bash
  sudo pacman -S --needed sssd openldap
  ```
* **Debian / Ubuntu:**
  ```bash
  sudo apt install sssd sssd-tools sssd-ldap libnss-sss libpam-sss ldap-utils
  ```

<p class="callout warning"><strong>Installing When Internet Is Down (Arch Linux):</strong> On Arch Linux, the <code>sssd</code> package depends on <code>libldap</code> (shared libraries) rather than <code>openldap</code> (CLI tools like <code>ldapsearch</code>). If the internet is down and a package is already in the local pacman cache, install it directly with <code>sudo pacman -U /var/cache/pacman/pkg/sssd-*.pkg.tar.zst</code>. If <code>ldapsearch</code> is not installed, use the built-in <code>openssl s_client</code> and <code>sssctl</code> diagnostic commands in Section 4 below.</p>

---

### Step 2: Configure SSSD (`/etc/sssd/sssd.conf`)

Create backups of existing authentication configs:

```bash
sudo cp /etc/nsswitch.conf /etc/nsswitch.conf.bak
sudo cp /etc/pam.d/system-auth /etc/pam.d/system-auth.bak
```

Create `/etc/sssd/sssd.conf`:

```bash
sudo vim /etc/sssd/sssd.conf
```

Paste the following configuration:

```ini
[sssd]
services = nss, pam
config_file_version = 2
domains = goauthentik.io
debug_level = 1

[nss]
debug_level = 1

[pam]
debug_level = 1

[domain/goauthentik.io]
debug_level = 1
id_provider = ldap
auth_provider = ldap
sudo_provider = none
ldap_uri = ldaps://ldap.drheck.dev:636
ldap_backup_uri = ldaps://ul.drheck.dev:636
ldap_id_use_start_tls = false
ldap_search_base = dc=ldap,dc=goauthentik,dc=io
ldap_user_search_base = ou=users,dc=ldap,dc=goauthentik,dc=io?sub?(memberOf=cn=locallogin,ou=groups,dc=ldap,dc=goauthentik,dc=io)
ldap_default_bind_dn = cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io
ldap_default_authtok = hgNbhl3t1hzyV4jdfsiQuAeYS

# Secure Production TLS Configuration (Let's Encrypt)
ldap_tls_reqcert = demand
ldap_tls_cacertdir = /etc/ssl/certs

ldap_schema = rfc2307bis
ldap_user_object_class = posixAccount
ldap_user_name = sAMAccountName
ldap_user_uid_number = uidNumber
ldap_user_gid_number = gidNumber
ldap_user_home_directory = homeDirectory
ldap_user_shell = loginShell
ldap_group_object_class = posixGroup
ldap_group_name = cn
ldap_group_gid_number = gidNumber
ldap_group_member = member
ldap_ignore_unreadable_references = true
default_shell = /bin/bash
fallback_homedir = /home/%u
enumerate = true
ldap_enumeration_refresh_offset = 0
ldap_entry_usn = entryUSN
cache_credentials = true
```

Set strict ownership and `0600` permissions (SSSD refuses to start if `/etc/sssd/sssd.conf` is group- or world-readable):

```bash
sudo chmod 600 /etc/sssd/sssd.conf
sudo chown root:root /etc/sssd/sssd.conf
```

---

### Step 3: Configure NSS (`/etc/nsswitch.conf`)

Open `/etc/nsswitch.conf`:

```bash
sudo vim /etc/nsswitch.conf
```

Ensure `sss` is appended to `passwd`, `group`, and `shadow`:

```text
passwd: compat mymachines systemd sss
group: compat mymachines systemd sss
shadow: compat systemd sss
```

---

### Step 4: Configure PAM (`/etc/pam.d/system-auth`, `su`, `su-l`)

Open `/etc/pam.d/system-auth`:

```bash
sudo vim /etc/pam.d/system-auth
```

Replace the contents with the SSSD + `pam_mkhomedir.so` PAM stack:

<p class="callout info"><strong>Automatic Home Directory Creation:</strong> The <code>pam_mkhomedir.so</code> module in the <code>session</code> block automatically creates <code>/home/&lt;username&gt;</code> on first login, copying skeleton files from <code>/etc/skel/</code> with <code>umask=0022</code> (<code>0755</code> permissions).</p>

```text
#%PAM-1.0

auth       required                    pam_faillock.so      preauth
auth       sufficient                  pam_sss.so           forward_pass
auth       [success=1 default=bad]     pam_unix.so          try_first_pass nullok
auth       [default=die]               pam_faillock.so      authfail
auth       optional                    pam_permit.so
auth       required                    pam_env.so
auth       required                    pam_faillock.so      authsucc

account    sufficient                  pam_sss.so
account    required                    pam_unix.so
account    optional                    pam_permit.so
account    required                    pam_time.so

password   sufficient                  pam_sss.so
password   required                    pam_unix.so          try_first_pass nullok sha512 shadow
password   optional                    pam_permit.so

session    required                    pam_limits.so
session    required                    pam_unix.so
session    required                    pam_mkhomedir.so     skel=/etc/skel/ umask=0022
session    optional                    pam_exec.so          seteuid /usr/local/bin/gdm-ldap-user-session.sh
session    optional                    pam_sss.so
session    optional                    pam_permit.so
```

Ensure `/etc/pam.d/su` and `/etc/pam.d/su-l` include `system-auth` so `su - <ldap-user>` works from a terminal:

```text
#%PAM-1.0
auth            sufficient      pam_rootok.so
auth            include         system-auth
account         include         system-auth
session         include         system-auth
password        include         system-auth
```

---

### Step 5: Unlock GDM Login Screen & Session Selector (`AccountsService`)

Arch Linux compiles `accountsservice` without native SSSD enumeration support (see [NSS + GDM Issue](https://wiki.drheck.dev/books/system-setup/page/nss-gdm-issue)). Without a profile in `/var/lib/AccountsService/users/<username>`, GDM may hide remote LDAP users from the login list or hide the desktop session selector (gear icon).

#### 1. Create the PAM Auto-Provisioning Script (`/usr/local/bin/gdm-ldap-user-session.sh`)

```shell
sudo tee /usr/local/bin/gdm-ldap-user-session.sh >/dev/null << 'EOS'
#!/bin/bash
# Automatically provision AccountsService files for remote SSSD/LDAP users
# on their first login, unlocking GDM session selection on Arch Linux.

USER="$PAM_USER"
if [ -z "$USER" ]; then
    exit 0
fi

# 1. Skip local users (already in /etc/passwd)
if ! grep -q "^$USER:" /etc/passwd && id "$USER" &>/dev/null; then
    UID_NUM=$(id -u "$USER")

    # 2. Only process standard users (UID >= 1000)
    if [ "$UID_NUM" -ge 1000 ]; then
        FILE="/var/lib/AccountsService/users/$USER"
        if [ ! -f "$FILE" ]; then
            cat <<EOF > "$FILE"
[User]
Session=gnome
Icon=/home/$USER/.face
SystemAccount=false
EOF
            chown root:root "$FILE"
            chmod 600 "$FILE"
        fi
    fi
fi
EOS
sudo chmod 755 /usr/local/bin/gdm-ldap-user-session.sh
```

#### 2. Pre-Provision Existing LDAP Users Immediately (Before First Login)

Once SSSD is started in Step 6, run the following command (handled automatically by `ldap_deploy.py`) to pre-populate `/var/lib/AccountsService/users/<user>` for all enumerated LDAP users so they appear on the GDM login screen immediately without needing a prior terminal login:

```shell
if [ -d /var/lib/AccountsService/users ]; then
    for u in $(getent passwd -s sss | awk -F: '$3 >= 2000 {print $1}'); do
        if ! grep -q "^${u}:" /etc/passwd && [ ! -f "/var/lib/AccountsService/users/${u}" ]; then
            sudo tee "/var/lib/AccountsService/users/${u}" >/dev/null <<EOF
[User]
Session=gnome
Icon=/home/${u}/.face
SystemAccount=false
EOF
            sudo chmod 600 "/var/lib/AccountsService/users/${u}"
        fi
    done
fi
```

---

### Step 6: Start SSSD & Verify Before Logging Out

Clear any stale SSSD cache from previous attempts, then enable and start `sssd`:

```bash
sudo systemctl stop sssd
sudo rm -f /var/lib/sss/db/cache_goauthentik.io.ldb
sudo systemctl enable --now sssd
```

---

## 3. Verification Checklist (Run Before Logging Out)

1. **Verify SSSD Backend Online Status:**
   ```bash
   sudo sssctl domain-status goauthentik.io
   ```
   Expected output:
   ```text
   Online status: Online

   Active servers:
   LDAP: ldap.drheck.dev

   Discovered LDAP servers:
   - ldap.drheck.dev
   - ul.drheck.dev
   ```

2. **Verify SSSD LDAP User Enumeration:**
   Query only the `sss` NSS module to list all LDAP POSIX users resolved from Authentik:
   ```bash
   getent passwd -s sss
   ```
   Expected output:
   ```text
   akadmin:*:2004:2004:akadmin:/home/akadmin:/bin/bash
   brett:*:2024:2024:brett:/home/brett:/bin/bash
   ikura:*:2022:2022:ikura:/home/ikura:/bin/bash
   haruka:*:2021:2021:haruka:/home/haruka:/bin/bash
   luke:*:2013:2013:luke:/home/luke:/bin/bash
   ```
   *(With `enumerate = true`, bare `getent passwd` also includes all five lines above at the bottom of the output. If `enumerate = false` was used on an older config, bare `getent passwd` only shows local `/etc/passwd` users and you must pass a username explicitly, e.g. `getent passwd ikura`.)*

3. **Verify UID/GID and Group Memberships:**
   ```bash
   id ikura
   id haruka
   ```

4. **Verify PAM Authentication & Automatic Home Directory Creation:**
   ```bash
   sudo su - ikura -c "pwd && id"
   ```
   Or test full password authentication via SSH/PAM:
   ```bash
   ssh ikura@localhost
   ```

---

## 4. Offline & LAN Troubleshooting Runbook

<p class="callout warning"><strong>No Internet Required:</strong> All troubleshooting steps below work completely offline on the local LAN using standard system binaries (<code>getent</code>, <code>sssctl</code>, <code>openssl</code>, <code>python3</code>, <code>journalctl</code>) even if <code>ldapsearch</code> (<code>openldap</code>) is not installed.</p>

### Issue A: `getent passwd` Only Shows `root` and Local User (`luke`), Nothing for LDAP Users

1. **Test Direct Username Lookup vs. Enumeration:**
   ```bash
   getent passwd -s sss ikura
   getent passwd -s sss haruka
   ```
   * **If `getent passwd -s sss ikura` returns a line (`ikura:*:2022:...`) while bare `getent passwd` does not:**
     `/etc/sssd/sssd.conf` has `enumerate = false` (or the initial 10-second enumeration delay has not finished). Set `enumerate = true` and `ldap_enumeration_refresh_offset = 0` in `/etc/sssd/sssd.conf` and restart `sssd`.
   * **If `getent passwd -s sss ikura` returns nothing (exit code `2`):**
     SSSD is failing to reach or bind to `ldap.drheck.dev:636`. Proceed to **Issue B** below.

---

### Issue B: SSSD Domain Is `Offline` or Cannot Resolve Any LDAP Users

Check SSSD's domain status and backend log:

```bash
sudo sssctl domain-status goauthentik.io
sudo tail -n 35 /var/log/sssd/sssd_goauthentik.io.log
```

#### Step 1: Verify DNS Resolution (`ldap.drheck.dev` and `ul.drheck.dev`)

`ldap.drheck.dev` resolves to `ul`'s ULA IPv6 address (`fd77:7777:5010:0:1337:f1ac:d00d:aaaa`), while `ul.drheck.dev` resolves to both IPv4 (`10.0.0.3`) and IPv6 (`fd77:7777:5010:0:1337:f1ac:d00d:337`):

```bash
getent ahosts ldap.drheck.dev
getent ahosts ul.drheck.dev
```

* If the client machine does not have a local ULA IPv6 address (`ip -6 addr show`), ensure `ldap_backup_uri = ldaps://ul.drheck.dev:636` is present in `/etc/sssd/sssd.conf` so SSSD connects over IPv4 (`10.0.0.3:636`).

#### Step 2: Test LDAPS TCP & TLS Certificate Handshake (Without `ldapsearch`)

Use `openssl` (installed on all base systems) to verify that port `636` on `ul` is reachable and presenting a valid Let's Encrypt certificate for `*.drheck.dev`:

```bash
openssl s_client -connect ldap.drheck.dev:636 -servername ldap.drheck.dev </dev/null 2>&1 | grep -E "CONNECTED|subject=|issuer=|Verify return code"
```

Or test over IPv4 via `ul.drheck.dev:636`:

```bash
openssl s_client -connect 10.0.0.3:636 -servername ul.drheck.dev </dev/null 2>&1 | grep -E "CONNECTED|subject=|issuer=|Verify return code"
```

Expected output:
```text
CONNECTED(00000003)
subject=CN=drheck.dev
issuer=C=US, O=Let's Encrypt, CN=YE1
    Verify return code: 0 (ok)
```

* **If `Verify return code` is non-zero or clock is skewed:** Check `timedatectl status` on the client laptop (if the CMOS clock is wrong when offline, TLS certificate validation against Let's Encrypt will fail).

#### Step 3: Check for `Invalid credentials (49)` / Bind Password Mismatch

If `openssl s_client` succeeds with `Verify return code: 0 (ok)`, check `/var/log/sssd/sssd_goauthentik.io.log` (with debug level 6 if needed):

```bash
sudo sssctl debug-level 6
sudo systemctl restart sssd
sleep 3
sudo grep -i -E "bind|credential|tls|error|fail|offline" /var/log/sssd/sssd_goauthentik.io.log | tail -n 30
sudo sssctl debug-level 1
```

If `openldap` (`ldapsearch`) is installed, test the `ldapservice` bind directly:

```bash
LDAPTLS_REQCERT=demand ldapsearch -LLL -x \
  -H ldaps://ldap.drheck.dev:636 \
  -D "cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io" \
  -w "hgNbhl3t1hzyV4jdfsiQuAeYS" \
  -b "ou=users,dc=ldap,dc=goauthentik,dc=io" "(cn=ikura)" cn uidNumber gidNumber
```

* **If `ldapsearch` returns `ldap_bind: Invalid credentials (49)`** (or `sudo -u docker-data XDG_RUNTIME_DIR=/run/user/1100 journalctl --user -u authentik-ldap.service` on `ul` shows `"error":"flow error password: Invalid password"` for `cn=ldapservice,ou=users,dc=ldap,dc=goauthentik,dc=io`):
  The `ldapservice` password in Authentik on `ul` does not match `ldap_default_authtok` in `/etc/sssd/sssd.conf` (or another service overwrote `ldapservice`'s password instead of using its own dedicated service account).
  To verify and restore `ldapservice` on `ul`:
  ```bash
  ssh ul '(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -i authentik-server ak shell -c "
  from authentik.core.models import User
  from authentik.providers.ldap.models import LDAPProvider
  p = LDAPProvider.objects.first()
  u = User.objects.get(username=\"ldapservice\")
  print(\"Password match:\", u.check_password(\"hgNbhl3t1hzyV4jdfsiQuAeYS\"))
  print(\"LDAPProvider perms:\", u.get_all_permissions(p))
  " 2>/dev/null)'
  ```
  If `Password match:` is `False`, restore it on `ul` and restart `authentik-ldap.service` so its cached bind session is flushed:
  ```bash
  ssh ul '(cd /tmp && sudo -u docker-data HOME=/home/docker-data XDG_RUNTIME_DIR=/run/user/1100 podman exec -i authentik-server ak shell -c "
  from authentik.core.models import User
  u = User.objects.get(username=\"ldapservice\")
  u.set_password(\"hgNbhl3t1hzyV4jdfsiQuAeYS\")
  u.save()
  " 2>/dev/null) && sudo systemctl --user -M docker-data@ restart authentik-ldap.service'
  ```

---

### Issue C: Flushing Stale SSSD Cache & Forcing SSSD Online

If SSSD marked the domain offline during a temporary network glitch, force it back online and clear the cache:

```bash
sudo systemctl stop sssd
sudo rm -f /var/lib/sss/db/cache_goauthentik.io.ldb
sudo systemctl start sssd
sudo sssctl domain-status goauthentik.io
getent passwd -s sss
```

---

### Issue D: User Resolves in `getent passwd` But Cannot Log In on GDM / Laptop

1. **Check if the user appears on the GDM login screen:**
   Verify `/var/lib/AccountsService/users/<username>` exists and has `SystemAccount=false`:
   ```bash
   sudo cat /var/lib/AccountsService/users/ikura
   ```
   If missing, run the pre-provisioning snippet in **Step 5.2** above and restart `accounts-daemon`:
   ```bash
   sudo systemctl restart accounts-daemon
   ```

2. **Test PAM authentication from a terminal (`su` / `login`):**
   ```bash
   su - ikura
   ```
   If `su - ikura` fails while `sudo su - ikura` works, inspect `/var/log/sssd/sssd_pam.log` and `journalctl -n 40 --no-pager`:
   * Ensure `/etc/pam.d/system-auth` contains `pam_sss.so` in `auth`, `account`, `password`, and `session`.
   * Ensure the user is active in Authentik and their password was set in Authentik (`pwdChangedTime` present on their LDAP entry).