CrowdSec & OpenBSD PF Perimeter Security Setup

1. Overview & Architectural Concept

This document outlines the distributed intrusion detection and perimeter enforcement setup across the local infrastructure (drheck.dev).

                              [ Public Internet (WAN) ]
                                          │
                                          ▼
                         ┌─────────────────────────────────┐
                         │   danknasty (OpenBSD Router)    │
                         │ ─────────────────────────────── │
                         │ • OpenBSD PF Kernel Firewall    │
                         │ • Kernel SSH Rate Limiter       │
                         │ • Syslog SSH Forwarder (UDP 514)│
                         │ • CrowdSec PF Firewall Bouncer  │
                         └────────────────┬────────────────┘
                                          │ Drops banned IPs at WAN edge
                                          │ Stream bans via IPv6 LAN ([fd77:...:c70d]:8080)
                                          ▼
                         ┌─────────────────────────────────┐
                         │      ul (Arch Linux Server)     │
                         │ ─────────────────────────────── │
                         │ • Dedicated IPv6: ...:c70d/64   │
                         │ • Nginx Web Reverse Proxy       │
                         │ • Remote Syslog Ingestion (514) │
                         │ • CrowdSec LAPI Decision Engine │
                         │ • 53-Hour Ban Policy Engine     │
                         │ • Public Active Enriched JSON   │
                         │ • Append-Only NDJSON Stream     │
                         └─────────────────────────────────┘

Design Philosophy & Conventions

Dedicated Service IPv6 Address (ul):
- Per local network convention (20-wired.network), CrowdSec LAPI listens on a dedicated IPv6 ULA: fd77:7777:5010:0:1337:f1ac:d00d:c70d on port 8080 (LAPI) and UDP port 514 (Syslog Ingestion).
Offload All Heavy Detection to Powerful Hardware (ul):
- All heavy log analysis, regex pattern matching, SQLite/LAPI database operations, and scenario evaluations run on the Arch Linux server (ul).
- Nginx Web Probes & Direct IP Hits: Parses Nginx access logs to detect scanner bots (e.g., .env harvesting, WordPress probes, webshell attempts, open proxy CONNECT attempts, and direct IP connection attempts).
- Router SSH Probes, Invalid Users & Root Attempts: Receives OpenBSD SSH authentication logs (/var/log/authlog) forwarded over UDP IPv6 syslog from danknasty to detect SSH brute-force attempts, instant invalid username bans, and instant root attempt bans.
- 53-Hour Ban Policy: Issues a global 53-hour ban for any detected web probe, direct IP scan, or SSH brute-force/invalid user/root attempt.
Perimeter Enforcement at Gateway (danknasty):
- The OpenBSD router (danknasty) runs only the lightweight CrowdSec PF Bouncer (crowdsec_firewall_bouncer) and native syslogd.
- Banned IPs are pulled from ul and injected into OpenBSD PF's kernel <crowdsec-blacklists> table. Banned packets are dropped at the network interface before entering the local network.
Zero-Overhead Edge SSH Protection (danknasty):
- OpenBSD PF tracks TCP connection rates directly in kernel state space (max-src-conn-rate 3/30).
- Any bot opening more than 3 SSH connections in 30 seconds is added to the OpenBSD PF <bruteforce> table and dropped with 0% CPU and ~0 MB RAM usage.
LAN Subnet & Dynamic IPv6 Whitelisting:
- Private subnets (10.0.0.0/24, fd77:7777:5010::/64), loopbacks (127.0.0.1/32, ::1/128), and the dynamic public ISP IPv6 prefix (auto-detected via /usr/local/bin/crowdsec-update-ipv6-whitelist) are whitelisted on ul so internal LAN users and SSH typos can never trigger bans.
Rich Context & OSINT Enriched Export (ul):
- Every banned IP is automatically enriched with Reverse DNS (PTR), Country, Region, City, Coordinates (Lat/Lon), Timezone, ISP, Organization, and Autonomous System (ASN) details.
- Active Array File: /var/log/crowdsec-bans-public.json (chmod 644).
- Append-Only NDJSON Stream: /var/log/crowdsec-bans.json (chmod 644).

Detection Latency & Pipeline Timeline

Protection Layer	Target Threat	Latency	Mechanism / Action
Layer 1: OpenBSD PF Kernel Limiter	SSH Connection Spikes	Instant (Nanoseconds / ~0s)	In-kernel TCP state tracking (`max-src-conn-rate 3/30`) instantly moves abusive IPs into `<bruteforce>` table.
Layer 2: CrowdSec Distributed Ban	Web Probes, SSH Invalid Users & Root Attempts	~5 seconds (Max 10s)	Syslog/Nginx logs stream to `ul`. `ul` triggers 53h ban. `danknasty` bouncer pulls decision and injects IP into OpenBSD PF `<crowdsec-blacklists>`.

CrowdSec Distributed Pipeline Timeline:

Event Generation: sshd-session or Nginx emits log event (< 1 ms).
Log Forwarding: UDP IPv6 syslog or Nginx log tailer streams log to ul (< 1 ms).
Scenario Trigger & Ban Creation: CrowdSec parses line, leaks scenario bucket, and writes 53h ban to LAPI database (~20–50 ms).
Router Bouncer Polling: danknasty bouncer polls ul LAPI over IPv6 LAN (update_frequency: 10s, average ~5 seconds).
Kernel Table Injection: Bouncer executes pfctl -t crowdsec-blacklists -T add <IP> (~10 ms).

2. Implementation & Configuration Details

A. Central Decision Engine on `ul` (Arch Linux)

Interface Configuration (/etc/systemd/network/20-wired.network):
```
# crowdsec
Address=fd77:7777:5010::1337:f1ac:d00d:c70d/64
```
Listen URI (/etc/crowdsec/config.yaml): Bound to dedicated IPv6 ULA [fd77:7777:5010:0:1337:f1ac:d00d:c70d]:8080:
```
api:
  server:
    listen_uri: "[fd77:7777:5010:0:1337:f1ac:d00d:c70d]:8080"
```

Nginx Log Acquisition (/etc/crowdsec/acquis.d/nginx.yaml):

filenames:
  - /var/log/nginx/*.log
  - /var/log/nginx/access.log
labels:
  type: nginx

Router SSH Syslog Ingestion (/etc/crowdsec/acquis.d/danknasty-ssh.yaml):

source: syslog
listen_addr: "fd77:7777:5010:0:1337:f1ac:d00d:c70d"
listen_port: 514
labels:
  type: syslog

LAN & Dynamic IPv6 Whitelist Override (/etc/crowdsec/parsers/s02-override/00-lan-whitelist.yaml):

name: custom/lan-whitelist
description: "Whitelist internal LAN subnets and dynamic public IPv6 prefix"
filter: "evt.Meta.source_ip != nil"
whitelist:
  reason: "Exempt internal LAN subnets and dynamic public IPv6 prefix"
  cidr:
    - "10.0.0.0/24"
    - "fd77:7777:5010::/64"
    - "127.0.0.1/32"
    - "::1/128"
    - "2603:3001:626:9d80::/64" # Auto-updated by crowdsec-update-ipv6-whitelist

Custom Honeypot Scenario (/etc/crowdsec/scenarios/honeypot-endpoints.yaml):

type: leaky
name: custom/honeypot-endpoints
description: "Instant trigger on bot/scanner endpoints"
filter: |
  evt.Parsed.request in ["/.env", "/cgi-bin/authLogin.cgi", "/IndoSec.php", "/alfa.php", "/luxe.php", "/BDKR28WP.php"] || evt.Parsed.request matches "(?i)^/(wp-admin|wp-login|wp-includes|\\.env|cgi-bin|alfa\\.php|1\\.php|222\\.php|8\\.php|shell\\.php)"
capacity: 1
leakspeed: 1h
groupby: evt.Meta.source_ip
blackhole: 1m
labels:
  service: nginx
  type: honeypot_scan
  remediation: true

Custom SSH Invalid User Scenario (/etc/crowdsec/scenarios/ssh-invalid-user.yaml): Instant 53-hour ban for any attempt to log in using an invalid username:

type: leaky
name: custom/ssh-invalid-user
description: "Instant 53h ban for any SSH login attempt using an invalid user"
filter: |
  evt.Meta.log_type == "ssh_failed-auth" && (
    evt.Parsed.sshd_invalid_user != "" ||
    evt.Meta.target_user != ""
  )
capacity: 1
leakspeed: 1h
groupby: evt.Meta.source_ip
blackhole: 1m
labels:
  service: ssh
  type: bruteforce
  remediation: true

Custom SSH Root Attempt Scenario (/etc/crowdsec/scenarios/ssh-root-login-attempt.yaml): Instant 53-hour ban for any attempt targeting the root account:

type: leaky
name: custom/ssh-root-login-attempt
description: "Instant 53h ban for any SSH login attempt targeting user root"
filter: |
  evt.Meta.log_type == "ssh_failed-auth" && (
    evt.Meta.target_user == "root" ||
    evt.Parsed.sshd_invalid_user == "root" ||
    evt.Parsed.sshd_user == "root"
  )
capacity: 1
leakspeed: 1h
groupby: evt.Meta.source_ip
blackhole: 1m
labels:
  service: ssh
  type: bruteforce
  remediation: true

Custom Direct IP Access Scenario (/etc/crowdsec/scenarios/direct-ip-access.yaml): Instant 53-hour ban for any HTTP/HTTPS connection made directly by IP address or invalid host header:

type: leaky
name: custom/direct-ip-access
description: "Instant 53h ban for any HTTP/HTTPS connection made directly to IP address or non-domain host"
filter: |
  evt.Meta.log_type == "http_access-log" && (
    evt.Meta.http_status in ["444", "400", "403"] ||
    evt.Parsed.status in ["444", "400", "403"] ||
    evt.Parsed.target_fqdn matches "^([0-9]{1,3}\\.){3}[0-9]{1,3}$" ||
    evt.Parsed.http_host matches "^([0-9]{1,3}\\.){3}[0-9]{1,3}$"
  )
capacity: 1
leakspeed: 1h
groupby: evt.Meta.source_ip
blackhole: 1m
labels:
  service: nginx
  type: direct_ip_scan
  remediation: true

53-Hour Ban Policy (/etc/crowdsec/profiles.yaml):

name: default_ip_remediation
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
 - type: ban
   duration: 53h
notifications:
 - file_default
on_success: break
---
name: default_range_remediation
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Range"
decisions:
 - type: ban
   duration: 53h
notifications:
 - file_default
on_success: break

File Notification Plugin (/etc/crowdsec/notifications/file.yaml): Appends real-time events upon trigger to /var/log/crowdsec-notifications.log:

type: file
name: file_default
log_level: info
group_wait: 0s
group_threshold: 1
format: |
  {{range . -}}
  {{$alert := . -}}
  {{range .Decisions -}}
  {"timestamp":"{{$alert.CreatedAt}}","ip":"{{.Value}}","action":"{{.Action}}","duration":"{{.Duration}}","reason":"{{.Scenario}}","country":"{{$alert.Source.Cn}}","as_name":"{{$alert.Source.AsName}}"}
  {{end -}}
  {{end -}}
log_path: "/var/log/crowdsec-notifications.log"
rotate:
  enabled: false

Host Firewall Rules (/etc/nftables.conf):

ip6 saddr fd77:7777:5010::/64 ip6 daddr fd77:7777:5010:0:1337:f1ac:d00d:c70d tcp dport 8080 accept comment "Allow CrowdSec LAPI from local IPv6 LAN"
ip6 saddr fd77:7777:5010::/64 ip6 daddr fd77:7777:5010:0:1337:f1ac:d00d:c70d udp dport 514 accept comment "Allow CrowdSec Syslog Ingestion from local IPv6 LAN"

Automated OSINT & Sync Pipeline (/usr/local/bin/crowdsec-export-bans & crowdsec-sync-authlog): Runs every 60 seconds (crowdsec-export-bans.timer):
1. Updates dynamic public IPv6 whitelist (/usr/local/bin/crowdsec-update-ipv6-whitelist).
2. Syncs /var/log/authlog from danknasty to ensure no historical SSH attackers are missed (/usr/local/bin/crowdsec-sync-authlog).
3. Updates /var/log/crowdsec-bans-public.json (active JSON array) and appends newly discovered ban events with full GeoIP/ASN/rDNS metadata to /var/log/crowdsec-bans.json (chmod 644).

B. Perimeter Enforcement & Syslog Forwarding on `danknasty` (OpenBSD 7.9 Router)

Syslog Forwarding Configuration (/etc/syslog.conf):

auth.info;authpriv.info					@[fd77:7777:5010:0:1337:f1ac:d00d:c70d]

Bouncer Configuration (/etc/crowdsec-firewall-bouncer/crowdsec-firewall-bouncer.yaml):

mode: pf
retry_initial_connect: true
update_frequency: 10s
api_url: http://[fd77:7777:5010:0:1337:f1ac:d00d:c70d]:8080/
api_key: <GENERATED_API_KEY>
deny_action: DROP
blacklists_ipv4: crowdsec-blacklists
blacklists_ipv6: crowdsec6-blacklists
pf:
  anchor_name: ""

Router Firewall Rules (/etc/pf.conf):

# CrowdSec Blacklist Tables
table <crowdsec-blacklists> persist
table <crowdsec6-blacklists> persist
block drop quick from <crowdsec-blacklists>
block drop quick from <crowdsec6-blacklists>

# Table for SSH Brute-force Auto-Blocking
table <bruteforce> persist
block drop quick from <bruteforce>

# SSH rules with stateful connection tracking
pass in quick on $ext proto tcp to self port ssh \
    flags S/SA keep state \
    (max-src-conn 10, max-src-conn-rate 3/30, overload <bruteforce> flush global)

3. Interaction, Operations & Data Format

A. Enriched Public JSON Data Schema

The exported JSON files (/var/log/crowdsec-bans-public.json and /var/log/crowdsec-bans.json) contain rich OSINT, network, and threat context for each banned IP:

{
  "decision_id": 4,
  "alert_id": 4,
  "ip": "119.30.119.3",
  "ip_version": "IPv4",
  "origin": "crowdsec",
  "reason": "crowdsecurity/netgear_rce",
  "action": "ban",
  "duration": "52h59m18s",
  "country": "Pakistan",
  "country_code": "PK",
  "region": "Punjab",
  "city": "Sialkot",
  "latitude": 32.426,
  "longitude": 74.3644,
  "timezone": "Asia/Karachi",
  "isp": "Mobilink GSM, Pakistan Mobile Communication Ltd.",
  "org": "Linkdotnet Telecom Limited",
  "as_name": "AS58470 IX Peering for Mobilink and Link Direct International.",
  "rdns": "static-host119-30-119-3.link.net.pk",
  "target_uri": "/setup.cgi?next_file=netgear.cfg&todo=syscmd&cmd=rm+-rf+/tmp/*...",
  "user_agent": "-",
  "method": "GET",
  "status_code": "301",
  "created_at": "2026-08-05T02:44:21Z"
}

B. Viewing Active System State & OpenBSD PF Kernel Tables

1. View Active Bans on `ul` (CrowdSec LAPI)

# View all active decisions (untruncated):
sudo cscli decisions list -a

# Query if a specific IP is banned in CrowdSec:
sudo cscli decisions list --ip <IP_ADDRESS>

2. View Active Enriched JSON Array

cat /var/log/crowdsec-bans-public.json

3. Tail the Append-Only Ban Stream (NDJSON)

tail -f /var/log/crowdsec-bans.json

4. View Active Bouncers & Connection Status

sudo cscli bouncers list

5. View Detection Metrics & Statistics

Shows lines read via Nginx logs and remote router syslog:

sudo cscli metrics

6. View Active OpenBSD PF Kernel Tables on Router (`danknasty`)

# List all blocked IPs in CrowdSec's PF Radix Tree table:
SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t crowdsec-blacklists -T show"

# View packet & byte drop statistics for the CrowdSec PF table:
SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -v -t crowdsec-blacklists -T show"

# Test if a specific IP is currently blocked in the OpenBSD PF kernel table:
SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t crowdsec-blacklists -T test <IP_ADDRESS>"

# View OpenBSD SSH kernel brute-force bans:
SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t bruteforce -T show"

C. Manual Interactivity & Ban Management

1. Manually Ban an IP for 53 Hours

sudo cscli decisions add --ip <IP_ADDRESS> --duration 53h --reason "Manual administrative ban"

2. Remove a Ban (Unban an IP)

sudo cscli decisions delete --ip <IP_ADDRESS>

3. Force Immediate Synchronization & OSINT Enrichment

sudo /usr/local/bin/crowdsec-export-bans

4. Inspect Recent Detection Alerts

sudo cscli alerts list
sudo cscli alerts inspect <ALERT_ID>

5. Flush Banned IP Tables on the Router (Emergency Reset)

SSH_AUTH_SOCK=/home/luke/.local/ssh_auth_sock/ssh_auth_sock ssh danknasty "doas pfctl -t crowdsec-blacklists -T flush"

D. Whitelisting Local Trusted Networks

Internal LAN subnets and your dynamic public IPv6 prefix are protected from accidental bans via /etc/crowdsec/parsers/s02-override/00-lan-whitelist.yaml:

name: custom/lan-whitelist
description: "Whitelist internal LAN subnets from all CrowdSec scenarios"
filter: "evt.Meta.source_ip != nil"
whitelist:
  reason: "Exempt internal LAN subnets"
  cidr:
    - "10.0.0.0/24"
    - "fd77:7777:5010::/64"
    - "127.0.0.1/32"
    - "::1/128"
    - "2603:3001:626:9d80::/64"

After modifying whitelists or scenarios, reload CrowdSec:

sudo systemctl reload crowdsec