SSH Command Tracking Guide
Detail of command execution auditing and tracking
Part 1: Arch Linux (ul, bmo95, studio-kvm-ssh)
The Linux Audit Framework (auditd) captures execve system calls triggered by human user sessions (auid >= 1000)., recording 100% of interactive and SSH payload commands while strictly filtering out background system and Docker container noise.
1. Installation & Service Setup
Install the audit package and enable the auditd daemon:
$ sudo pacman -S --noconfirm audit
$ sudo systemctl enable --now auditd
2. Configuration Rules & Interactive User Filtering
Create /etc/audit/rules.d/execve.rules with strict Audit User ID filtering (auid >= 1000):
[!IMPORTANT] CRITICAL RULE: Do NOT include catch-all
execverules without-F auid>=1000. Background system daemons and Docker containers (e.g. MinIO healthchecks, Authentik workers) haveauid = 4294967295(unset). An unfiltered rule generates ~8 MB of log noise every 7 minutes, rotating out entire log histories within 30 minutes.
$ sudo tee /etc/audit/rules.d/execve.rules > /dev/null << EOF'EOF'
# Delete pre-existing rules
-D
# Set kernel buffer size
-b 8192
# Log all command executions strictly for human/login users (auid >= 1000)
-a always,exit -F arch=b64 -S execve -F auid>=1000 -k remote_commands
-a always,exit -F arch=b32 -S execve -F auid>=1000 -k remote_commands
EOF
Load rules into the active kernel:
$ sudo auditctl -D && sudo augenrules --load
$ sudo auditctl -e 1
Verify active rules:
$ sudo auditctl -l
Process UID (uid) vs Audit User ID (auid)
PUID=1108): Controls unprivileged file permissions inside Docker containers.
Audit User ID (auid / loginuid): Assigned exclusively by PAM when a human authenticates via SSH or TTY (auid = 1000). Background containers started at boot have auid = 4294967295 (unset).
Result: Filtering by auid >= 1000 completely excludes container healthcheck noise while capturing 100% of human shell commands and sudo executions, extending audit log retention from 30 minutes to months.
3. Real-TimePython Command Inspection & Summarization (audit-log)
QueryThe Linux kernel encodes command executionsarguments usingcontaining thespaces, high-speedquotes, or special characters as hexadecimal byte strings (e.g., 6670696E67 for fping). The /usr/local/bin/audit-log helperPython script:utility automatically traverses rotated audit logs (/var/log/audit/audit.log.*) in chronological order, parses EXECVE records, decodes hexadecimal tokens into UTF-8 plaintext, auto-elevates privileges via sudo, and provides flexible date filtering, noise reduction, and daily activity summarization.
Tool Implementation (/usr/local/bin/audit-log)
#!/usr/bin/env python3
"""
audit-log: High-performance Linux auditd & OpenBSD command execution tracker.
Parses, decodes, filters, and summarizes EXECVE command history.
"""
import sys
import os
import re
import glob
import argparse
import datetime
from collections import defaultdict
NOISE_PATTERNS = [
r'loas_time_remaining\.sh',
r'zfs-stats',
r'systemctl (?:is-active|status)',
r'/usr/local/bin/audit-log',
r'/usr/bin/python3 /usr/local/bin/audit-log',
r'audit-log',
r'^uptime$',
r'^ss -tulpn',
r'^docker ps',
r'^ps aux',
r'^cat /etc/passwd',
r'^stty',
r'^tty$',
r'^which ',
r'^getopt ',
r'^sed -e',
r'^xargs',
r'find /etc/debuginfod',
r'cat /etc/debuginfod',
r'ssh-add',
r'^base64',
r'^tr \n',
r'/usr/bin/python /usr/bin/vapoursynth',
r'^cut -d',
r'mkdir -p /home/[^/]+/\.config/ssh_auth_sock',
r'ln -sf /home/[^/]+/\.config/ssh_auth_sock'
]
NOISE_RE = re.compile('|'.join(NOISE_PATTERNS))
def decode_token(tok: str) -> str:
"""Decode hex-encoded audit argument strings into UTF-8 plaintext."""
if len(tok) >= 2 and len(tok) % 2 == 0 and re.fullmatch(r'[0-9A-Fa-f]+', tok):
try:
return bytes.fromhex(tok).decode('utf-8', errors='replace')
except Exception:
return tok
return tok
EXEC_RE = re.compile(r'type=EXECVE\s+msg=audit\((\d+)\.\d+:\d+\):\s+(.*)')
ARG_RE = re.compile(r'a\d+=(?:\"([^\"]*)\"|([0-9A-Fa-f]+)|(\S+))')
def get_log_files():
"""Return rotated audit.log files in chronological order (oldest to newest)."""
def sort_key(p):
part = p.split('.')[-1]
return int(part) if part.isdigit() else 0
rotated = sorted(glob.glob('/var/log/audit/audit.log.*'), key=sort_key, reverse=True)
current = ['/var/log/audit/audit.log'] if os.path.exists('/var/log/audit/audit.log') else []
return rotated + current
def parse_audit_stream():
"""Yields (timestamp_str, date_str, command_str) for all EXECVE records."""
for fpath in get_log_files():
try:
with open(fpath, 'r', encoding='utf-8', errors='replace') as f:
for line in f:
m = EXEC_RE.search(line)
if not m:
continue
ts = int(m.group(1))
dt = datetime.datetime.fromtimestamp(ts)
dt_str = dt.strftime('%Y-%m-%d %H:%M:%S')
date_str = dt.strftime('%Y-%m-%d')
args_part = m.group(2)
tokens = []
for am in ARG_RE.finditer(args_part):
q, h, r = am.groups()
if q is not None:
tokens.append(q)
elif h is not None:
tokens.append(decode_token(h))
elif r is not None:
tokens.append(r)
if tokens:
yield dt_str, date_str, ' '.join(tokens)
except PermissionError:
print(f'Permission denied accessing {fpath}. Please run with sudo.', file=sys.stderr)
sys.exit(1)
except Exception:
continue
def main():
if os.geteuid() != 0 and sys.platform.startswith('linux'):
try:
os.execvp('sudo', ['sudo', sys.executable, os.path.abspath(__file__)] + sys.argv[1:])
except Exception as e:
print(f'Failed to elevate with sudo: {e}', file=sys.stderr)
sys.exit(1)
parser = argparse.ArgumentParser(
description='Inspect, search, and summarize Linux auditd EXECVE command history.',
formatter_class=argparse.RawDescriptionHelpFormatter
)
parser.add_argument('limit_pos', nargs='?', default=None, help='Number of recent entries or "all"')
parser.add_argument('-n', '--lines', type=int, default=None, help='Number of entries to show')
parser.add_argument('-a', '--all', action='store_true', help='Show all entries')
parser.add_argument('-d', '--day', type=str, default=None, help='Filter by exact date (YYYY-MM-DD)')
parser.add_argument('--since', type=str, default=None, help='Filter commands since date (YYYY-MM-DD)')
parser.add_argument('--until', type=str, default=None, help='Filter commands until date (YYYY-MM-DD)')
parser.add_argument('-s', '--search', type=str, default=None, help='Search commands by regex / substring')
parser.add_argument('-c', '--clean', action='store_true', help='Filter out repetitive status/monitoring noise')
parser.add_argument('-u', '--unique', action='store_true', help='Deduplicate identical commands in output')
parser.add_argument('--daily', '--summary', dest='daily_summary', action='store_true', help='Group by day and output deduplicated activity summary')
args = parser.parse_args()
limit = 20
is_all = args.all
if args.limit_pos:
if args.limit_pos.lower() == 'all':
is_all = True
elif args.limit_pos.isdigit():
limit = int(args.limit_pos)
elif args.lines is not None:
limit = args.lines
search_re = re.compile(args.search, re.IGNORECASE) if args.search else None
if args.daily_summary:
daily_commands = defaultdict(list)
for dt_str, date_str, cmd in parse_audit_stream():
if args.day and date_str != args.day:
continue
if args.since and date_str < args.since:
continue
if args.until and date_str > args.until:
continue
if search_re and not search_re.search(cmd):
continue
if args.clean and NOISE_RE.search(cmd):
continue
daily_commands[date_str].append(cmd)
for day in sorted(daily_commands.keys()):
cmds = daily_commands[day]
print(f'=== {day} ({len(cmds)} commands) ===')
seen = set()
for c in cmds:
if c not in seen:
seen.add(c)
print(f' * {c}')
print()
return
results = []
seen_unique = set()
for dt_str, date_str, cmd in parse_audit_stream():
if args.day and date_str != args.day:
continue
if args.since and date_str < args.since:
continue
if args.until and date_str > args.until:
continue
if search_re and not search_re.search(cmd):
continue
if args.clean and NOISE_RE.search(cmd):
continue
if args.unique:
if cmd in seen_unique:
continue
seen_unique.add(cmd)
formatted = f'{dt_str} | {cmd}'
if is_all:
print(formatted)
else:
results.append(formatted)
if not is_all:
for line in results[-limit:]:
print(line)
if __name__ == '__main__':
main()
CLI Command Options & Usage
# View1. theDefault view: last 20 executed commands
(Default)
$ sudo audit-log
# 2. View a specific number of recent commandsentries (e.g. last 50 entries)commands)
$ sudo audit-log 50
$ audit-log -n 50
# View3. ALLStream all recorded command history
$ sudo audit-log all
# Count total logged command executions
$ sudo audit-log --all
|# wc4. Filter by specific day (YYYY-MM-DD)
$ audit-log -l-day 2026-08-29
# 5. Filter date range (since date / until date)
$ audit-log --since 2026-08-25 --clean
# 6. Filter repetitive background noise (tmux status bars, pollers)
$ audit-log --clean -n 30
# 7. Regex / substring search across command history
$ audit-log --search "funkwhale|docker compose" --clean
# 8. Deduplicate identical commands in output
$ audit-log --unique --since 2026-08-28
# 9. Generate automated daily change log summary outline
$ audit-log --daily --since 2026-08-25 --clean
Sample Output: Daily Activity Summary (--daily --clean)
2026-08-09 13:06:51 | sudo audit-log 20=== 2026-08-0928 13:06:51(38 |commands) tail===
* cd /data/docker/funkwhale && docker compose up -nd
20* docker exec funkwhale nginx -t && docker exec funkwhale nginx -s reload
* curl -k -i -r 0-65535 https://funkwhale.drheck.dev/api/v1/listen/.../?upload=...
* sed -i "s/docker-compose.yml data/docker-compose.yml mime.types funkwhale_nginx.template data/" /home/luke/git/backups/ul/scripts/backup/funkwhale
* shellcheck /home/luke/git/backups/ul/scripts/backup/funkwhale /home/luke/git/backups/ul/scripts/restore/funkwhale
=== 2026-08-0929 13:06:51(84 |commands) ===
* sqlite3 /opt/kavita/config/kavita.db "PRAGMA integrity_check;"
* systemctl restart kavita.service
* cd /data/docker/authentik && docker compose up -d --force-recreate
* cd /data/docker/immich && docker compose pull && docker compose up -d --force-recreate
* /usr/bin/unix_chkpwdenv rootbash chkexpiry/home/luke/git/backups/ul/scripts/upload/protondrive 2026-08-29_112824
4. auditd vs journalctl Clarification
auditd (/var/log/audit/audit.log): Captures ALL shell execution commands (execve) for human user sessions (auid >= 1000), whether run with or without sudo.
journalctl: Captures sudo commands via PAM/syslog, but does NOT receive non-sudo user shell commands because auditd writes directly to kernel audit log files.
Part 2: OpenBSD (Router danknasty)
OpenBSD uses native kernel process accounting (acct) to track binary executions, alongside syslog-based sshd verbose logging and doas auditing.
1. Kernel Process Accounting (acct)
Enable Accounting:
# Create log file
$ doas touch /var/account/acct
# Enable accounting at boot
$ echo "accounting=YES" | doas tee -a /etc/rc.conf.local
# Start accounting daemon immediately
$ doas /usr/sbin/accton /var/account/acct
2. Real-Time Command Inspection (audit-log)
Query process execution history using the matching audit-log helper script:
# View the last 20 executed commands (Default)
$ doas audit-log
# View a specific number of recent commands (e.g. 50 entries)
$ doas audit-log 50
# View ALL recorded command history
$ doas audit-log all
# Count total logged command executions
$ doas audit-log all | wc -l
Sample Output:
2026-08-09 13:06:00 | date
2026-08-09 13:06:00 | chmod
2026-08-09 13:06:00 | tee
2026-08-09 13:06:00 | lastcomm
3. SSH Verbose Payload & doas Logging
To log non-interactive SSH connections and payload commands to /var/log/authlog:
- Edit
/etc/ssh/sshd_config:LogLevel VERBOSE - Reload
sshd:$ doas rcctl reload sshd - Monitor logs:
$ doas tail -f /var/log/authlog