Skip to main content

SSH Command Tracking Guide

SSH & Agent Command Tracking Guide

Detail of command execution auditing and tracking across human SSH sessions and AI agent sessions (amux).

Part 1: Arch Linux (ul, bmo95, studio-kvm-ssh)

The Linux Audit Framework (auditd) captures execve system calls triggered by human user sessions (auid >= 1000), recording all interactive and SSH payload commands while strictly filtering out background system and Docker container noise.

1. Installation & Service Setup

Install the audit package and enable the auditd daemon:

$ sudo pacman -S --noconfirm audit
$ sudo systemctl enable --now auditd

2. Configuration Rules & Interactive User Filtering

Create /etc/audit/rules.d/execve.rules with Audit User ID filtering (auid >= 1000):

CRITICAL: Do NOT include catch-all execve rules without -F auid>=1000. Background system daemons and Docker containers (e.g. MinIO healthchecks, Authentik workers) have auid = 4294967295 (unset). An unfiltered rule generates ~8 MB of log noise every 7 minutes, rotating out entire log histories within 30 minutes.

$ sudo tee /etc/audit/rules.d/execve.rules > /dev/null << 'EOF'
# Delete pre-existing rules
-D

# Set kernel buffer size
-b 8192

# Log all command executions strictly for human/login users (auid >= 1000)
-a always,exit -F arch=b64 -S execve -F auid>=1000 -k remote_commands
-a always,exit -F arch=b32 -S execve -F auid>=1000 -k remote_commands
EOF

Load rules into the active kernel:

$ sudo auditctl -D && sudo augenrules --load
$ sudo auditctl -e 1

Verify active rules:

$ sudo auditctl -l

Process UID (uid) vs Audit User ID (auid)

  • Process UID (PUID=1108): Controls unprivileged file permissions inside Docker containers.
  • Audit User ID (auid / loginuid): Assigned exclusively by PAM when a human authenticates via SSH or TTY (auid = 1000). Background containers started at boot have auid = 4294967295 (unset).
  • Result: Filter by auid >= 1000

Part 2: Agent Execution Auditing (amux / systemd journal)

When AI agents or automated scripts run multi-line code via write-pane -f <file> or write-pane -b <base64> in amux, those commands can bypass standard auditd execve logging (because they are streamed directly into an already-running shell's PTY or an interactive Python/Node REPL).

To ensure complete, tamper-resistant auditing and forensic recovery:

3.1. Structured Journal Logging

Every call to write-pane in amux automatically logs directly into the systemd journal with structured fields:

    SYSLOG_IDENTIFIER=amux AMUX_PANE_ID: Target pane (%0, %1, etc.) AMUX_WINDOW_NAME: Hierarchical task window (e.g. mc-pokelovers-bluemap) AMUX_SOURCE: Original script file path or base64 AMUX_ARCHIVE_PATH: Permanent snapshot location in /data/amux/scripts/ AMUX_PAYLOAD_SIZE: Injected byte count

    Querying agent execution logs:

    # Stream live agent injections
$ journalctl -t amux -f

# Inspect detailed metadata for recent agent commands
$ journalctl -t amux -n 20 -o verbose

    2. Permanent Script Archival (/data/amux/scripts/)

    Before any byte is written to the terminal PTY, amux automatically saves a bit-for-bit snapshot to:

    /data/amux/scripts/YYYY-MM-DD_HHMM.SS_<window_name>.sh
      Durability: Stored on /data with ownership luke:luke. Zero Loss: Even if an agent immediately removes a temporary file from /dev/shm after execution, the exact script executed is preserved permanently. Disaster Recovery: /data/amux/scripts/ is automatically backed up during the daily backup run via /home/luke/git/backups/ul/scripts/backup/amux.

      Part 3: Python Command Inspection & Summarization (audit-log)

      The Linux kernel encodes command arguments containing spaces, quotes, or special characters as hexadecimal byte strings (e.g., 6670696E67 for fping). The /usr/local/bin/audit-log Python utility automaticallyparses, traversesdecodes, rotatedand auditsummarizes logsEXECVE (records from /var/log/audit/audit.log.*) in chronological order, parses EXECVE records, decodes hexadecimal tokens into UTF-8 plaintext, auto-elevates privileges via sudo, and provides flexible date filtering, noise reduction, and daily activity summarization..

      Tool Implementation (/usr/local/bin/audit-log)

      #!/usr/bin/env python3
"""
audit-log: High-performance Linux auditd & OpenBSD command execution tracker.
Parses, decodes, filters, and summarizes EXECVE command history.
"""

import sys
import os
import re
import glob
import argparse
import datetime
from collections import defaultdict

NOISE_PATTERNS = [
    r'zfs-stats',
    r'systemctl (?:is-active|status)',
    r'/usr/local/bin/audit-log',
    r'/usr/bin/python3 /usr/local/bin/audit-log',
    r'audit-log',
    r'^uptime$',
    r'^ss -tulpn',
    r'^docker ps',
    r'^ps aux',
    r'^cat /etc/passwd',
    r'^stty',
    r'^tty$'
]
NOISE_RE = re.compile('|'.join(NOISE_PATTERNS))

def decode_token(tok: str) -> str:
    """Decode hex-encoded audit argument strings into UTF-8 plaintext."""
    if len(tok) >= 2 and len(tok) % 2 == 0 and re.fullmatch(r'[0-9A-Fa-f]+', tok):
        try:
            return bytes.fromhex(tok).decode('utf-8', errors='replace')
        except Exception:
            return tok
    return tok

EXEC_RE = re.compile(r'type=EXECVE\s+msg=audit\((\d+)\.\d+:\d+\):\s+(.*)')
ARG_RE = re.compile(r'a\d+=(?:\"([^\"]*)\"|([0-9A-Fa-f]+)|(\S+))')

def get_log_files():
    """Return rotated audit.log files in chronological order (oldest to newest)."""
    def sort_key(p):
        part = p.split('.')[-1]
        return int(part) if part.isdigit() else 0

    rotated = sorted(glob.glob('/var/log/audit/audit.log.*'), key=sort_key, reverse=True)
    current = ['/var/log/audit/audit.log'] if os.path.exists('/var/log/audit/audit.log') else []
    return rotated + current

def parse_audit_stream():
    """Yields (timestamp_str, date_str, command_str) for all EXECVE records."""
    for fpath in get_log_files():
        try:
            with open(fpath, 'r', encoding='utf-8', errors='replace') as f:
                for line in f:
                    m = EXEC_RE.search(line)
                    if not m:
                        continue
                    ts = int(m.group(1))
                    dt = datetime.datetime.fromtimestamp(ts)
                    dt_str = dt.strftime('%Y-%m-%d %H:%M:%S')
                    date_str = dt.strftime('%Y-%m-%d')
                    args_part = m.group(2)
                    tokens = []
                    for am in ARG_RE.finditer(args_part):
                        q, h, r = am.groups()
                        if q is not None:
                            tokens.append(q)
                        elif h is not None:
                            tokens.append(decode_token(h))
                        elif r is not None:
                            tokens.append(r)
                    if tokens:
                        yield dt_str, date_str, ' '.join(tokens)
        except PermissionError:
            print(f'Permission denied accessing {fpath}. Please run with sudo.', file=sys.stderr)
            sys.exit(1)
        except Exception:
            continue

def main():
    if os.geteuid() != 0 and sys.platform.startswith('linux'):
        try:
            os.execvp('sudo', ['sudo', sys.executable, os.path.abspath(__file__)] + sys.argv[1:])
        except Exception as e:
            print(f'Failed to elevate with sudo: {e}', file=sys.stderr)
            sys.exit(1)

    parser = argparse.ArgumentParser(
        description='Inspect, search, and summarize Linux auditd EXECVE command history.',
        formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument('limit_pos', nargs='?', default=None, help='Number of recent entries or "all"')
    parser.add_argument('-n', '--lines', type=int, default=None, help='Number of entries to show')
    parser.add_argument('-a', '--all', action='store_true', help='Show all entries')
    parser.add_argument('-d', '--day', type=str, default=None, help='Filter by exact date (YYYY-MM-DD)')
    parser.add_argument('--since', type=str, default=None, help='Filter commands since date (YYYY-MM-DD)')
    parser.add_argument('--until', type=str, default=None, help='Filter commands until date (YYYY-MM-DD)')
    parser.add_argument('-s', '--search', type=str, default=None, help='Search commands by regex / substring')
    parser.add_argument('-c', '--clean', action='store_true', help='Filter out repetitive status/monitoring noise')
    parser.add_argument('-u', '--unique', action='store_true', help='Deduplicate identical commands in output')
    parser.add_argument('--daily', '--summary', dest='daily_summary', action='store_true', help='Group by day and output deduplicated activity summary')

    args = parser.parse_args()

    limit = 20
    is_all = args.all
    if args.limit_pos:
        if args.limit_pos.lower() == 'all':
            is_all = True
        elif args.limit_pos.isdigit():
            limit = int(args.limit_pos)
    elif args.lines is not None:
        limit = args.lines

    search_re = re.compile(args.search, re.IGNORECASE) if args.search else None

    if args.daily_summary:
        daily_commands = defaultdict(list)
        for dt_str, date_str, cmd in parse_audit_stream():
            if args.day and date_str != args.day:
                continue
            if args.since and date_str < args.since:
                continue
            if args.until and date_str > args.until:
                continue
            if search_re and not search_re.search(cmd):
                continue
            if args.clean and NOISE_RE.search(cmd):
                continue
            daily_commands[date_str].append(cmd)

        for day in sorted(daily_commands.keys()):
            cmds = daily_commands[day]
            print(f'=== {day} ({len(cmds)} commands) ===')
            seen = set()
            for c in cmds:
                if c not in seen:
                    seen.add(c)
                    print(f'  * {c}')
            print()
        return

    results = []
    seen_unique = set()
    for dt_str, date_str, cmd in parse_audit_stream():
        if args.day and date_str != args.day:
            continue
        if args.since and date_str < args.since:
            continue
        if args.until and date_str > args.until:
            continue
        if search_re and not search_re.search(cmd):
            continue
        if args.clean and NOISE_RE.search(cmd):
            continue
        if args.unique:
            if cmd in seen_unique:
                continue
            seen_unique.add(cmd)

        formatted = f'{dt_str} | {cmd}'
        if is_all:
            print(formatted)
        else:
            results.append(formatted)

    if not is_all:
        for line in results[-limit:]:
            print(line)

if __name__ == '__main__':
    main()

      CLI Command Options & Usage

      # 1. Default view: last 20 executed commands
$ audit-log

# 2. View specific number of entries (e.g. last 50 commands)
$ audit-log 50
$ audit-log -n 50

# 3. Stream all recorded command history
$ audit-log all
$ audit-log --all

# 4. Filter by specific day (YYYY-MM-DD)
$ audit-log --day 2026-08-29

# 5. Filter date range (since date / until date)
$ audit-log --since 2026-08-25 --clean

# 6. Filter repetitive background noise (tmux status bars, pollers)
$ audit-log --clean -n 30

# 7. Regex / substring search across command history
$ audit-log --search "funkwhale|docker compose" --clean

# 8. Deduplicate identical commands in output
$ audit-log --unique --since 2026-08-28

# 9. Generate automated daily change log summary outline
$ audit-log --daily --since 2026-08-25 --clean

      Sample Output: Daily Activity Summary (--daily --clean)

      === 2026-08-28 (38 commands) ===
  * cd /data/docker/funkwhale && docker compose up -d
  * docker exec funkwhale nginx -t && docker exec funkwhale nginx -s reload
  * curl -k -i -r 0-65535 https://funkwhale.drheck.dev/api/v1/listen/.../?upload=...
  * sed -i "s/docker-compose.yml data/docker-compose.yml mime.types funkwhale_nginx.template data/" /home/luke/git/backups/ul/scripts/backup/funkwhale
  * shellcheck /home/luke/git/backups/ul/scripts/backup/funkwhale /home/luke/git/backups/ul/scripts/restore/funkwhale

=== 2026-08-29 (84 commands) ===
  * sqlite3 /opt/kavita/config/kavita.db "PRAGMA integrity_check;"
  * systemctl restart kavita.service
  * cd /data/docker/authentik && docker compose up -d --force-recreate
  * cd /data/docker/immich && docker compose pull && docker compose up -d --force-recreate
  * /usr/bin/env bash /home/luke/git/backups/ul/scripts/upload/protondrive 2026-08-29_112824

      4. auditd vs journalctl Clarification

        auditd (/var/log/audit/audit.log): Captures ALL shell execution commands (execve) for human user sessions (auid >= 1000), whether run with or without sudo. journalctl: Captures sudo commands via PAM/syslog, but does NOT receive non-sudo user shell commands because auditd writes directly to kernel audit log files.

        Part 2:4: OpenBSD (Router danknasty)

        OpenBSD uses native kernel process accounting (acct) to track binary executions, alongside syslog-based sshd verbose logging and doas auditing.

        1. Kernel Process Accounting (acct)

        Enable Accounting:

        # Create log file
$ doas touch /var/account/acct

# Enable accounting at boot
$ echo "accounting=YES" | doas tee -a /etc/rc.conf.local

# Start accounting daemon immediately
$ doas /usr/sbin/accton /var/account/acct

        2. Real-Time Command Inspection (audit-log)

        Query process execution history using the matching audit-log helper script:

        # View the last 20 executed commands (Default)
$ doas audit-log

# View a specific number of recent commands (e.g. 50 entries)
$ doas audit-log 50

# View ALL recorded command history
$ doas audit-log all

        3. SSH Verbose Payload & doas Logging

        To log non-interactive SSH connections and payload commands to /var/log/authlog:

        1. Edit /etc/ssh/sshd_config: 
          LogLevel VERBOSE
        2. Reload sshd: 
          $ doas rcctl reload sshd
        3. Monitor logs: 
          $ doas tail -f /var/log/authlog