Core Code
Core Code Modern Dev Handbook
LINUX-PRODUCTION Contents
Linux Fundamentals for Production
Filesystem Layout (FHS) in Practice Users, Groups, and Service Accounts Permissions & ACLs (chmod/chown/setuid) Sudoers Safe Defaults Packages: apt/dnf, Repos, and Pinning Shell & Environment Hygiene Editing Configs Safely Time, Timezones, and NTP System Introspection Toolkit
Processes & Signals
Linux Process Model (PID, PPID, Fork/Exec) ps/top/htop: Finding What Hurts Signals in Practice (TERM/KILL/HUP) nice/renice and Scheduling Basics ulimits and Resource Limits Background Jobs (nohup, tmux, systemd-run) /proc and What It Tells You cgroups Basics (CPU/Memory Isolation) Zombies, Orphans, and Stuck Processes
systemd & Services
systemd Unit Types Overview Service Unit Best Practices Dependencies & Ordering (After/Wants/Requires) Restart Policies and Backoff Environment Files and Secrets Handling journald Basics (Reading, Filtering, Retention) Logging Practices with systemd Timers vs Cron Debugging systemd Services
Networking Basics for Production
ip, routes, and interfaces DNS Debugging (dig, resolvectl, hosts) Ports and Sockets with ss curl for HTTP Debugging TCP Basics (Handshake, States, Retries) Firewall Overview (nftables/iptables/ufw) NAT and Port Forwarding Concepts TLS and Certificates Basics Network Troubleshooting Playbook
Storage & Filesystems
Disk Layout and Partitioning (lsblk/fdisk) Filesystems (ext4/xfs) Basics Mounts and /etc/fstab Safety Disk Usage: df vs du (and Why They Differ) Inodes and “Disk Full” Mysteries Swap Strategy in Production Log Rotation (logrotate) Basics Temporary and Runtime Storage Hygiene Backup Storage Considerations
Security Basics
SSH Hardening in Production Users and Least Privilege in Production Firewall Minimum Rules for Production Linux Patching Strategy in Production Managing Secrets on Linux Hosts Linux Audit and Basic Logging in Production Fail2ban and Rate Limiting in Production SELinux and AppArmor Overview in Production Linux Security Baseline Checklist for Production
Observability on Linux
Linux Log Sources Map in Production journalctl Power Usage in Production Log Structure and Correlation in Production Metrics Hooks Overview in Production Health Checks and Readiness in Production Tracing Mindset for Linux and Production Systems Core Dumps Basics in Production Debug Packages and Symbols in Production Linux Observability Checklist for Production
Performance & Debugging
Load Average Explained for Production Debugging CPU Profiling Basics in Linux Production Memory Pressure Basics in Linux Production IO Bottlenecks and iostat in Production lsof and Open Files in Production Debugging Practical strace Usage in Production Sockets Debugging Tools in Linux Production dmesg and Kernel Signals for Production Debugging Production Debugging Workflow for Linux Systems
Deployment Patterns
Environment and Configuration Management in Production Release Directory Pattern for Safe Linux Deployments Zero-Downtime Restarts in Production Running Migrations Safely in Production Secrets Deploy Patterns in Production Backups and Restore Drills in Production Rollback Strategies for Production Deployments Maintenance Windows in Production Operations Production Deployment Checklist for Linux Systems
Production Checklists
Writing Effective Runbooks for Production Systems Failure Modes Mapping for Linux Production Systems Incident Response Basics for Linux Production Break-Glass Access in Production Capacity Planning Basics for Linux Production Systems Monitoring and Alerting Baseline for Linux Production Production Readiness Review for Linux Systems Configuration Drift in Linux Production Systems Go-Live Checklist

Linux Audit and Basic Logging in Production

Implement production-grade logging and audit basics: verify authentication logs, track sudo activity, detect suspicious behavior, and ensure logs survive incidents for real forensic visibility.
On this page

Why Logging Is a Security Control, Not Just Debug Output

If you cannot see what happened, you cannot respond to an incident. Logging is your memory. In production, audit visibility determines whether you detect compromise early or discover it weeks later.

Symptom

  • Security team asks: “Who logged in?” and no answer exists
  • No trace of sudo activity
  • Logs rotated too aggressively and lost
  • After incident, no reliable timeline
  • Authentication failures unnoticed for days

Root Cause

  • Default logging left unverified
  • No centralized log aggregation
  • Auditd not enabled
  • Log retention misconfigured
  • Critical services not logging at all

Investigation

1) Check SSH and Authentication Logs

Debian/Ubuntu:

sudo tail -n 100 /var/log/auth.log

RHEL/CentOS/Alma:

sudo tail -n 100 /var/log/secure

Search failed logins:

sudo grep "Failed password" /var/log/auth.log

2) Inspect systemd Journal

sudo journalctl -xe

Filter SSH logs:

sudo journalctl -u ssh --since "24 hours ago"

Filter sudo activity:

sudo journalctl _COMM=sudo --since "24 hours ago"

3) Verify auditd Status

sudo systemctl status auditd

If not installed:

sudo apt install auditd

Mitigation

1) Ensure Authentication Logging Is Active

Confirm rsyslog or journald is enabled:

sudo systemctl status rsyslog
sudo systemctl status systemd-journald

2) Enable auditd for Critical Monitoring

Example: monitor changes to /etc/passwd:

sudo auditctl -w /etc/passwd -p wa -k passwd_changes

List audit rules:

sudo auditctl -l

Search audit logs:

sudo ausearch -k passwd_changes

3) Configure Log Retention Safely

Check logrotate configuration:

cat /etc/logrotate.conf
ls /etc/logrotate.d/

Ensure logs are rotated but not lost too quickly. Security logs should survive long enough for investigation.

Hardening Strategy

  • Centralize logs (ELK, cloud logging, SIEM)
  • Protect log integrity (restrict write access)
  • Monitor authentication failures
  • Audit privilege escalation attempts
  • Track configuration file changes
  • Define retention policy

Real Incident Pattern

  • Attacker brute-forces SSH
  • Successfully logs in
  • Runs sudo to escalate privileges
  • Creates persistence via cron or systemd unit
  • Deletes or rotates logs to hide activity

If logging is weak, this entire chain becomes invisible.

Verification Checklist

sudo journalctl --disk-usage
sudo auditctl -l
sudo tail -n 50 /var/log/auth.log
  • SSH logins are recorded
  • Sudo usage is visible
  • Audit rules active for critical files
  • Logs are not world-writable

Why This Matters in Real Infrastructure

Security incidents are not prevented by hope — they are detected by visibility. Logging is the foundation of incident response. A production system without reliable audit logs is operating blind.

Fail2ban and Rate Limiting in Production →