Core Code
Core Code Modern Dev Handbook
LINUX-PRODUCTION Contents
Linux Fundamentals for Production
Filesystem Layout (FHS) in Practice Users, Groups, and Service Accounts Permissions & ACLs (chmod/chown/setuid) Sudoers Safe Defaults Packages: apt/dnf, Repos, and Pinning Shell & Environment Hygiene Editing Configs Safely Time, Timezones, and NTP System Introspection Toolkit
Processes & Signals
Linux Process Model (PID, PPID, Fork/Exec) ps/top/htop: Finding What Hurts Signals in Practice (TERM/KILL/HUP) nice/renice and Scheduling Basics ulimits and Resource Limits Background Jobs (nohup, tmux, systemd-run) /proc and What It Tells You cgroups Basics (CPU/Memory Isolation) Zombies, Orphans, and Stuck Processes
systemd & Services
systemd Unit Types Overview Service Unit Best Practices Dependencies & Ordering (After/Wants/Requires) Restart Policies and Backoff Environment Files and Secrets Handling journald Basics (Reading, Filtering, Retention) Logging Practices with systemd Timers vs Cron Debugging systemd Services
Networking Basics for Production
ip, routes, and interfaces DNS Debugging (dig, resolvectl, hosts) Ports and Sockets with ss curl for HTTP Debugging TCP Basics (Handshake, States, Retries) Firewall Overview (nftables/iptables/ufw) NAT and Port Forwarding Concepts TLS and Certificates Basics Network Troubleshooting Playbook
Storage & Filesystems
Disk Layout and Partitioning (lsblk/fdisk) Filesystems (ext4/xfs) Basics Mounts and /etc/fstab Safety Disk Usage: df vs du (and Why They Differ) Inodes and “Disk Full” Mysteries Swap Strategy in Production Log Rotation (logrotate) Basics Temporary and Runtime Storage Hygiene Backup Storage Considerations
Security Basics
SSH Hardening in Production Users and Least Privilege in Production Firewall Minimum Rules for Production Linux Patching Strategy in Production Managing Secrets on Linux Hosts Linux Audit and Basic Logging in Production Fail2ban and Rate Limiting in Production SELinux and AppArmor Overview in Production Linux Security Baseline Checklist for Production
Observability on Linux
Linux Log Sources Map in Production journalctl Power Usage in Production Log Structure and Correlation in Production Metrics Hooks Overview in Production Health Checks and Readiness in Production Tracing Mindset for Linux and Production Systems Core Dumps Basics in Production Debug Packages and Symbols in Production Linux Observability Checklist for Production
Performance & Debugging
Load Average Explained for Production Debugging CPU Profiling Basics in Linux Production Memory Pressure Basics in Linux Production IO Bottlenecks and iostat in Production lsof and Open Files in Production Debugging Practical strace Usage in Production Sockets Debugging Tools in Linux Production dmesg and Kernel Signals for Production Debugging Production Debugging Workflow for Linux Systems
Deployment Patterns
Environment and Configuration Management in Production Release Directory Pattern for Safe Linux Deployments Zero-Downtime Restarts in Production Running Migrations Safely in Production Secrets Deploy Patterns in Production Backups and Restore Drills in Production Rollback Strategies for Production Deployments Maintenance Windows in Production Operations Production Deployment Checklist for Linux Systems
Production Checklists
Writing Effective Runbooks for Production Systems Failure Modes Mapping for Linux Production Systems Incident Response Basics for Linux Production Break-Glass Access in Production Capacity Planning Basics for Linux Production Systems Monitoring and Alerting Baseline for Linux Production Production Readiness Review for Linux Systems Configuration Drift in Linux Production Systems Go-Live Checklist

Log Structure and Correlation in Production

Correlate logs across services and hosts using timestamps, request IDs, user/session context, and error signatures. Build incident timelines and avoid false conclusions from single-log views.
On this page

Why Correlation Beats “Single Log Debugging”

Production incidents rarely live in one place. A user request may touch load balancer, reverse proxy, app, database, cache, and external APIs. If you only read one log file, you will likely blame the wrong component. Correlation is how you reconstruct reality.

Symptom

  • 500 errors in the app but database looks normal
  • Intermittent timeouts that disappear when you look
  • Load balancer health checks fail but app seems fine locally
  • Security incident requires “who did what, when” timeline

Root Cause

  • Logs lack stable correlation fields (request_id, trace_id)
  • Time sync drift between hosts breaks ordering
  • Inconsistent log formats across services
  • Only error logs exist; no context logs

Investigation: Build a Timeline First

Step 1: Confirm time sync (a must for correlation):

timedatectl

If NTP is not synced, cross-host correlation becomes unreliable.

Step 2: Anchor on an Observable Event

Pick one hard anchor:

  • User-facing error timestamp
  • Alert fired time
  • Deploy time
  • Host reboot time

Example: get service errors around an alert window:

journalctl -u nginx --since "2026-02-24 03:00" --until "2026-02-24 03:10"

Step 3: Correlate by Request or Connection

Ideal: logs include request IDs (or trace IDs). If present, grep that ID across logs:

grep "request_id=abc123" /var/log/nginx/access.log
grep "request_id=abc123" /var/log/app/app.log

If request IDs do not exist, correlate by:

  • client IP
  • timestamp windows
  • URL path
  • status code + latency
  • database query signature

Common Correlation Fields to Standardize

  • timestamp (with timezone)
  • request_id / trace_id
  • user_id (if authenticated)
  • session_id
  • client_ip
  • method and path
  • status and latency
  • host and service identifiers

Mitigation: Make Logs Correlation-Friendly

1) Use structured logs (JSON) where possible

Text logs are grep-friendly but hard to parse at scale. JSON logs allow consistent parsing and querying.

2) Ensure every request gets an ID

At the edge (nginx or gateway), generate or forward an ID into the app and downstream calls.

3) Log both errors and context

An error without request context leads to guesswork. Minimal context includes request_id, path, user_id, and latency.

Operational Correlation Techniques

Correlate Nginx Access with Upstream Failures

Find 5xx spikes:

awk '$9 ~ /^5/ {print $4, $7, $9}' /var/log/nginx/access.log | tail -n 50

(Adjust fields depending on your log format.)

Correlate with System Resource Events

When apps time out, check if the host was under pressure:

journalctl -k --since "30 minutes ago" | tail -n 200

Look for OOM killer events, disk errors, network resets.

Hardening Strategy

  • Enforce NTP across fleet
  • Standardize log format and required fields
  • Centralize logs (searchable across hosts)
  • Ensure request_id propagation through reverse proxies and apps
  • Define retention so incident windows remain available

Verification Checklist

  • All hosts show NTP synchronized
  • Logs include timestamps with timezone
  • Request_id exists and is searchable end-to-end
  • Can reconstruct an incident timeline within minutes
timedatectl
journalctl --since "10 minutes ago" -p err

Why This Matters in Real Infrastructure

Incidents are solved by timelines, not opinions. Correlation turns scattered logs into a single story: what happened first, what failed next, and what was a downstream effect. Without correlation, teams chase the wrong cause and waste the critical early minutes of response.

Metrics Hooks Overview in Production →