Core Code
Core Code Modern Dev Handbook
LINUX-PRODUCTION Contents
Linux Fundamentals for Production
Filesystem Layout (FHS) in Practice Users, Groups, and Service Accounts Permissions & ACLs (chmod/chown/setuid) Sudoers Safe Defaults Packages: apt/dnf, Repos, and Pinning Shell & Environment Hygiene Editing Configs Safely Time, Timezones, and NTP System Introspection Toolkit
Processes & Signals
Linux Process Model (PID, PPID, Fork/Exec) ps/top/htop: Finding What Hurts Signals in Practice (TERM/KILL/HUP) nice/renice and Scheduling Basics ulimits and Resource Limits Background Jobs (nohup, tmux, systemd-run) /proc and What It Tells You cgroups Basics (CPU/Memory Isolation) Zombies, Orphans, and Stuck Processes
systemd & Services
systemd Unit Types Overview Service Unit Best Practices Dependencies & Ordering (After/Wants/Requires) Restart Policies and Backoff Environment Files and Secrets Handling journald Basics (Reading, Filtering, Retention) Logging Practices with systemd Timers vs Cron Debugging systemd Services
Networking Basics for Production
ip, routes, and interfaces DNS Debugging (dig, resolvectl, hosts) Ports and Sockets with ss curl for HTTP Debugging TCP Basics (Handshake, States, Retries) Firewall Overview (nftables/iptables/ufw) NAT and Port Forwarding Concepts TLS and Certificates Basics Network Troubleshooting Playbook
Storage & Filesystems
Disk Layout and Partitioning (lsblk/fdisk) Filesystems (ext4/xfs) Basics Mounts and /etc/fstab Safety Disk Usage: df vs du (and Why They Differ) Inodes and “Disk Full” Mysteries Swap Strategy in Production Log Rotation (logrotate) Basics Temporary and Runtime Storage Hygiene Backup Storage Considerations
Security Basics
SSH Hardening in Production Users and Least Privilege in Production Firewall Minimum Rules for Production Linux Patching Strategy in Production Managing Secrets on Linux Hosts Linux Audit and Basic Logging in Production Fail2ban and Rate Limiting in Production SELinux and AppArmor Overview in Production Linux Security Baseline Checklist for Production
Observability on Linux
Linux Log Sources Map in Production journalctl Power Usage in Production Log Structure and Correlation in Production Metrics Hooks Overview in Production Health Checks and Readiness in Production Tracing Mindset for Linux and Production Systems Core Dumps Basics in Production Debug Packages and Symbols in Production Linux Observability Checklist for Production
Performance & Debugging
Load Average Explained for Production Debugging CPU Profiling Basics in Linux Production Memory Pressure Basics in Linux Production IO Bottlenecks and iostat in Production lsof and Open Files in Production Debugging Practical strace Usage in Production Sockets Debugging Tools in Linux Production dmesg and Kernel Signals for Production Debugging Production Debugging Workflow for Linux Systems
Deployment Patterns
Environment and Configuration Management in Production Release Directory Pattern for Safe Linux Deployments Zero-Downtime Restarts in Production Running Migrations Safely in Production Secrets Deploy Patterns in Production Backups and Restore Drills in Production Rollback Strategies for Production Deployments Maintenance Windows in Production Operations Production Deployment Checklist for Linux Systems
Production Checklists
Writing Effective Runbooks for Production Systems Failure Modes Mapping for Linux Production Systems Incident Response Basics for Linux Production Break-Glass Access in Production Capacity Planning Basics for Linux Production Systems Monitoring and Alerting Baseline for Linux Production Production Readiness Review for Linux Systems Configuration Drift in Linux Production Systems Go-Live Checklist

journald Basics (Reading, Filtering, Retention)

Use journalctl effectively and configure retention without losing critical data.
On this page

Why journald Matters in Production

In production, logs are your timeline. systemd-journald centralizes logs from services, kernel, and system components. If you do not know journalctl, you will waste time tailing random files and miss critical context.

What journald Collects

  • systemd service stdout/stderr (when configured)
  • Kernel messages
  • System services logs
  • Boot and shutdown events

This means you can correlate incidents across services using a single interface.

Basic Queries You Must Know

Logs for a Service

journalctl -u myapp -n 200 --no-pager

Follow Logs (Like tail -f)

journalctl -u myapp -f

Logs Since a Time Window

journalctl -u myapp --since "10 minutes ago" --no-pager
journalctl -u myapp --since "2026-02-24 18:00:00" --no-pager

Logs for Current Boot

Extremely useful after reboot: 

journalctl -b --no-pager
journalctl -b -u myapp --no-pager

Previous Boot (Why Did It Crash?)

journalctl -b -1 --no-pager

Filter by Priority (Errors Only)

Priorities:

  • 0 emerg
  • 1 alert
  • 2 crit
  • 3 err
  • 4 warning
  • 5 notice
  • 6 info
  • 7 debug
journalctl -u myapp -p err --no-pager
journalctl -p warning..err --no-pager

Structured Fields: More Than Text

journald stores metadata fields like:

  • _PID, _UID
  • _SYSTEMD_UNIT
  • _COMM
  • MESSAGE

View fields: 

journalctl -u myapp -o verbose | head -n 60

Output Formats

Useful formats:

  • -o short-iso (readable timestamps)
  • -o json (machine parsing)
journalctl -u myapp -o short-iso --no-pager
journalctl -u myapp -o json --no-pager | head

Disk Usage and Retention

In production, journals can grow. Check size: 

journalctl --disk-usage

Vacuum old logs: 

journalctl --vacuum-time=7d
journalctl --vacuum-size=1G

Do not vacuum blindly during incidents; preserve evidence first.

Persistent vs Volatile Journals

Some systems keep logs only in memory (lost on reboot). If you want persistence, ensure: 

/var/log/journal

And journald config: 

/etc/systemd/journald.conf
Storage=persistent

Rate Limiting

During crash loops, logs can flood. journald has rate limiting to protect the system. If you see missing logs, rate limiting may be dropping messages. Check journald settings:

  • RateLimitIntervalSec
  • RateLimitBurst

Incident Workflow with journalctl

A reliable workflow:

  • Check unit status
  • Check logs for current boot
  • Check last 10 minutes and errors only
  • Check previous boot if needed
systemctl status myapp --no-pager
journalctl -b -u myapp -n 200 --no-pager
journalctl -u myapp --since "10 minutes ago" -p err --no-pager

Common Production Mistakes

  • Relying only on /var/log files and missing unit stdout/stderr
  • Not using -b / -b -1 after reboots
  • Letting journals grow without retention policy
  • Vacuuming logs during an incident (destroying evidence)
  • Ignoring rate-limiting signs during crash loops

Mental Model

journald is your central timeline. Use it like an event database: filter by unit, time, priority, and boot. Production debugging becomes faster when you stop hunting logs across the filesystem.

Production Checklist

  • Know journalctl -u, -f, --since, -b, -b -1
  • Use priority filters (-p err) during incidents
  • Set retention policy (vacuum-time/size)
  • Ensure persistent storage if required
  • Be aware of rate limiting during crash loops
Logging Practices with systemd →