Core Code
Core Code Modern Dev Handbook
LINUX-PRODUCTION Contents
Linux Fundamentals for Production
Filesystem Layout (FHS) in Practice Users, Groups, and Service Accounts Permissions & ACLs (chmod/chown/setuid) Sudoers Safe Defaults Packages: apt/dnf, Repos, and Pinning Shell & Environment Hygiene Editing Configs Safely Time, Timezones, and NTP System Introspection Toolkit
Processes & Signals
Linux Process Model (PID, PPID, Fork/Exec) ps/top/htop: Finding What Hurts Signals in Practice (TERM/KILL/HUP) nice/renice and Scheduling Basics ulimits and Resource Limits Background Jobs (nohup, tmux, systemd-run) /proc and What It Tells You cgroups Basics (CPU/Memory Isolation) Zombies, Orphans, and Stuck Processes
systemd & Services
systemd Unit Types Overview Service Unit Best Practices Dependencies & Ordering (After/Wants/Requires) Restart Policies and Backoff Environment Files and Secrets Handling journald Basics (Reading, Filtering, Retention) Logging Practices with systemd Timers vs Cron Debugging systemd Services
Networking Basics for Production
ip, routes, and interfaces DNS Debugging (dig, resolvectl, hosts) Ports and Sockets with ss curl for HTTP Debugging TCP Basics (Handshake, States, Retries) Firewall Overview (nftables/iptables/ufw) NAT and Port Forwarding Concepts TLS and Certificates Basics Network Troubleshooting Playbook
Storage & Filesystems
Disk Layout and Partitioning (lsblk/fdisk) Filesystems (ext4/xfs) Basics Mounts and /etc/fstab Safety Disk Usage: df vs du (and Why They Differ) Inodes and “Disk Full” Mysteries Swap Strategy in Production Log Rotation (logrotate) Basics Temporary and Runtime Storage Hygiene Backup Storage Considerations
Security Basics
SSH Hardening in Production Users and Least Privilege in Production Firewall Minimum Rules for Production Linux Patching Strategy in Production Managing Secrets on Linux Hosts Linux Audit and Basic Logging in Production Fail2ban and Rate Limiting in Production SELinux and AppArmor Overview in Production Linux Security Baseline Checklist for Production
Observability on Linux
Linux Log Sources Map in Production journalctl Power Usage in Production Log Structure and Correlation in Production Metrics Hooks Overview in Production Health Checks and Readiness in Production Tracing Mindset for Linux and Production Systems Core Dumps Basics in Production Debug Packages and Symbols in Production Linux Observability Checklist for Production
Performance & Debugging
Load Average Explained for Production Debugging CPU Profiling Basics in Linux Production Memory Pressure Basics in Linux Production IO Bottlenecks and iostat in Production lsof and Open Files in Production Debugging Practical strace Usage in Production Sockets Debugging Tools in Linux Production dmesg and Kernel Signals for Production Debugging Production Debugging Workflow for Linux Systems
Deployment Patterns
Environment and Configuration Management in Production Release Directory Pattern for Safe Linux Deployments Zero-Downtime Restarts in Production Running Migrations Safely in Production Secrets Deploy Patterns in Production Backups and Restore Drills in Production Rollback Strategies for Production Deployments Maintenance Windows in Production Operations Production Deployment Checklist for Linux Systems
Production Checklists
Writing Effective Runbooks for Production Systems Failure Modes Mapping for Linux Production Systems Incident Response Basics for Linux Production Break-Glass Access in Production Capacity Planning Basics for Linux Production Systems Monitoring and Alerting Baseline for Linux Production Production Readiness Review for Linux Systems Configuration Drift in Linux Production Systems Go-Live Checklist

Backup Storage Considerations

Snapshots vs file backups, retention, integrity checks, and restore testing.
On this page

Backup & Storage Strategy in Production

Backups are not about storing data.

Backups are about recovery guarantees.

The only question that matters:

Can you restore under pressure?

This lesson focuses on production-safe backup design, consistency, isolation, and restore validation.

Production Scenario 1 — Ransomware or Accidental Deletion

Symptoms

  • Critical data deleted
  • Files encrypted
  • Application unusable

Real Question

Do you have:

  • Recent backup?
  • Off-host backup?
  • Immutable backup?
  • Tested restore procedure?

Backup Types

File-Level (rsync, tar)

rsync -aHAX --delete /data/ /backup/data/
Pros:
  • Simple
  • Flexible
Cons:
  • May produce inconsistent state during writes
  • No atomic snapshot

Filesystem Snapshots (LVM)

lvcreate --size 5G --snapshot --name data_snap /dev/vg/data
Pros:
  • Point-in-time consistency
  • Fast
Cons:
  • Requires LVM

Cloud Snapshots

  • Block-level
  • Instant
  • Usually crash-consistent

Backup Consistency (Critical Concept)

Filesystem Consistency

Data blocks consistent, but app state may not be.

Application-Aware Backup

  • Database dump (mysqldump, pg_dump)
  • Flush and freeze operations
  • Pre-backup hooks

Never rely only on file copy for active databases.

Scenario 2 — rsync During Active Writes

If rsync runs while app writes:

  • Partial file copies
  • Corrupted state
Safer approach:
  • Stop app briefly
  • Use snapshot
  • Or database-native backup tools

Incremental vs Full Backup

Full

  • Complete copy
  • Large storage cost

Incremental

  • Only changed files
  • Efficient
  • More complex restore chain

Production strategy often combines:

  • Weekly full
  • Daily incremental

Retention Strategy

Common policy:

  • Daily: 7 days
  • Weekly: 4 weeks
  • Monthly: 12 months

Retention must balance:

  • Storage cost
  • Compliance
  • Recovery window

Ransomware Protection

  • Backup must be off-host
  • Use object storage with versioning
  • Enable immutability (WORM mode)
  • Separate credentials

If attacker can delete backups, backups are useless.

Scenario 3 — Backup Exists but Restore Fails

This is common.

Reasons:

  • Incomplete backup
  • Permission issues
  • Missing dependencies
  • Restore never tested

Restore Testing (Most Important Step)

Simulate recovery:

rsync -a /backup/data/ /restore-test/

Start application against restore-test environment.

Validate:

  • Data integrity
  • Permissions
  • Application startup

Backup without restore testing = false security.

Backup Monitoring

  • Alert on failed backup jobs
  • Check backup file size anomalies
  • Track duration changes

Mental Model

  • Backup is a recovery contract
  • Consistency matters more than frequency
  • Off-host is mandatory
  • Immutable storage protects against ransomware
  • Restore testing is non-optional

Common Production Mistakes

  • Keeping backups on same server
  • Not testing restore
  • Backing up corrupted data
  • Ignoring database-specific backup needs
  • No retention policy
  • No monitoring of backup jobs

Production Checklist

  • Use consistent backup method (snapshot or app-aware)
  • Keep backups off-host
  • Enable immutability/versioning
  • Implement retention strategy
  • Monitor backup success
  • Test restore quarterly
  • Document recovery procedure