Core Code
Core Code Modern Dev Handbook
LINUX-PRODUCTION Contents
Linux Fundamentals for Production
Filesystem Layout (FHS) in Practice Users, Groups, and Service Accounts Permissions & ACLs (chmod/chown/setuid) Sudoers Safe Defaults Packages: apt/dnf, Repos, and Pinning Shell & Environment Hygiene Editing Configs Safely Time, Timezones, and NTP System Introspection Toolkit
Processes & Signals
Linux Process Model (PID, PPID, Fork/Exec) ps/top/htop: Finding What Hurts Signals in Practice (TERM/KILL/HUP) nice/renice and Scheduling Basics ulimits and Resource Limits Background Jobs (nohup, tmux, systemd-run) /proc and What It Tells You cgroups Basics (CPU/Memory Isolation) Zombies, Orphans, and Stuck Processes
systemd & Services
systemd Unit Types Overview Service Unit Best Practices Dependencies & Ordering (After/Wants/Requires) Restart Policies and Backoff Environment Files and Secrets Handling journald Basics (Reading, Filtering, Retention) Logging Practices with systemd Timers vs Cron Debugging systemd Services
Networking Basics for Production
ip, routes, and interfaces DNS Debugging (dig, resolvectl, hosts) Ports and Sockets with ss curl for HTTP Debugging TCP Basics (Handshake, States, Retries) Firewall Overview (nftables/iptables/ufw) NAT and Port Forwarding Concepts TLS and Certificates Basics Network Troubleshooting Playbook
Storage & Filesystems
Disk Layout and Partitioning (lsblk/fdisk) Filesystems (ext4/xfs) Basics Mounts and /etc/fstab Safety Disk Usage: df vs du (and Why They Differ) Inodes and “Disk Full” Mysteries Swap Strategy in Production Log Rotation (logrotate) Basics Temporary and Runtime Storage Hygiene Backup Storage Considerations
Security Basics
SSH Hardening in Production Users and Least Privilege in Production Firewall Minimum Rules for Production Linux Patching Strategy in Production Managing Secrets on Linux Hosts Linux Audit and Basic Logging in Production Fail2ban and Rate Limiting in Production SELinux and AppArmor Overview in Production Linux Security Baseline Checklist for Production
Observability on Linux
Linux Log Sources Map in Production journalctl Power Usage in Production Log Structure and Correlation in Production Metrics Hooks Overview in Production Health Checks and Readiness in Production Tracing Mindset for Linux and Production Systems Core Dumps Basics in Production Debug Packages and Symbols in Production Linux Observability Checklist for Production
Performance & Debugging
Load Average Explained for Production Debugging CPU Profiling Basics in Linux Production Memory Pressure Basics in Linux Production IO Bottlenecks and iostat in Production lsof and Open Files in Production Debugging Practical strace Usage in Production Sockets Debugging Tools in Linux Production dmesg and Kernel Signals for Production Debugging Production Debugging Workflow for Linux Systems
Deployment Patterns
Environment and Configuration Management in Production Release Directory Pattern for Safe Linux Deployments Zero-Downtime Restarts in Production Running Migrations Safely in Production Secrets Deploy Patterns in Production Backups and Restore Drills in Production Rollback Strategies for Production Deployments Maintenance Windows in Production Operations Production Deployment Checklist for Linux Systems
Production Checklists
Writing Effective Runbooks for Production Systems Failure Modes Mapping for Linux Production Systems Incident Response Basics for Linux Production Break-Glass Access in Production Capacity Planning Basics for Linux Production Systems Monitoring and Alerting Baseline for Linux Production Production Readiness Review for Linux Systems Configuration Drift in Linux Production Systems Go-Live Checklist

Background Jobs (nohup, tmux, systemd-run)

Run long tasks reliably without losing sessions or leaving orphaned processes.
On this page

Why Background Execution Is Dangerous in Production

Running something in the background is easy. Running it safely in production is not. Improper background execution can result in:

  • Half-completed migrations
  • Lost logs
  • Orphaned processes
  • Jobs silently dying after SSH disconnect

In production, background execution must be intentional.

The SSH Problem

When you close an SSH session, the shell sends SIGHUP to child processes. If your job does not handle it, it may terminate.

Basic Background Operator (&)

./script.sh &

This runs in background but still tied to the shell. Closing SSH can kill it. Not production-safe by default.

nohup: Survive SIGHUP

nohup prevents the process from receiving SIGHUP. 

nohup ./script.sh > output.log 2>&1 &

Important:

  • Always redirect output explicitly
  • Monitor logs
  • Confirm process is still running

Check Background Job Status

jobs
ps aux | grep script.sh

Do not assume it is running — verify.

Better Tool: tmux

tmux creates a persistent terminal session independent of SSH. 

tmux new -s maintenance

Detach: 

Ctrl+B then D

Reattach later: 

tmux attach -t maintenance

tmux is safer than nohup for interactive or long-running maintenance tasks.

Best Practice for Production: systemd-run

Instead of manual background jobs, prefer systemd supervision. 

systemd-run --unit=manual-job --scope ./script.sh

Advantages:

  • Proper logging in journal
  • Controlled lifecycle
  • Resource tracking
  • Better auditability

Orphaned Processes

If a parent shell dies, background processes may become orphaned and adopted by PID 1. This can make debugging harder because:

  • No clear owner
  • No supervision
  • No structured logs

When NOT to Use Background Jobs

  • Database migrations
  • Production schema changes
  • Critical deploy steps
  • Anything without clear monitoring

If the job matters, supervise it.

How to Verify a Job Is Still Running

ps -ef --forest | grep script
journalctl -u manual-job

Common Production Mistakes

  • Running migrations with & and disconnecting
  • Forgetting output redirection
  • Using screen/tmux but never checking logs
  • Leaving orphaned heavy jobs consuming CPU

Mental Model

Background execution without supervision is risk. Production jobs should be:

  • Observable
  • Supervised
  • Logged
  • Recoverable

Production Checklist

  • Prefer systemd-run or proper service units
  • If using nohup, redirect logs explicitly
  • Use tmux for interactive maintenance
  • Always verify process survival after SSH exit
  • Avoid unsupervised background critical tasks
/proc and What It Tells You →