Core Code
Core Code Modern Dev Handbook
LINUX-PRODUCTION Contents
Linux Fundamentals for Production
Filesystem Layout (FHS) in Practice Users, Groups, and Service Accounts Permissions & ACLs (chmod/chown/setuid) Sudoers Safe Defaults Packages: apt/dnf, Repos, and Pinning Shell & Environment Hygiene Editing Configs Safely Time, Timezones, and NTP System Introspection Toolkit
Processes & Signals
Linux Process Model (PID, PPID, Fork/Exec) ps/top/htop: Finding What Hurts Signals in Practice (TERM/KILL/HUP) nice/renice and Scheduling Basics ulimits and Resource Limits Background Jobs (nohup, tmux, systemd-run) /proc and What It Tells You cgroups Basics (CPU/Memory Isolation) Zombies, Orphans, and Stuck Processes
systemd & Services
systemd Unit Types Overview Service Unit Best Practices Dependencies & Ordering (After/Wants/Requires) Restart Policies and Backoff Environment Files and Secrets Handling journald Basics (Reading, Filtering, Retention) Logging Practices with systemd Timers vs Cron Debugging systemd Services
Networking Basics for Production
ip, routes, and interfaces DNS Debugging (dig, resolvectl, hosts) Ports and Sockets with ss curl for HTTP Debugging TCP Basics (Handshake, States, Retries) Firewall Overview (nftables/iptables/ufw) NAT and Port Forwarding Concepts TLS and Certificates Basics Network Troubleshooting Playbook
Storage & Filesystems
Disk Layout and Partitioning (lsblk/fdisk) Filesystems (ext4/xfs) Basics Mounts and /etc/fstab Safety Disk Usage: df vs du (and Why They Differ) Inodes and “Disk Full” Mysteries Swap Strategy in Production Log Rotation (logrotate) Basics Temporary and Runtime Storage Hygiene Backup Storage Considerations
Security Basics
SSH Hardening in Production Users and Least Privilege in Production Firewall Minimum Rules for Production Linux Patching Strategy in Production Managing Secrets on Linux Hosts Linux Audit and Basic Logging in Production Fail2ban and Rate Limiting in Production SELinux and AppArmor Overview in Production Linux Security Baseline Checklist for Production
Observability on Linux
Linux Log Sources Map in Production journalctl Power Usage in Production Log Structure and Correlation in Production Metrics Hooks Overview in Production Health Checks and Readiness in Production Tracing Mindset for Linux and Production Systems Core Dumps Basics in Production Debug Packages and Symbols in Production Linux Observability Checklist for Production
Performance & Debugging
Load Average Explained for Production Debugging CPU Profiling Basics in Linux Production Memory Pressure Basics in Linux Production IO Bottlenecks and iostat in Production lsof and Open Files in Production Debugging Practical strace Usage in Production Sockets Debugging Tools in Linux Production dmesg and Kernel Signals for Production Debugging Production Debugging Workflow for Linux Systems
Deployment Patterns
Environment and Configuration Management in Production Release Directory Pattern for Safe Linux Deployments Zero-Downtime Restarts in Production Running Migrations Safely in Production Secrets Deploy Patterns in Production Backups and Restore Drills in Production Rollback Strategies for Production Deployments Maintenance Windows in Production Operations Production Deployment Checklist for Linux Systems
Production Checklists
Writing Effective Runbooks for Production Systems Failure Modes Mapping for Linux Production Systems Incident Response Basics for Linux Production Break-Glass Access in Production Capacity Planning Basics for Linux Production Systems Monitoring and Alerting Baseline for Linux Production Production Readiness Review for Linux Systems Configuration Drift in Linux Production Systems Go-Live Checklist

Temporary and Runtime Storage Hygiene

Manage /tmp, /var/tmp, /run, and cleanup policies without breaking apps.
On this page

Runtime & Temporary Storage in Production (/tmp, tmpfs, Cleanup)

Temporary directories are silent failure zones in production systems.

Most engineers ignore them — until:

  • Disk fills unexpectedly
  • Inodes are exhausted
  • Memory pressure increases due to tmpfs
  • Applications fail with "No space left on device"

This lesson focuses on runtime storage behavior, tmpfs risks, cleanup automation, and real production incidents.

/tmp vs /var/tmp (Production Differences)

  • /tmp — intended for short-lived temporary files
  • /var/tmp — files expected to survive reboots

On many modern systems, /tmp is mounted as tmpfs (memory-backed).

Check mount type:

findmnt /tmp
If output shows tmpfs, it consumes RAM + swap, not disk.

Scenario 1 — tmpfs Consumes All Memory

Symptoms

  • High memory usage
  • Swap increasing
  • System slow under load

Diagnosis

df -hT /tmp

free -h

If /tmp is tmpfs, large files directly impact memory pressure.

Root Cause

  • Application writes large temp files
  • Image processing jobs
  • Batch exports

Mitigation

  • Limit tmpfs size in fstab
  • Redirect temp storage to disk-backed path
Example limit:

tmpfs /tmp tmpfs defaults,size=1G 0 0

Scenario 2 — /tmp Fills Disk

If /tmp is disk-backed, runaway temp files can fill root.

Diagnosis

du -xhd1 /tmp
Find large files:

sudo find /tmp -type f -size +100M -exec ls -lh {} \;

systemd-tmpfiles (Automatic Cleanup)

Modern systems use systemd-tmpfiles for cleanup policies.

Check configuration:

ls /usr/lib/tmpfiles.d/
ls /etc/tmpfiles.d/
Default policy often clears /tmp after a time threshold.

Manual cleanup run:

sudo systemd-tmpfiles --clean

Scenario 3 — Inode Leak via Temp Files

Thousands of small files in /tmp exhaust inodes.

Check inode usage:

df -i
Find file count:

sudo find /tmp -type f | wc -l
Root cause often:
  • Uncleaned session files
  • Crash loops
  • Bad cron jobs

Container-Specific tmp Issues

Containers may mount /tmp inside overlay or memory-limited environment.

Check container mount:

mount | grep tmp
Memory limits via cgroups may restrict tmpfs unexpectedly.

Safe Cleanup Strategy

Before deleting anything in /tmp:

  • Verify active processes
  • Check file age
  • Avoid deleting sockets or runtime locks
Delete files older than 7 days:

sudo find /tmp -type f -mtime +7 -delete

Security Considerations

/tmp is world-writable. Always ensure correct mount options:

nodev,nosuid,noexec
Example fstab entry:

tmpfs /tmp tmpfs defaults,nodev,nosuid,noexec,size=1G 0 0

Scenario 4 — Service Crash Due to /run Fill

/run is also tmpfs and used for runtime sockets and PID files.

Check:

df -hT /run
If full → services cannot start.

Root cause usually runaway socket files or broken daemons.

Mental Model

  • /tmp may be RAM-backed
  • tmpfs consumes memory + swap
  • Temporary does not mean self-cleaning
  • Runtime paths are critical infrastructure
  • Inodes can exhaust even in temp storage

Common Production Mistakes

  • Ignoring tmpfs memory impact
  • Not limiting tmpfs size
  • Blindly deleting active runtime files
  • Ignoring inode usage in /tmp
  • Not setting secure mount options
  • Assuming container cleans temp automatically

Production Checklist

  • Check mount type of /tmp
  • Monitor tmpfs size and usage
  • Limit tmpfs size explicitly
  • Monitor inode usage
  • Configure systemd-tmpfiles cleanup
  • Set secure mount options
  • Validate container runtime temp behavior
Backup Storage Considerations →