Core Code
Core Code Modern Dev Handbook
LINUX-PRODUCTION Contents
Linux Fundamentals for Production
Filesystem Layout (FHS) in Practice Users, Groups, and Service Accounts Permissions & ACLs (chmod/chown/setuid) Sudoers Safe Defaults Packages: apt/dnf, Repos, and Pinning Shell & Environment Hygiene Editing Configs Safely Time, Timezones, and NTP System Introspection Toolkit
Processes & Signals
Linux Process Model (PID, PPID, Fork/Exec) ps/top/htop: Finding What Hurts Signals in Practice (TERM/KILL/HUP) nice/renice and Scheduling Basics ulimits and Resource Limits Background Jobs (nohup, tmux, systemd-run) /proc and What It Tells You cgroups Basics (CPU/Memory Isolation) Zombies, Orphans, and Stuck Processes
systemd & Services
systemd Unit Types Overview Service Unit Best Practices Dependencies & Ordering (After/Wants/Requires) Restart Policies and Backoff Environment Files and Secrets Handling journald Basics (Reading, Filtering, Retention) Logging Practices with systemd Timers vs Cron Debugging systemd Services
Networking Basics for Production
ip, routes, and interfaces DNS Debugging (dig, resolvectl, hosts) Ports and Sockets with ss curl for HTTP Debugging TCP Basics (Handshake, States, Retries) Firewall Overview (nftables/iptables/ufw) NAT and Port Forwarding Concepts TLS and Certificates Basics Network Troubleshooting Playbook
Storage & Filesystems
Disk Layout and Partitioning (lsblk/fdisk) Filesystems (ext4/xfs) Basics Mounts and /etc/fstab Safety Disk Usage: df vs du (and Why They Differ) Inodes and “Disk Full” Mysteries Swap Strategy in Production Log Rotation (logrotate) Basics Temporary and Runtime Storage Hygiene Backup Storage Considerations
Security Basics
SSH Hardening in Production Users and Least Privilege in Production Firewall Minimum Rules for Production Linux Patching Strategy in Production Managing Secrets on Linux Hosts Linux Audit and Basic Logging in Production Fail2ban and Rate Limiting in Production SELinux and AppArmor Overview in Production Linux Security Baseline Checklist for Production
Observability on Linux
Linux Log Sources Map in Production journalctl Power Usage in Production Log Structure and Correlation in Production Metrics Hooks Overview in Production Health Checks and Readiness in Production Tracing Mindset for Linux and Production Systems Core Dumps Basics in Production Debug Packages and Symbols in Production Linux Observability Checklist for Production
Performance & Debugging
Load Average Explained for Production Debugging CPU Profiling Basics in Linux Production Memory Pressure Basics in Linux Production IO Bottlenecks and iostat in Production lsof and Open Files in Production Debugging Practical strace Usage in Production Sockets Debugging Tools in Linux Production dmesg and Kernel Signals for Production Debugging Production Debugging Workflow for Linux Systems
Deployment Patterns
Environment and Configuration Management in Production Release Directory Pattern for Safe Linux Deployments Zero-Downtime Restarts in Production Running Migrations Safely in Production Secrets Deploy Patterns in Production Backups and Restore Drills in Production Rollback Strategies for Production Deployments Maintenance Windows in Production Operations Production Deployment Checklist for Linux Systems
Production Checklists
Writing Effective Runbooks for Production Systems Failure Modes Mapping for Linux Production Systems Incident Response Basics for Linux Production Break-Glass Access in Production Capacity Planning Basics for Linux Production Systems Monitoring and Alerting Baseline for Linux Production Production Readiness Review for Linux Systems Configuration Drift in Linux Production Systems Go-Live Checklist

ulimits and Resource Limits

Prevent outages with file descriptor limits, process limits, and sane defaults.
On this page

Why Resource Limits Matter in Production

Linux allows you to limit how much a process can consume:

  • Open files
  • Number of processes
  • Memory
  • CPU time

If these limits are too low, services fail silently. If they are too high without control, runaway processes can crash the system. Production stability depends on sane limits.

Check Current Limits

ulimit -a

Common critical values:

  • open files (-n)
  • max user processes (-u)
  • core file size (-c)

The Classic Production Failure: Too Many Open Files

Symptoms:

  • New connections fail
  • Error: EMFILE
  • Service appears alive but not accepting traffic

Check file descriptor usage: 

ulimit -n
cat /proc/<pid>/limits
ls /proc/<pid>/fd | wc -l

Increase File Descriptor Limit (Temporary)

ulimit -n 65535

This only affects the current shell. It does NOT persist across restarts.

Persistent Limits via systemd

For production services, define limits in the unit file: 

[Service]
LimitNOFILE=65535
LimitNPROC=4096

Then reload: 

sudo systemctl daemon-reload
sudo systemctl restart myapp

Always prefer systemd limits over shell ulimit hacks.

Global Limits Configuration

System-wide limits can be defined in: 

/etc/security/limits.conf
/etc/security/limits.d/*.conf

Example: 

myapp soft nofile 65535
myapp hard nofile 65535

Process Count Limits

If a service forks too many processes, you may hit:

  • fork: Resource temporarily unavailable
  • Cannot allocate memory

Check: 

ulimit -u
cat /proc/<pid>/limits

Core Dumps and Debugging

If core dump size is 0, crashes produce no dump. Enable for debugging: 

ulimit -c unlimited

For production, be careful: core dumps can be large and contain secrets.

Hard vs Soft Limits

  • Soft limit: current usable limit
  • Hard limit: maximum value soft can be raised to

Only privileged users can raise hard limits.

Investigating Limit-Related Incidents

If you suspect limit exhaustion: 

cat /proc/<pid>/limits
lsof | wc -l
journalctl -u myapp -n 100 --no-pager

Look for:

  • EMFILE
  • Cannot fork
  • Too many open files

Common Production Mistakes

  • Raising limits blindly without understanding workload
  • Setting huge global limits instead of per-service limits
  • Ignoring file descriptor leaks
  • Using shell ulimit instead of systemd Limit directives

Mental Model

Resource limits are safety rails. Too low = service failure. Too high = runaway damage. Production engineering means setting limits aligned with expected workload.

Production Checklist

  • Review LimitNOFILE for high-connection services
  • Configure limits in systemd units
  • Monitor file descriptor usage
  • Check limits during unexplained connection failures
  • Align limits with capacity planning
Background Jobs (nohup, tmux, systemd-run) →