Core Code
Core Code Modern Dev Handbook
LINUX-PRODUCTION Contents
Linux Fundamentals for Production
Filesystem Layout (FHS) in Practice Users, Groups, and Service Accounts Permissions & ACLs (chmod/chown/setuid) Sudoers Safe Defaults Packages: apt/dnf, Repos, and Pinning Shell & Environment Hygiene Editing Configs Safely Time, Timezones, and NTP System Introspection Toolkit
Processes & Signals
Linux Process Model (PID, PPID, Fork/Exec) ps/top/htop: Finding What Hurts Signals in Practice (TERM/KILL/HUP) nice/renice and Scheduling Basics ulimits and Resource Limits Background Jobs (nohup, tmux, systemd-run) /proc and What It Tells You cgroups Basics (CPU/Memory Isolation) Zombies, Orphans, and Stuck Processes
systemd & Services
systemd Unit Types Overview Service Unit Best Practices Dependencies & Ordering (After/Wants/Requires) Restart Policies and Backoff Environment Files and Secrets Handling journald Basics (Reading, Filtering, Retention) Logging Practices with systemd Timers vs Cron Debugging systemd Services
Networking Basics for Production
ip, routes, and interfaces DNS Debugging (dig, resolvectl, hosts) Ports and Sockets with ss curl for HTTP Debugging TCP Basics (Handshake, States, Retries) Firewall Overview (nftables/iptables/ufw) NAT and Port Forwarding Concepts TLS and Certificates Basics Network Troubleshooting Playbook
Storage & Filesystems
Disk Layout and Partitioning (lsblk/fdisk) Filesystems (ext4/xfs) Basics Mounts and /etc/fstab Safety Disk Usage: df vs du (and Why They Differ) Inodes and “Disk Full” Mysteries Swap Strategy in Production Log Rotation (logrotate) Basics Temporary and Runtime Storage Hygiene Backup Storage Considerations
Security Basics
SSH Hardening in Production Users and Least Privilege in Production Firewall Minimum Rules for Production Linux Patching Strategy in Production Managing Secrets on Linux Hosts Linux Audit and Basic Logging in Production Fail2ban and Rate Limiting in Production SELinux and AppArmor Overview in Production Linux Security Baseline Checklist for Production
Observability on Linux
Linux Log Sources Map in Production journalctl Power Usage in Production Log Structure and Correlation in Production Metrics Hooks Overview in Production Health Checks and Readiness in Production Tracing Mindset for Linux and Production Systems Core Dumps Basics in Production Debug Packages and Symbols in Production Linux Observability Checklist for Production
Performance & Debugging
Load Average Explained for Production Debugging CPU Profiling Basics in Linux Production Memory Pressure Basics in Linux Production IO Bottlenecks and iostat in Production lsof and Open Files in Production Debugging Practical strace Usage in Production Sockets Debugging Tools in Linux Production dmesg and Kernel Signals for Production Debugging Production Debugging Workflow for Linux Systems
Deployment Patterns
Environment and Configuration Management in Production Release Directory Pattern for Safe Linux Deployments Zero-Downtime Restarts in Production Running Migrations Safely in Production Secrets Deploy Patterns in Production Backups and Restore Drills in Production Rollback Strategies for Production Deployments Maintenance Windows in Production Operations Production Deployment Checklist for Linux Systems
Production Checklists
Writing Effective Runbooks for Production Systems Failure Modes Mapping for Linux Production Systems Incident Response Basics for Linux Production Break-Glass Access in Production Capacity Planning Basics for Linux Production Systems Monitoring and Alerting Baseline for Linux Production Production Readiness Review for Linux Systems Configuration Drift in Linux Production Systems Go-Live Checklist

Sockets Debugging Tools in Linux Production

Debug TCP/UDP socket issues using ss, netstat, lsof, tcpdump, and connection state analysis. Identify SYN floods, TIME-WAIT buildup, backlog exhaustion, and hidden network bottlenecks.
On this page

Why Socket Debugging Matters

Most production services communicate over TCP or UDP. When latency spikes or connections fail, the problem is often at the socket layer: connection backlog saturation, SYN floods, TIME-WAIT buildup, or file descriptor exhaustion.

Symptom

  • Connection refused or connection reset errors
  • Service reachable locally but not remotely
  • High number of TIME-WAIT connections
  • Intermittent timeouts under load
  • Load balancer marks instance unhealthy

Root Cause

  • Backlog queue full
  • Too many concurrent connections
  • File descriptor limits reached
  • Network packet drops or retransmissions
  • Firewall or NAT misconfiguration

Step 1: Inspect Listening Sockets

ss -ltnp

Check which process is bound to which port.

Step 2: Inspect Connection States

ss -tan

Common TCP states:

  • LISTEN
  • ESTABLISHED
  • SYN-SENT
  • SYN-RECV
  • TIME-WAIT
  • CLOSE-WAIT

Count Connections Per State

ss -tan | awk '{print $1}' | sort | uniq -c

Step 3: Detect Backlog Issues

ss -ltn

Compare Recv-Q and Send-Q values. High Recv-Q on a listening socket indicates backlog pressure.

Step 4: Identify Per-Process Connections

ss -tanp | grep PID

Or:

sudo lsof -iTCP -sTCP:ESTABLISHED

Step 5: Inspect Kernel Network Stats

netstat -s

Look for:

  • TCP retransmissions
  • Listen queue overflows
  • Dropped packets

Step 6: Packet-Level Inspection

sudo tcpdump -ni any port 443

Useful for confirming whether packets arrive or are dropped upstream.

Common Production Failure Patterns

  • SYN flood: many SYN-RECV entries
  • TIME-WAIT storm: high connection churn
  • CLOSE-WAIT accumulation: app not closing sockets properly
  • Connection resets: backend crashing or overloaded

Mitigation

  • Increase backlog size if justified
  • Tune net.core.somaxconn
  • Fix application connection handling
  • Enable connection pooling
  • Scale horizontally
  • Adjust load balancer settings

Check somaxconn:

cat /proc/sys/net/core/somaxconn

Production Safety

  • Investigate before tuning kernel parameters
  • Avoid random sysctl changes without load testing
  • Correlate socket states with application logs
  • Check file descriptor limits alongside socket counts

Verification Checklist

ss -ltnp
ss -tan | awk '{print $1}' | sort | uniq -c
netstat -s | grep -i listen
  • No excessive SYN-RECV buildup
  • No abnormal CLOSE-WAIT accumulation
  • Backlog not saturated
  • Connection states consistent with traffic level

Why This Matters in Real Infrastructure

Many “application bugs” are actually socket-level issues. Understanding TCP states and connection patterns allows engineers to isolate network bottlenecks quickly and prevent cascading outages during traffic spikes.

dmesg and Kernel Signals for Production Debugging →