Core Code
Core Code Modern Dev Handbook
LINUX-PRODUCTION Contents
Linux Fundamentals for Production
Filesystem Layout (FHS) in Practice Users, Groups, and Service Accounts Permissions & ACLs (chmod/chown/setuid) Sudoers Safe Defaults Packages: apt/dnf, Repos, and Pinning Shell & Environment Hygiene Editing Configs Safely Time, Timezones, and NTP System Introspection Toolkit
Processes & Signals
Linux Process Model (PID, PPID, Fork/Exec) ps/top/htop: Finding What Hurts Signals in Practice (TERM/KILL/HUP) nice/renice and Scheduling Basics ulimits and Resource Limits Background Jobs (nohup, tmux, systemd-run) /proc and What It Tells You cgroups Basics (CPU/Memory Isolation) Zombies, Orphans, and Stuck Processes
systemd & Services
systemd Unit Types Overview Service Unit Best Practices Dependencies & Ordering (After/Wants/Requires) Restart Policies and Backoff Environment Files and Secrets Handling journald Basics (Reading, Filtering, Retention) Logging Practices with systemd Timers vs Cron Debugging systemd Services
Networking Basics for Production
ip, routes, and interfaces DNS Debugging (dig, resolvectl, hosts) Ports and Sockets with ss curl for HTTP Debugging TCP Basics (Handshake, States, Retries) Firewall Overview (nftables/iptables/ufw) NAT and Port Forwarding Concepts TLS and Certificates Basics Network Troubleshooting Playbook
Storage & Filesystems
Disk Layout and Partitioning (lsblk/fdisk) Filesystems (ext4/xfs) Basics Mounts and /etc/fstab Safety Disk Usage: df vs du (and Why They Differ) Inodes and “Disk Full” Mysteries Swap Strategy in Production Log Rotation (logrotate) Basics Temporary and Runtime Storage Hygiene Backup Storage Considerations
Security Basics
SSH Hardening in Production Users and Least Privilege in Production Firewall Minimum Rules for Production Linux Patching Strategy in Production Managing Secrets on Linux Hosts Linux Audit and Basic Logging in Production Fail2ban and Rate Limiting in Production SELinux and AppArmor Overview in Production Linux Security Baseline Checklist for Production
Observability on Linux
Linux Log Sources Map in Production journalctl Power Usage in Production Log Structure and Correlation in Production Metrics Hooks Overview in Production Health Checks and Readiness in Production Tracing Mindset for Linux and Production Systems Core Dumps Basics in Production Debug Packages and Symbols in Production Linux Observability Checklist for Production
Performance & Debugging
Load Average Explained for Production Debugging CPU Profiling Basics in Linux Production Memory Pressure Basics in Linux Production IO Bottlenecks and iostat in Production lsof and Open Files in Production Debugging Practical strace Usage in Production Sockets Debugging Tools in Linux Production dmesg and Kernel Signals for Production Debugging Production Debugging Workflow for Linux Systems
Deployment Patterns
Environment and Configuration Management in Production Release Directory Pattern for Safe Linux Deployments Zero-Downtime Restarts in Production Running Migrations Safely in Production Secrets Deploy Patterns in Production Backups and Restore Drills in Production Rollback Strategies for Production Deployments Maintenance Windows in Production Operations Production Deployment Checklist for Linux Systems
Production Checklists
Writing Effective Runbooks for Production Systems Failure Modes Mapping for Linux Production Systems Incident Response Basics for Linux Production Break-Glass Access in Production Capacity Planning Basics for Linux Production Systems Monitoring and Alerting Baseline for Linux Production Production Readiness Review for Linux Systems Configuration Drift in Linux Production Systems Go-Live Checklist

Debug Packages and Symbols in Production

Use debug symbols and debug packages to turn crashes into actionable stack traces. Learn when to install symbols, how to fetch them safely, and how to avoid bloating production images.
On this page

Why Debug Symbols Change Everything

A core dump without symbols often shows meaningless function names, missing line numbers, and unreadable stack traces. Debug symbols let you map memory addresses back to real functions and source lines. This turns “it crashed” into “it crashed here, for this reason”.

Symptom

  • Core dump backtrace shows “??” frames
  • Stack traces contain only memory addresses
  • Native crashes cannot be attributed to a library or function
  • Post-mortem debugging provides no actionable detail

Root Cause

  • Binary stripped of symbols (common in production builds)
  • Debug packages not installed
  • Separate debuginfo repository not enabled
  • Symbols not matching the exact binary version

Investigation

1) Confirm If a Binary Is Stripped

file /usr/bin/nginx

If it says “stripped”, debug symbols are not present.

2) Check If You Have a Core Dump Without Symbols

coredumpctl list
coredumpctl gdb PID

Inside gdb:

bt

If frames show “??”, symbols are missing.

Mitigation: Acquire Matching Debug Symbols

Key rule: symbols must match the exact package version of the binary and its libraries.

Debian/Ubuntu: dbgsym Packages

Ubuntu and Debian often provide separate dbgsym packages via a debug symbol repository.

Check package version:

dpkg -l | grep nginx

Install symbols (example pattern):

sudo apt install nginx-dbgsym

For libraries involved in the crash, install their dbgsym packages as well.

RHEL-based: debuginfo Packages

Enable debuginfo repositories and install debuginfo:

sudo dnf debuginfo-install nginx

List installed debuginfo:

rpm -qa | grep debuginfo

Operational Safety in Production

  • Do not permanently bloat production nodes with debuginfo unless required
  • Prefer reproducing in staging with the same build and symbols
  • If production analysis is required, install symbols temporarily and remove after investigation

Debug Symbols for Your Own Binaries

If you build native binaries, keep separate symbol files and store them in artifact storage.

Example build mindset:

  • Production binary can be stripped
  • Symbols stored separately and fetched when needed
  • Symbols versioned with build ID

Mitigation Workflow: Turning “Crash” Into “Fix”

  • Capture core dump
  • Confirm binary version and library versions
  • Install matching debug symbols (or fetch from artifact store)
  • Re-run gdb backtrace
  • Identify the crashing function and call chain
  • Correlate with deploy changes and inputs

Verification Checklist

After installing symbols, gdb backtrace should become readable:

coredumpctl gdb PID

Inside gdb:

bt
info threads
  • Stack frames show function names
  • Thread context visible
  • Crash location attributable to a library or module

Hardening Strategy

  • Maintain symbol artifacts for every production build
  • Ensure symbol/binary version matching discipline
  • Use staging reproduction as primary analysis environment
  • Keep debuginfo installation as an emergency-only step on prod

Why This Matters in Real Infrastructure

Many “random production crashes” are solvable, but only if you can see where they happen. Debug symbols turn post-mortem debugging into a reliable engineering loop. Without symbols, teams restart services blindly and accept recurring outages as fate.

Linux Observability Checklist for Production →