Core Code
Core Code Modern Dev Handbook
INFRA-DEVOPS Contents
Linux for Production
Processes, Threads, and Signals Filesystems, Mounts, and Permissions systemd Units and Service Lifecycle journald and Production Logging ulimits, cgroups, and Resource Limits /proc, /sys, and Safe Debugging Performance Basics: load, iowait, run queue Package Management and Patch Strategy Users, Groups, sudo, Capabilities Production Troubleshooting Playbook
Networking for Production
Operator Networking Mental Model (TCP/IP) DNS in Production: TTL, caching, outages TLS Handshake and Certificates Timeouts and Retries Across Layers Load Balancers and Health Checks Reverse Proxies and API Gateways Firewalls, NAT, and Egress Identity Network Debugging Tools (ss, tcpdump, curl) Failure Patterns: partial outage, brownout, storm Connectivity Checklist (DNS→TCP→TLS→HTTP)
Compute, Storage, and Resource Management
CPU, Memory, and I/O Basics Capacity Planning and Headroom Disk Latency, IOPS, Throughput, Queue Depth Disk Full vs Inode Full and Cleanup Strategy Resource Isolation and Noisy Neighbors Quotas, Limits, and Guardrails Horizontal vs Vertical Scaling Ephemeral vs Persistent State Virtualization vs Containers (Failure Domains) Resource Metrics Primer (what to alert on)
Containers & OCI
OCI and Runtime Architecture (containerd, runc) Dockerfile Production Patterns Image Layering and Build Caching Registries: Digests vs Tags and Promotion Container Networking Model (bridge/NAT/ports) OverlayFS and Volumes (persistence & perf) Signals, PID 1, and Graceful Shutdown Rootless Containers and Tradeoffs Container Security Basics (caps/seccomp) Debugging Containers (inspect/logs/exec)
Kubernetes Core
Kubernetes Architecture (control plane vs data plane) Workloads: Deployment, Job, DaemonSet, StatefulSet Deployments and Rollouts (readiness, rollback) Services, Endpoints, and Cluster DNS Ingress and Service Exposure ConfigMaps and Secrets Probes: readiness, liveness, startup Resource Requests and Limits Scheduling Basics (taints/affinity/topology) Namespaces, RBAC, and Isolation Basics
Kubernetes Operations
kubectl Production Workflow Debugging Pods and Nodes (events, logs, states) Cluster Upgrades and Versioning Strategy Disruption Budgets, Drains, and Maintenance Autoscaling: HPA, VPA, Cluster Autoscaler NetworkPolicy and Traffic Control RBAC and Access Control (least privilege) etcd Basics and Backup Strategy Backups and Disaster Recovery for Kubernetes Incident Response on Kubernetes
Release Engineering & CI/CD
CI Pipeline Architecture Build Once, Promote Everywhere (Artifacts) Testing Strategy: unit→integration→e2e Versioning: SemVer, tags, and changelogs Deployment Strategies: rolling, blue/green, canary Feature Flags and Progressive Delivery GitOps Introduction (pull-based delivery) Secrets in CI/CD and Environment Injection Rollback Strategy and Release Safety Supply Chain Validation in Pipelines
Observability
Logs vs Metrics vs Traces Structured Logging and Correlation IDs Metrics Design Principles SLIs, SLOs, and Error Budgets Alerting Strategy and Alert Fatigue Distributed Tracing Basics Sampling and High Cardinality Problems Dashboards That Drive Action Golden Signals and RED/USE methods Observability During Rollouts (what to watch)
Incident Response & Reliability
Incident Lifecycle and Roles Triage and Stabilization (reduce blast radius) Runbooks and Operational Checklists Root Cause Analysis (RCA) Blameless Postmortems and Learning On-call Design and Escalation Error Budgets and Reliability vs Velocity Disaster Recovery Basics (RPO/RTO) Game Days and Restore Drills Chaos Engineering Basics
Security for Operators
Principle of Least Privilege Secrets Management (ops view) Certificate Rotation and Expiry Prevention Container Hardening (seccomp, caps, RO fs) Kubernetes SecurityContext Patterns Kubernetes RBAC Deep Dive Network Segmentation and Policy Vulnerability Scanning and Patch Loops Runtime Security Signals (what to monitor) Threat Modeling for Operators
Supply Chain Security
SBOM Basics (what and why) Provenance and Attestation Dependency Scanning and Policies Image Signing and Verification Artifact Immutability and Promotion Reproducible Builds CI Hardening and Runner Security Registry Security and Access Controls Tamper Detection and Incident Response Zero Trust Delivery Concepts
Infrastructure as Code & Environments
Infrastructure as Code Principles Declarative vs Imperative IaC State Management (remote state, locking) Drift Detection and Repair Secrets in IaC (avoid leaks) Review Workflows and Change Control Environment Parity and Config Strategy Multi-Environment Promotion Model Testing IaC (plan checks, policy tests) Modules, Standards, and Reuse
Platform Engineering & Internal Developer Platforms
Platform as Product (mental model) Golden Paths and Paved Roads Self-Service Infrastructure Workflows Internal Developer Platform Architecture Service Templates and Scaffolding Platform SLIs, SLOs, and Reliability Multi-Tenancy and Isolation Models Cost Visibility and Chargeback Models Platform Governance and Guardrails Platform Maturity Model
Advanced: Scaling, Multi-Cluster, and Governance
Multi-Cluster Topologies Federation and Regional Failover Control Plane Hardening Large Cluster Scaling Limits Cross-Region Networking Patterns Advanced Progressive Delivery Patterns Global Traffic Management Org-wide Policy Enforcement (OPA/Gatekeeper) Capacity Modeling at Scale Failure Domain and Blast Radius Design

Disk Latency, IOPS, Throughput, Queue Depth

Troubleshoot storage performance using latency, IOPS, throughput, and queue depth with actionable thresholds.
On this page

Key Storage Metrics

  • Latency: how long ops take (most important for user experience).
  • IOPS: operations per second.
  • Throughput: MB/s.
  • Queue depth: pending ops; rising queues mean saturation.

Collect Evidence

iostat -xz 1 10 2>/dev/null || true
# key fields: r/s w/s, await, svctm (if present), %util

Spot Saturation

  • Sustained %util ~ 100% with rising await indicates saturated device.
  • High await + low util can still happen with remote/network storage issues.

Find Top I/O Processes

pidstat -d 1 10 2>/dev/null || true
iotop -oPa 2>/dev/null || true

Filesystem vs Device

df -hT
lsblk -f
mount | head

Failure Modes

  • Log storm: sudden write amplification fills disk and saturates I/O.
  • Compaction/backup: background jobs consume IOPS; latency spikes.
  • Queue collapse: apps time out, retry, and further increase load.

Mitigations

  • Throttle background jobs.
  • Move hot paths to faster disks; separate logs from data.
  • Reduce sync writes where safe; batch operations.
Disk Full vs Inode Full and Cleanup Strategy →