Core Code
Core Code Modern Dev Handbook
RUST Contents
Rust Foundations for Production
Toolchain and Cargo Basics Project Layout Conventions Dependency Management Basics Configuration and Environment Basics Logging First Steps
Ownership, Borrowing, Lifetimes
Ownership Model Explained Borrowing Rules Lifetimes in Practice Smart Pointers: Box, Rc, Arc Interior Mutability
Types, Traits, and APIs
Traits and Generics Newtype Pattern API Design with Result and Option Serde Derive Basics Module Visibility Patterns
Error Handling & Reliability
Error Types with anyhow and thiserror Error Context and Chaining Retries and Timeouts Basics Idempotency Basics Input Validation Basics
Testing & Quality Gates
Unit and Integration Tests Mocking Strategies Coverage and Quality Gates CI Basics Contract Tests Basics
Concurrency & Shared State
Threads and Channels Mutex and RwLock Arc and Shared Ownership Deadlocks and Debugging Send and Sync Traits
Async Rust in Production
Async Runtime with Tokio Async Cancellation Async Timeouts Async Backpressure Basics Async Common Pitfalls
Web Services Basics
HTTP Server with Axum Routing and Middleware Basics JSON Errors and Problem Details Request IDs and Correlation Graceful Shutdown
Database & Migrations Basics
Choosing SQLx or Diesel Connection Pooling Migrations Basics Transactions Basics Query Timeouts
Observability Basics
Tracing Basics Metrics Basics Health Checks and Readiness Structured Logging Production Debugging Basics
Build, Release, Deployment Intro
Cross Compilation Basics Dockerizing Rust Services Configuration and Secrets Basics Release Profiles Rollback Strategies Basics
Security & Hardening Basics
Input Sanitization Deserialization Hardening SSRF and Egress Basics TLS Basics Dependency Vulnerability Scanning

Production Debugging Basics

Debug production issues with a disciplined workflow: start from symptoms, form hypotheses, verify with metrics and traces, and act with minimal risk. Combine request IDs, structured logs, and bounded timeouts to reduce mean time to recovery.
On this page

Production debugging is a workflow, not a talent

In production, you rarely have perfect information. The goal is not to find the most complex root cause first. The goal is to restore service safely and learn. A good debugging workflow is repeatable, minimizes risk, and improves with every incident.

The production loop

Use a simple loop that keeps you focused:

  1. Observe symptoms: error rate, latency, saturation, customer reports.
  2. Form a small hypothesis set: pick 2 to 3 likely causes, not 20.
  3. Verify quickly: use metrics to confirm direction, tracing and logs for specifics.
  4. Act with minimal risk: rollback, reduce traffic, disable a feature, raise timeouts cautiously.
  5. Confirm recovery: verify metrics and health signals return to normal.

Triangulation: metrics, tracing, logs

Each signal answers a different question:

  • Metrics: what is the overall impact? (error percent, p95 latency, RPS)
  • Tracing: where is time spent in a request? (db span slow, handler slow)
  • Logs: what exactly failed? (error message and context)

Production debugging is often: metrics detect, tracing localizes, logs explain.

Start with the simplest questions

  • Is the service up? Check /healthz.
  • Is it ready to serve? Check /readyz.
  • Is the problem global or isolated to a route? Check per-route error metrics.
  • Is it a latency problem or an error problem? Look at latency histograms and status counts.

Fast checks you can run without deep tools

Even without a full observability stack, you can validate basic behavior quickly.

# Health and readiness
curl -i http://localhost:3000/healthz
curl -i http://localhost:3000/readyz

# Example request with explicit request id for correlation
curl -i -H "x-request-id: debug-001" http://localhost:3000/hello

Common production failure modes for Rust web services

These are frequent and diagnosable with your current baseline:

Failure mode: increased latency

Typical causes at this stage:

  • Database slow queries or lock contention
  • Pool saturation: waiting for connections
  • Downstream slowness causing timeouts

How to verify:

  • Metrics: p95 latency increasing, request duration histogram shifts.
  • Tracing: db spans have long duration.
  • Logs: warnings about timeouts or slow operations.

Failure mode: spike in 500 errors

Typical causes:

  • Unhandled error paths or internal exceptions
  • Database connectivity issues
  • Deployment mismatch: schema not migrated

How to verify:

  • Metrics: error counters spike on specific routes.
  • Logs: error_type internal or db error appears with request_id.
  • Readiness: /readyz returns 503 if dependency checks fail.

Failure mode: rollout instability

Symptoms: errors during deploy, connection resets, flapping readiness.

Typical causes:

  • Readiness not aligned with graceful shutdown
  • Migrations not run before traffic shift
  • Timeouts too low or inconsistent across layers

How to verify:

  • Readiness flips during shutdown and becomes stable.
  • Deploy order: migrate then deploy app.

Use request IDs as your primary handle

When debugging a specific failure, always start with a request id. If a user reports an issue, ask for the x-request-id from the response. If you are investigating internally, generate one when reproducing.

Minimal reproduction discipline

  • Reproduce safely: prefer staging. If production is needed, use low traffic endpoints and minimal load.
  • Control variables: send the same request id and the same payload.
  • Compare healthy vs failing: two traces are often enough to localize the difference.

Actions that are low risk

When you need to restore service quickly, prefer actions that reduce risk:

  • Rollback: safest if the issue is introduced by a recent deploy.
  • Disable a feature: if you have feature flags, turn off the failing path.
  • Reduce concurrency: limit pressure on a struggling dependency.
  • Fail fast: keep timeouts and backpressure to prevent full resource exhaustion.

Actions that are higher risk

  • Changing database schema manually without a migration
  • Increasing timeouts dramatically without understanding downstream behavior
  • Restarting everything at once

These can make an incident worse. Use them only with a clear hypothesis and verification plan.

Post-incident: capture learning while it is fresh

The fastest way to improve reliability is a short postmortem checklist:

  • What was the trigger?
  • What signals detected it first?
  • What was the fastest safe mitigation?
  • What single change would prevent recurrence?

Even a brief write-up improves the system and the team.

What comes next

With tracing, metrics, health checks, and structured logging in place, you have a minimal observability baseline. The next step is usually deployment fundamentals and environment configuration, so the service behaves predictably across dev, staging, and production.