graceful shutdown, readiness/liveness

Why Abrupt Shutdown Causes Data Loss

In containerized production environments, pods are restarted frequently: - Rolling deploy - Auto-scaling - Crash recovery - Node eviction If your application does not shut down gracefully: - In-flight requests are dropped - Transactions are interrupted - Clients receive connection reset - Partial writes may occur Graceful shutdown is required for correctness.

Incident Scenario: Rolling Deploy Caused 502 Storm

Deployment triggered rolling restart. Old pod terminated immediately. Load balancer still routed traffic. Requests were cut mid-flight. Clients retried aggressively. Error rate spiked. Root cause: No graceful shutdown configuration.

Spring Boot Graceful Shutdown

Spring Boot supports graceful shutdown. Configuration:

server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=30s

Behavior: - Stop accepting new requests - Wait for in-flight requests to complete - Close context cleanly - Release resources Without this, termination is abrupt.

Readiness vs Liveness

Liveness probe: Is the application alive? Readiness probe: Is the application ready to receive traffic? On shutdown: - Liveness should remain true - Readiness should switch to false This signals load balancer to stop routing new traffic.

Correct Shutdown Flow

1. Pod receives SIGTERM 2. Application marks readiness false 3. Load balancer drains connections 4. In-flight requests finish 5. Context closes 6. Pod terminates If readiness is not implemented correctly, traffic continues during shutdown.

Handling Long-Running Tasks

If background jobs exist: - Ensure they respond to interruption - Use executor shutdown hooks - Avoid blocking indefinitely Example executor shutdown:

import jakarta.annotation.PreDestroy;
import org.springframework.stereotype.Component;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

@Component
public class ExecutorManager {

    private final ExecutorService executor = Executors.newFixedThreadPool(10);

    @PreDestroy
    public void shutdown() throws InterruptedException {
        executor.shutdown();
        executor.awaitTermination(30, TimeUnit.SECONDS);
    }
}

Ignoring background thread shutdown leads to abrupt termination.

Database and Connection Cleanup

Ensure: - Connection pools close cleanly - No new transactions start during shutdown - Long transactions are avoided Keep transactions short to prevent shutdown blocking.

Common Anti-Patterns

- Blocking shutdown hook forever - Ignoring InterruptedException - Not differentiating readiness and liveness - Relying on default termination without testing Test shutdown behavior locally with SIGTERM.

Testing Graceful Shutdown

Simulate: - Ongoing HTTP requests - In-flight DB transaction - Long downstream call Then send SIGTERM and observe: - No request loss - No connection reset - No partial state Shutdown behavior must be verified before production.

Checklist

- Enable server.shutdown=graceful - Configure shutdown timeout properly - Implement correct readiness and liveness probes - Ensure background tasks handle interruption - Keep transactions short - Test SIGTERM handling before production - Verify load balancer drains connections properly Graceful shutdown prevents silent request loss. In production, correctness during deploy is as important as correctness during runtime.