Delivery Semantics: At-Least-Once

At-Least-Once Delivery: Durable but Duplicate-Prone

At-least-once delivery guarantees that every message will be delivered one or more times. In other words, the system prioritizes durability over uniqueness. Messages are never silently dropped, but duplicates are possible and expected under failure.

This is the most common delivery model in production messaging systems because it balances reliability and performance. However, it shifts responsibility for correctness to the consumer.

What At-Least-Once Actually Guarantees

• A message will not be lost if the system is configured correctly.
• A message may be delivered more than once.
• Consumers must tolerate duplicates safely.

Duplicates occur primarily when failures happen between processing and acknowledgment.

How Duplicates Happen

Typical flow:

receive(message)
process(message)
ack(message)

If the consumer crashes after processing but before acknowledgment, the broker will redeliver the message. The operation will execute again.

This behavior is correct under at-least-once semantics.

Production Scenario: Double Payment Incident

Symptom

Customers report being charged twice after a brief service restart.

Root Cause

Payment processing was not idempotent. Consumer crashed after charging card but before acknowledging message. Broker redelivered the message after restart.

Diagnosis

• Redelivery count spikes in broker metrics.
• Consumer restart logs align with duplicate events.
• No idempotency key enforcement in payment service.

Resolution

• Introduce idempotency keys tied to business operation.
• Persist processed message IDs.
• Make side-effect operations conditional on prior completion.

At-least-once is safe only if consumers are idempotent.

Designing Idempotent Consumers

Idempotency ensures that repeated processing produces the same result as a single execution.

Common Strategies

• Store processed message IDs in durable storage.
• Use unique business operation identifiers (idempotency keys).
• Perform conditional updates (compare-and-set).
• Make writes commutative when possible.

Idempotency is a data-layer concern, not just an application-level check.

Exactly-Once vs Effectively-Once

Many systems claim “exactly-once” delivery. In reality, most achieve effectively-once processing by combining:

• At-least-once delivery
• Idempotent consumers
• Transactional offsets or commit logs

True exactly-once delivery across distributed boundaries is extremely complex.

Interaction with Transactions

Consumers that update databases must coordinate message acknowledgment with state changes.

Safe pattern:

1. Start database transaction.
2. Check idempotency record.
3. Apply business logic if not processed.
4. Persist idempotency record.
5. Commit transaction.
6. Acknowledge message.

Acknowledging before commit risks lost updates. Committing without idempotency risks duplication.

Observability Signals

• Redelivery count per topic
• Consumer restart frequency
• Dead-letter queue rate
• Duplicate detection metrics
• Processing latency distribution

Redelivery spikes should trigger investigation.

Backpressure and Retry Amplification

Under heavy load, slow consumers cause message accumulation. Crashes increase redelivery. Retries amplify load further.

Without careful flow control, at-least-once systems can enter retry storms.

Failure Injection Test

# At-least-once duplication test
1) Produce message with unique ID
2) Process message but crash consumer before ack
3) Restart consumer
4) Confirm message is redelivered
5) Verify idempotent logic prevents duplicate side effects

Operational Checklist

• Are consumers idempotent by design?
• Are idempotency records durable and transactional?
• Is redelivery rate monitored?
• Is duplicate processing measurable?
• Are retry policies bounded and rate-limited?

When to Use At-Least-Once

• Financial transactions (with idempotency)
• Order processing
• State mutation events
• Any workflow where loss is unacceptable

At-least-once is the default safe choice for critical business workflows — but only if idempotency is enforced strictly.

Key Takeaways

• At-least-once guarantees no message loss but allows duplicates.
• Duplicates are expected, not exceptional.
• Idempotent consumer design is mandatory.
• Transactional acknowledgment prevents race conditions.
• Monitoring redelivery is critical for production safety.

At-least-once delivery shifts complexity from the broker to the application. If the application is not designed for duplication, reliability guarantees become a source of data corruption instead of protection.