Outbox Pattern (Basics)

Production incident

Order is created in the database and then a message is published to the broker. A transient network failure happens after DB commit but before publish. The order exists but downstream never sees the event. Customers see paid orders that never ship. You built a distributed transaction by accident and it failed as expected.

Symptom

• Downstream systems miss events for records that exist in the DB.
• Manual replays create duplicates because you do not have idempotency.
• Incidents are "data mismatch" between services.

Cause

• DB commit and message publish are not atomic.
• Retries publish duplicates or skip publishes depending on failure timing.
• No durable event store with retryable delivery.

Diagnosis

// Look for patterns: SaveChanges then broker publish in the same request
grep -R "SaveChangesAsync" -n .
grep -R "Publish" -n . | head

// Incident signal: orders exist but no downstream processing record

Anti-pattern

await db.SaveChangesAsync();
await broker.PublishAsync(new OrderCreated { OrderId = order.Id });
// If this fails after commit, you are inconsistent

Correct pattern

Write the event to an Outbox table in the same transaction, then deliver asynchronously with retries and idempotency.

// Outbox entity
public class OutboxMessage
{
    public Guid Id { get; set; }
    public DateTime CreatedAtUtc { get; set; }
    public string Type { get; set; } = "";
    public string PayloadJson { get; set; } = "";
    public DateTime? ProcessedAtUtc { get; set; }
    public int AttemptCount { get; set; }
    public string? LastError { get; set; }
}

// In the same transaction as your business write
await using var tx = await db.Database.BeginTransactionAsync();

db.Orders.Add(order);

db.Outbox.Add(new OutboxMessage
{
    Id = Guid.NewGuid(),
    CreatedAtUtc = DateTime.UtcNow,
    Type = "OrderCreated",
    PayloadJson = Serialize(new { order.Id, order.CreatedAtUtc })
});

await db.SaveChangesAsync();
await tx.CommitAsync();

// Separate background worker delivers outbox messages

Delivery worker rules

• Batch fetch unprocessed messages with a limit.
• Mark processed only after broker ack.
• Retry with backoff and a max attempt policy.
• Use idempotency keys on consumer side (message Id).

// Worker sketch
var batch = await db.Outbox
    .Where(m => m.ProcessedAtUtc == null)
    .OrderBy(m => m.CreatedAtUtc)
    .Take(100)
    .ToListAsync();

foreach (var msg in batch)
{
    try
    {
        await broker.PublishAsync(msg.Type, msg.PayloadJson, msg.Id);
        msg.ProcessedAtUtc = DateTime.UtcNow;
        msg.LastError = null;
    }
    catch (Exception ex)
    {
        msg.AttemptCount++;
        msg.LastError = ex.Message;
        // Backoff policy handled by scheduling or next run
    }
}

await db.SaveChangesAsync();

Security and performance impact

• Performance: async delivery decouples request latency from broker latency.
• Security: outbox payloads must avoid sensitive data. Treat payload as potentially logged and replicated.

Operational notes

• Monitoring: outbox backlog size, oldest unprocessed age, publish failure rate, attempts distribution.
• Rollout: start writing outbox first, then enable worker, then switch consumers if needed.
• Rollback: you can stop the worker to halt propagation while preserving durability. You can reprocess later.

Checklist

• Business write and outbox insert are in the same transaction.
• Delivery is async with retries and backoff.
• Consumers are idempotent using message Id.
• Outbox backlog and age are monitored with alerts.
• Payload avoids secrets and PII where possible.