Log Replication Internals (Commit Index, Match Index)

Log Replication Internals: The Safety Engine of Raft

Leader election determines who coordinates the cluster. Log replication determines what the cluster agrees on. In Raft, correctness depends entirely on how logs are appended, matched, committed, and recovered.

Understanding log replication internals is critical for diagnosing data divergence, slow commits, and replication stalls in production environments.

The Replication Pipeline

When a leader receives a client command, the following sequence occurs:

1. Leader appends entry to its local log.
2. Leader sends AppendEntries RPC to followers.
3. Followers validate log consistency.
4. Followers append entry if consistent.
5. Leader tracks acknowledgments.
6. When majority acknowledges, entry is committed.
7. Committed entries are applied to the state machine.

Only committed entries are considered durable cluster decisions.

AppendEntries Consistency Check

Each AppendEntries request contains:

• Leader term
• Previous log index
• Previous log term
• New entries
• Leader commit index

Followers perform a critical check:

if follower.log[prevLogIndex].term != prevLogTerm:
    reject AppendEntries

This ensures log matching: if logs diverge, replication halts and correction begins.

Log Matching Property

Raft guarantees:

• If two logs contain an entry at the same index and term, all preceding entries are identical.

This property prevents conflicting histories once entries are committed.

Handling Divergence

If a follower rejects AppendEntries due to mismatch, the leader decrements the nextIndex for that follower and retries.

This backtracking continues until a matching prefix is found.

Once aligned, new entries overwrite conflicting entries on the follower.

Production Scenario: Diverged Follower After Network Partition

Symptom

After a temporary partition, one node takes significantly longer to catch up.

Root Cause

The isolated node accepted entries from a former leader that were never committed. When the partition healed, those entries conflicted with the majority log.

Diagnosis

• Follower log length differs from leader.
• High AppendEntries rejection count.
• Repeated nextIndex decrement retries.

Resolution

• Allow leader backtracking to find matching prefix.
• Ensure snapshot installation is enabled for large divergence.
• Monitor replication catch-up time.

Commit Index Advancement

The leader advances the commit index when a log entry is stored on a majority of nodes and belongs to the current term.

This second condition prevents older-term entries from being committed incorrectly during leader changes.

Snapshot and Log Compaction

Long-running clusters accumulate large logs. Raft supports snapshotting:

• State machine snapshot created.
• Committed log entries before snapshot index removed.
• New nodes can install snapshot instead of replaying full history.

Snapshot transfer reduces recovery time dramatically for lagging nodes.

Latency and Throughput Implications

Commit latency equals the time required to replicate to majority.

Factors affecting latency:

• Slowest majority node RTT
• Disk fsync latency
• Network jitter
• Batching configuration

High commit latency often traces back to replication bottlenecks.

Common Operational Pitfalls

• Running clusters across high-latency regions without adjusting timeouts.
• Disabling fsync for performance, weakening durability guarantees.
• Ignoring follower lag metrics.
• Misconfiguring snapshot thresholds, causing log bloat.

Observability Signals

• AppendEntries failure rate
• Follower matchIndex per node
• Commit index progression rate
• Snapshot installation frequency
• Replication round-trip time

A healthy cluster shows steady commit index advancement and low rejection rate.

Failure Injection Test

# Log divergence validation
1) Isolate follower from leader
2) Force leader change
3) Restore follower
4) Observe backtracking and alignment
5) Verify no committed entries lost

Key Takeaways

• Log replication enforces cluster-wide agreement.
• AppendEntries validation preserves log matching.
• Majority acknowledgment defines commit safety.
• Divergence recovery relies on prefix matching and backtracking.
• Snapshotting is essential for long-running clusters.

Log replication is the safety backbone of Raft. If leader election determines authority, log replication determines truth. Production stability depends on both working flawlessly under failure and recovery scenarios.