On January 10, 2011, GitHub went down for 18 minutes because a network partition split their MySQL cluster. The leader and the followers could no longer communicate. The system could not determine which node held the most recent copy of the data. Two replicas each believed they were the authoritative source of truth. When the partition healed, the cluster had two conflicting histories, and resolving them required manual intervention.
That incident illustrates the central difficulty of database replication. Replication — keeping multiple copies of data on multiple machines — solves one problem completely: if one machine fails, the data is not lost. But it introduces a new problem that is harder: all the copies must agree.
The marketing for read replicas makes them sound like free scaling. Spin up a replica, point your reads at it, scale your read capacity. The reality is more constrained. Read replicas are an availability and capacity mechanism, not a transparent extension of the primary database. Treating them as such produces bugs that are hard to find and easy to ship.
The Three Replication Architectures
Replication maintains copies of data on multiple nodes. The copies serve two purposes: fault tolerance (if one node fails, the data survives on another) and read scaling (reads distributed across replicas rather than all hitting one node).
Three architectures govern how writes are handled:
| Architecture | Write path | Read path | Conflict risk | Use case |
|---|---|---|---|---|
| Leader-follower | All writes → leader; replicated to followers | Reads from followers (potentially stale) or leader | None (one writer) | Most web applications (PostgreSQL, MySQL) |
| Multi-leader | Writes accepted by any leader; leaders sync | Reads local to leader | Yes (concurrent writes to different leaders) | Multi-datacentre active-active |
| Leaderless (Dynamo) | Writes to any W of N nodes; reads from any R | Reads from any R nodes | Yes (concurrent writes to different nodes) | High availability, eventual consistency |
Most production systems run leader-follower. The other two exist for specific use cases where the leader-follower constraints do not fit.
How Leader-Follower Actually Works
The leader maintains a write-ahead log. Every change is written to the log before being applied to data files. Each entry has a monotonically increasing log sequence number (LSN). Followers receive the log stream, replay each entry in LSN order, and advance their last_applied_lsn.
Replication lag is the difference between the leader's current LSN and the follower's last-applied LSN. For healthy followers on fast networks, lag is usually under one second. Under sustained write load, network degradation, or follower CPU saturation, lag can grow to minutes or hours.
This is the part teams forget: the lag is not bounded. A follower that is one second behind the leader is normal. A follower that is one hour behind the leader is also possible. Nothing automatically catches this — it must be monitored.
The Read-Your-Own-Write Problem
The classic bug. A user updates their profile and is immediately redirected to the profile page. The write goes to the leader. The read is load-balanced to a follower with 500ms of lag. The user sees the old data they just edited.
function update_profile(user_id, new_bio):
leader.write("user:" + user_id + ":bio", new_bio)
// Committed at leader LSN = 10042
function get_profile(user_id):
// Load balancer routes to follower_2 with last_applied_lsn = 10038
return follower_2.read("user:" + user_id + ":bio")
// Returns the OLD bio
The fix is sticky reads after writes: track the LSN of the write, route subsequent reads to a follower that has caught up to that LSN, or fall back to the leader.
function get_profile_after_write(user_id, write_lsn):
for follower in followers:
if follower.last_applied_lsn >= write_lsn:
return follower.read("user:" + user_id + ":bio")
return leader.read("user:" + user_id + ":bio")
Most ORMs do not do this. Most teams discover the bug in production.
The Bigger Read-Lag Problem
Read-your-own-write is the visible bug. The hidden one is worse: any read from a lagging follower returns data that reflects the database at some point in the past, not the present.
An order management system reading shipping status from a lagging replica. An inventory system reading stock levels from a lagging replica. An access control system reading permission changes from a lagging replica. All produce incorrect business outcomes from technically successful queries.
The customer-support example is the most painful. A customer support agent in Europe checks a user's order status. The order was placed 30 seconds ago. The EU replica has 60 seconds of lag. The agent sees "no order found" — and tells the customer their payment never went through.
AT1 in Action: Sync vs Async Replication
The central tradeoff at the database layer is between asynchronous replication (high availability for reads, possible stale reads) and synchronous replication (no stale reads, write latency dependent on follower network latency).
Asynchronous replication lets the leader acknowledge the write immediately, before any follower has received the log entry. If the leader fails after acknowledging but before replicating, the write is lost. The lost-write window is small (milliseconds) but non-zero.
Synchronous replication requires at least one follower to confirm receipt before the leader acknowledges. No write is lost on leader failure (it has been replicated). But write latency now includes the round-trip to the follower. If the follower is in a different datacentre, that round trip is 50–100ms, which is often more than the entire write budget for the operation.
PostgreSQL's synchronous_commit setting controls this per transaction. Most teams set it once and forget it. The right choice depends on what you are willing to lose: a few milliseconds of writes on rare leader failures, or a few hundred milliseconds of latency on every write.
Multi-Leader and the Split-Brain Failure Mode
When you need writes accepted in multiple regions with low latency, multi-leader replication is the architecture. Each region has its own leader; leaders replicate to each other asynchronously.
The new failure mode is concurrent conflicting writes. The US leader gets key="config" value="A" at t=100. The EU leader gets key="config" value="B" at t=101. After sync, both leaders have both writes. Which wins?
Last-write-wins (LWW) based on timestamp is the simplest rule and the most dangerous. Wall clocks drift. The leader with the slightly faster clock wins every race regardless of which write was actually later. The result is silent data corruption — writes that were acknowledged successfully but discarded during reconciliation.
CRDTs (Conflict-Free Replicated Data Types) provide automatic merge semantics for specific data structures (counters, sets, sequences). Application-defined merge logic works when the application understands the semantics of the data. Asking the user to resolve conflicts is the last resort, but for shared documents (Google Docs, Notion) it is the right answer.
FM12 (Split-Brain) is the failure mode: when the network connection between two leaders is lost, both leaders continue accepting writes in their respective regions. Each believes it is authoritative. When the partition heals, both sides have writes the other has not seen. The corruption is discovered later, during reconciliation, with no automatic resolution.
Leaderless (Dynamo-Style): Quorums
Leaderless replication removes the leader concept. Every key has N replicas. Writes succeed when W replicas confirm. Reads query R replicas and reconcile.
The constraint for strong consistency is R + W > N. With N=3, W=2, R=2: any read quorum of two overlaps with any write quorum of two by at least one node. That node has the latest version. Vector clocks identify the latest version when responses disagree. Read repair updates stale replicas in the background.
The cost is that every operation requires contacting multiple nodes. Read latency is the slowest of R responses, not the median. Write latency is the slowest of W. The benefit is no single leader to fail and no failover orchestration to get wrong.
Three Real Systems
PostgreSQL — leader-follower with write-ahead-log streaming. Per-transaction control over synchronous vs asynchronous via synchronous_commit. The default in most teams' deployments.
CockroachDB — multi-active using Raft consensus per range of data. Every key range has a Raft group of three replicas. Writes confirmed by majority. Linearisable consistency with no leader-follower lag. The cost is write latency: every write needs a Raft round trip (1–5ms same-region, 50–100ms cross-region).
DynamoDB — leaderless with eventual consistency by default and optional strong consistency (quorum read). Default eventually-consistent reads are fastest. Strongly-consistent reads double the latency and consume more read capacity. Last-write-wins on timestamp for conflicts — a deliberate simplification that accepts lost writes under clock skew.
The Question Before Adding Replicas
Before you add a read replica, the questions are:
- What is your tolerance for stale reads? (Define it numerically. "A few seconds" is not a specification.)
- Will read-your-own-write be a problem? (For most user-facing apps, yes — plan the sticky-read mechanism up front.)
- Who monitors replication lag, and what is the alert threshold? (Without an alert, the lag will grow silently until the day it breaks something.)
- What is the failover procedure when the leader dies? (Untested failover is theatre.)
- Does the system actually need read scaling, or are you adding replicas because the dashboards make it look easy? (Vertical scaling of the primary is often cheaper.)
Read replicas are not free. They are a tradeoff with consistency, and a discipline with operations.
This article extracts the core of Book 3, Chapter 8 — Database Replication. The chapter includes the full pseudocode for all three architectures (leader-follower with LSN-based replication, multi-leader with conflict resolution, leaderless with quorum reads), the read-after-write sticky-read implementation, the multi-leader conflict resolution patterns (LWW, CRDT, application merge), and the design problem for a global financial ledger service.