In 2007, Amazon published the Dynamo paper. The company had built a shopping cart service and kept running into the same problem: a relational database, no matter how well tuned, eventually became the bottleneck. Worse, the database’s strong consistency guarantee meant that a network partition — a routine event at Amazon’s scale — could take the entire cart service down. The paper described a different trade: give up strong consistency, accept eventual consistency, and in return get a system that stays available even when nodes fail and network partitions occur. Dynamo became the blueprint for a generation of distributed key-value stores.
The distributed key-value store is the foundational read/write primitive of large-scale systems. Understanding how it works — and what it costs — is prerequisite knowledge for every other system in this book.
Store billions of key-value pairs across hundreds of servers. Support reads and writes at hundreds of thousands of operations per second. Tolerate individual server failures without downtime. Distribute data automatically without manual shard assignment. Scale horizontally by adding nodes.
The constraints are fundamental: no single server can hold all the data, no single server can handle all the traffic, and any server can fail at any time.