Paper · Distributed Systems

Dynamo: Amazon's Highly Available Key-value Store

G. DeCandia et al. · SOSP · 2007

Consistent hashing, quorums, and choosing availability over consistency.

Summary

For services such as Amazon's shopping cart, being unable to accept a write costs money directly. Dynamo was designed to stay writable even during server failures and network partitions, accepting weaker consistency in exchange.

Data is partitioned and replicated with consistent hashing using virtual nodes. Reads and writes use configurable quorums, so each service can tune the balance between latency, durability and consistency. Conflicting versions are tracked with vector clocks and reconciled by the application. Sloppy quorums with hinted handoff, anti-entropy using Merkle trees and gossip-based membership handle failures.

Dynamo shaped a generation of distributed databases, including Cassandra, Riak and Voldemort. It is a practical example of deliberately choosing availability over consistency, and of the machinery that choice requires.

Key ideas

  • Consistent hashing with virtual nodes for partitioning and replication.
  • Tunable quorums for reads and writes.
  • Vector clocks to track versions, with reconciliation by the application.
  • Sloppy quorums, hinted handoff and Merkle-tree anti-entropy for failures.
  • Gossip-based membership and failure detection.

More Distributed Systems papers