Your consistent hashing is working. Your cluster has twenty nodes. One key is getting forty percent of your traffic. One node is at ninety percent CPU. Nineteen nodes sit idle.
You have capacity. You cannot use it. This is FM6 — Hotspotting.
The incident
A distributed cache cluster runs twenty nodes behind a consistent hash ring. Per-node throughput dashboards show a clean picture on most days. Requests distribute. Load spreads. The system holds its shape.
Then a single product goes viral. Every request routes to the same key. That key hashes to one position on the ring. The first node clockwise from that position owns it.
That one node absorbs the entire spike. It hits its connection limit. Latency climbs. Requests time out. The other nineteen nodes report twenty percent load and healthy P99.
The dashboard says the cluster is fine. The users say the product is broken. Both are correct.
What FM6 actually is
FM6 — Hotspotting is the failure mode where one node in a distributed system receives disproportionately more traffic than others. That node degrades under load. The other nodes sit idle. The system appears to have capacity but cannot use it.
This is the specific failure of AT5 — Centralization vs Distribution. You chose distribution. You partitioned the work across nodes. The distribution mechanism failed to distribute.
Distribution on paper is not distribution in traffic. A ring with twenty positions is not a system that spreads twenty ways. A partition scheme is only as even as the keys flowing through it.
How the hot key forms
The source chapter names four mechanisms. Each one produces the same outcome from a different starting point.
Poor key distribution in hash-based partitioning. The hash function spreads keys evenly across the ring. It cannot spread traffic evenly if traffic itself is not evenly spread across keys. A viral post, a trending product, a celebrity account — the hash does its job. One node still burns.
A hash ring with too few nodes and no virtual nodes. With ten nodes, each owns ten percent of the ring by position. Small imbalances in ring geometry become large imbalances in load. Add or remove a node and the neighbours inherit whole arcs of traffic.
A partition key that concentrates writes. All writes for a popular user go to the same shard. Every order for one enterprise customer routes to one partition. The partition key looked reasonable in design. In production it acts like a funnel.
A load balancer using IP hash routing where one IP has many clients behind it. One corporate NAT. One mobile carrier gateway. Ten thousand users behind one address. The load balancer sees one client and treats it accordingly.
Each mechanism produces FM6. The remedy differs by mechanism. The failure signature is identical.
How to detect it before it takes you down
The source names two detection paths. Both are cheap. Both are usually absent.
Per-node throughput metrics. If one node is at eighty percent load and others are at twenty percent, there is a hotspot. This is not a subtle signal. It is visible on any dashboard that plots per-node CPU or QPS side by side.
Most engineers plot cluster-wide averages instead. A cluster averaging thirty percent CPU hides one node at ninety and nineteen at fifteen. The average tells you nothing about the failure mode you are about to hit. Plot per-node. Always.
Check partition key cardinality. A low-cardinality key produces as many shards as there are distinct values. Sharding by country code gives you two hundred shards. Sharding by user ID gives you millions. If the cardinality of your partition key is small, the ceiling on your distribution is small.
Ask one question of every partition scheme. If the highest-volume single entity doubled its traffic tomorrow, which node absorbs it? If the answer is one node, you have a hotspot waiting.
What FM6 does to the rest of the system
FM6 does not stay contained. The hot node runs out of connections. It runs out of threads. It runs out of memory. This is FM3 — Unbounded Resource Consumption on one node while the cluster has capacity to spare.
Then the hot node stops responding. If it was the only node handling a particular partition, its failure removes that partition from service. This is FM1 — Single Point of Failure emerging inside a system that was designed to have none.
The architecture diagram shows redundancy. The traffic pattern reveals a de facto SPOF. Twenty nodes on the diagram. One node on the critical path. When that node dies, the partition dies with it.
Distribution that concentrates load on one node is not distribution. It is centralization with extra hops.
The mitigations, by mechanism
The source names three prevention techniques. Match each to the mechanism that produced the hotspot.
Virtual nodes in consistent hashing. Each physical node claims many positions on the ring instead of one. Traffic distributes across the virtual positions. A hot key still lands on one virtual node, but the virtual nodes owned by any physical node come from many arcs of the ring. Load evens out.
Virtual nodes fix the geometry problem. They do not fix the popular-key problem.
High-cardinality partition keys. User ID instead of country code. Order ID instead of merchant ID. The cardinality of the key sets the ceiling on how many ways the work can split. Choose partition keys with cardinality far higher than your node count.
High-cardinality keys fix the concentration problem. They do not fix the celebrity problem.
Adding randomness to the partition key for high-volume single entities. Celebrity accounts get a random suffix. Writes for one account fan out across many shards. Reads must gather from all of them, or from a designated aggregation path.
This mitigation is targeted. It costs read complexity. You pay it only for the small number of entities that would otherwise burn one node down.
Each mitigation addresses one mechanism. Applying the wrong mitigation leaves the hotspot in place.
The tradeoff that produced the hotspot
Every FM6 incident traces back to AT5 — Centralization vs Distribution. You chose distribution. You did not verify that distribution held under the actual traffic distribution of your users.
Distribution assumes traffic spreads. Traffic does not spread. Traffic clusters. On popular users. On trending items. On the front page. On the one endpoint that everyone hits at nine in the morning.
Consistent hashing distributes keys. It does not distribute requests. If ten percent of your keys carry ninety percent of your requests, no hash function saves you.
The tradeoff is not resolved by picking distribution. It is resolved by measuring where traffic actually goes and adjusting the distribution mechanism to match.
The signal that tells you this applies to your system
Per-node CPU or QPS graphs that spread wide instead of stacking tight. One node consistently at the top of the range. Others consistently at the bottom. A small number of partition keys accounting for a large fraction of writes or reads.
If your dashboard only plots cluster averages, you cannot see this signal. The average lies about the shape of the load. Split by node. Split by shard. Split by partition key. The hotspot is visible the moment you stop averaging over it.
The harder question
You added virtual nodes. You picked a high-cardinality partition key. You added random suffixes to your celebrity accounts. Your per-node load graphs stack tight.
A new product launches. Every user opens it at the same second. Every request hits the same three keys for the first thirty seconds of that product's life.
Your mitigations were designed for steady-state distribution. Your traffic just became a coordinated event. Which mitigation still holds? Which one just became a new hotspot?
The full framework treatment — compression blocks, three-level exercises, and the complete AT/FM mapping — is in the Reference Book, Chapter 7 (Failure Modes). Free chapter available at computingseries.com/books/ref.