Lady Gaga has 80 million Twitter followers. When she tweets, 80 million feeds must be updated. At the time Twitter was scaling to handle celebrity posts, their architecture made a simple assumption: when a user posts, write the post to a feed table for each follower. For a typical user with 200 followers, that is 200 writes. For Lady Gaga, that is 80 million writes. In 2012, this triggered cascading outages that Twitter called the "Fail Whale" era. The problem was not the tweet. It was the fan-out.
The news feed is deceptively simple on the surface — show the user their friends' recent posts. Behind it is one of the most architecturally nuanced problems in distributed systems: how do you deliver content from N writers to M readers, when some writers have millions of readers, in under 200ms?
This is the system design question that interview prep books skip. The two textbook approaches both fail at production scale. The interesting work is in the hybrid.
The Two Approaches That Both Fail
Fan-out-on-write (push). At post time, push the post to every follower's feed. Feeds are stored as sorted sets in Redis, keyed by follower user ID, scored by post timestamp. Feed reads are O(1): retrieve top-K items from the sorted set.
For a typical user with 500 followers, 500 writes per post is manageable. For a celebrity with 50M followers, this is 50M writes for a single post. The fan-out is async (the poster's request returns immediately) but for celebrities it never catches up. 80M followers × 1ms per write = 22 hours to complete a single post's fan-out. The system falls further behind every time the celebrity posts again.
Fan-out-on-read (pull). At feed load time, retrieve all accounts the user follows and query their recent posts in parallel, then merge. One write per post (perfect write scaling). But for a user following 5,000 accounts, that is 5,000 parallel reads per feed load. The merge step is computationally heavy. Feed load latency is seconds, not milliseconds.
Neither extreme works. The production solution is a hybrid.
The Hybrid Model: Classify, Then Pick
Classify accounts by follower count:
- Regular users (< 1M followers): use fan-out-on-write. Their posts are pushed to follower feeds at write time. Write cost is bounded by the typical follower count.
- Celebrity users (> 1M followers): use fan-out-on-read. Their posts are not pushed. When a follower loads their feed, the system fetches recent celebrity posts on-demand and merges them with the pre-pushed feed.
Feed load for a regular user:
1. Load pre-pushed feed from Redis (posts from regular users they follow)
2. For each celebrity they follow (e.g., 3 celebrities):
fetch celebrity's last 50 posts from celebrity post cache
3. Merge pre-pushed feed + celebrity posts, rank by timestamp/relevance
4. Return top 50
The celebrity post cache is populated when the celebrity posts (write-through) and replicated aggressively — every celebrity post is cached because millions of fans will read it.
This works because:
- Regular users' write fan-out is bounded by typical follower counts (hundreds, thousands).
- Celebrity post reads happen at most once per feed load per celebrity per user — bounded by how many celebrities a single user follows (almost always under 50).
- The expensive operation (writing to 80M followers) never happens. The expensive operation is replaced by 80M readers each doing one extra cache read.
Chronological vs Ranked Feed
Chronological feed is simple: sort by post timestamp. Ranked feed (Facebook/Instagram model) applies a ranking model: factors include post recency, engagement rate (likes, comments within minutes of posting), relationship strength (how frequently you interact with the poster), content type preferences.
Ranked feeds require running the ranking model over candidate posts — typically a subset (top 500 candidates by recency, then re-ranked). This adds ~50ms latency for model scoring but significantly improves relevance.
The architectural change to support ranking is the introduction of a candidate generation stage (gather posts) separate from the scoring stage (rank them). Most of the operational cost shifts from feed assembly to feature retrieval (each candidate needs its engagement metrics, the user-poster interaction history, etc.).
The Key Tradeoffs
AT4 (Precomputation vs On-Demand). Fan-out-on-write is precomputation — the feed is ready before the user requests it. Fan-out-on-read is on-demand — the feed is assembled at request time. The hybrid model applies precomputation where the cost is bounded (regular users) and on-demand where precomputation would overwhelm the system (celebrities). The exact threshold is a tuning parameter set by measurement, not theory.
AT2 (Latency vs Throughput). Pre-pushed feeds in Redis are O(1) reads. On-demand celebrity fan-out adds N parallel reads for N followed celebrities. Keeping the celebrity follow list short (or limiting the merge to a bounded number of sources) controls read latency.
AT10 (Synchronous vs Asynchronous). Fan-out must be async. Completing 80M writes synchronously before returning the post response would make posting take hours. The poster's request completes immediately; feed updates propagate asynchronously.
Where the Hybrid Fails
FM7 (Thundering Herd). A celebrity posts for the first time after a period of inactivity. Millions of fans load their feeds simultaneously. Each fan's feed load hits the celebrity post cache. If the cache is cold, all requests miss simultaneously and hit the backing database — a thundering herd. Warming the celebrity post cache aggressively before the post propagates to fan feeds prevents this. The cache warm-up must complete before the notification fan-out.
FM4 (Data Consistency Failure). Feed staleness. A post was deleted after it appeared in a follower's feed. The follower sees a deleted post until their pre-cached feed TTL expires. For most content, this is acceptable; for violating content (hate speech, illegal material), a fast invalidation path — pushing a delete notification to all cached feeds — is required, not optional.
FM3 (Unbounded Resource Consumption). The fan-out queue accumulates posts faster than workers can process them during peak posting hours. Queue depth must be monitored; if it grows unboundedly, fan-out workers must scale out. A queue that never drains is FM3 in the fan-out path — the social-platform equivalent of consumer lag.
How It Evolves at Scale
At 10× current load: the fan-out worker pool scales horizontally. The celebrity threshold lowers — accounts with 100K followers rather than 1M may switch to fan-out-on-read if write volume is too high. Feed storage moves from per-user sorted sets to a more compressed representation to reduce Redis memory.
At 100×: geographic fan-out. Users in Europe should not receive their feeds from a fan-out worker running in the US. Regional fan-out workers process regional follower feeds locally, reducing cross-region traffic. The celebrity post cache replicates to every region.
Real Systems
Twitter uses the hybrid model described here. Regular accounts use fan-out-on-write into a Redis-backed feed cache. Highly-followed accounts (the "celebrities") use fan-out-on-read. Twitter has published detailed blog posts about the "timeline" architecture and the "celebrity problem" — the term is theirs.
Instagram builds chronological feeds from a pre-computed "inbox" per user. The ranking model runs as a re-ranking step over inbox candidates. Instagram's architecture emphasises the ranking model more than Twitter's, reflecting Facebook's investment in ML-based relevance.
Facebook has evolved through multiple feed architectures. The current model uses a ranking pipeline that retrieves candidates from multiple sources (posts, stories, recommended content) and applies a deep neural network ranking model. Feed generation is almost entirely on-demand with aggressive caching of intermediate results.
The Interview Question That Reveals Real Understanding
In a system design interview, the standard answer is "fan-out on write, store feeds in Redis." The follow-up question — "what happens when Lady Gaga tweets?" — is the test.
The candidates who say "we shard the writes across more workers" have not understood the problem. Sharding does not reduce 80M writes to a manageable number. It just spreads them across more machines, which costs more money and still falls behind.
The candidates who say "we switch celebrities to pull, regular users stay on push, and the feed merges at read time" have understood the structure of the problem. The celebrity problem is not a scaling problem. It is a fundamental asymmetry between write-fan-out and read-fan-out costs at extreme follower counts. The architectural answer is to make the cheap operation cheap and accept that the other is expensive only when it must be.
This article extracts the core of Book 4, Chapter 11 — News Feed System. The chapter includes the full Redis sorted-set feed structure, the celebrity post cache write-through pattern, the ranking pipeline (candidate generation + scoring), and the worked design problem for a Breaking News feature with a 10-second delivery SLA against a 100K writes/sec fan-out budget for a 3M-follower journalist.