Recommendation Engine

Introduction

Netflix once ran a $1M competition to improve its recommendation algorithm by 10%. Hundreds of teams competed for three years. The winning team improved the algorithm by 10.06% — barely over the threshold. Netflix never fully deployed the winning system. The winning algorithm was an ensemble of 107 models that required too much compute time to run in production. The recommendation system deployed was simpler, less accurate, and fast enough.

That contest outcome captures the central tension of recommendation systems: accuracy and latency are in constant tension. The most accurate models require the most compute. Production systems must serve recommendations in under 100ms. The architecture of a recommendation engine is the story of how to get accuracy close to the theoretical maximum within an engineering budget.

The Problem

Given a user and a context — what they are watching, what they have liked, what similar users have engaged with — return a ranked list of N items predicted to maximise engagement. The system must serve recommendations within 100ms for millions of concurrent users. The underlying models must be trained on petabytes of interaction data. Item embeddings must be updated as new items are added and old items become stale.

Naive Approach and Why It Fails

A naive approach runs the full ranking model on all items for each request. If the catalogue has 10M items and the model takes 1ms per item, a single recommendation request requires 10,000 seconds of compute. This fails immediately.

The second naive approach is pure collaborative filtering at query time: compute cosine similarity between the requesting user and all other users, find the most similar users, return items those users liked. With 100M users, pairwise similarity computation takes hours, not milliseconds.

The production solution is a multi-stage pipeline: candidate generation narrows 10M items to 1,000; scoring ranks those 1,000 items; re-ranking applies business rules. Each stage is progressively more expensive and progressively more accurate.

Architecture Walkthrough

Multi-Stage Pipeline

Candidate Generation

Two approaches dominate:

Collaborative filtering: items similar to what the user has interacted with (item-to-item similarity), computed using embeddings. Pre-compute item embedding vectors offline; at query time, find nearest neighbours using approximate nearest neighbour search (T2 — a tree-indexed vector space).

Content-based filtering: items similar in attributes to what the user prefers. A user who watches action films gets action films. This does not require interaction data — it works for new users (cold start).

Matrix factorisation: decompose the user-item interaction matrix into user embeddings and item embeddings. Each user and item is a dense vector; the dot product of a user vector and an item vector predicts the interaction probability. Pre-compute all user and item embeddings offline.

Feature Store

Candidate scoring requires features: user demographic features, item attributes, interaction history, contextual signals (time of day, device type). Features come from two stores:

Offline feature store: a data warehouse with historical features. Used for model training and for features that do not need to be fresh (user age, item category).

Online feature store: a low-latency key-value store (Redis) with pre-computed features that must be fresh for serving. User’s recent interactions, item’s current popularity, time-since-last-viewed.

Feature store architecture:
  Offline:  Data warehouse (Snowflake, BigQuery) ← batch pipelines
  Online:   Redis cluster ← streaming pipelines (Kafka → Flink → Redis)
  Serving:  Feature retrieval joins online + offline at query time

Scoring Model

The scoring model is typically a neural network or gradient boosted tree that takes features as input and outputs a predicted engagement probability. It runs on every candidate in the shortlist (~1,000 items) at query time. Batching all 1,000 candidates into a single model call amortises the model loading cost.

Re-ranking

Business rules applied after scoring: diversity (avoid 10 consecutive items from the same creator), freshness boost (prefer newer items over equally scored older items), policy filters (age-gated content, sponsored items), serendipity injection (surface a fraction of items outside predicted preferences).

Feedback Loop

User interactions — clicks, views, likes, skips — are fed back into the training pipeline. More interactions → better model → better recommendations → more interactions. This is T11 (Feedback) operating at the system level.

Recommendation → User interaction → Interaction log → Training data
→ Updated model → Better recommendation (loop)

Key Design Decisions

AT4 — Precomputation/On-Demand: Candidate generation is precomputed (item embeddings are pre-built, ANN index is pre-built). Scoring is on-demand (features retrieved and model scored at query time). Re-ranking is on-demand. The boundary between precomputed and on-demand sits between candidate generation and scoring.

AT2 — Latency/Throughput: The online feature store (Redis) provides sub-millisecond feature retrieval for individual users. The offline feature store provides high-throughput training data. Training and serving use different stores because latency and throughput requirements are incompatible in a single system.

AT9 — Correctness/Performance: Approximate nearest neighbour search (HNSW, FAISS) finds candidates in O(log N) with >95% recall, not O(N) with 100% recall. The recall drop is acceptable; the latency reduction (from seconds to milliseconds) is not optional.

Failure Modes in This System

FM4 — Data Consistency Failure: Training-serving skew is the most insidious failure in ML systems. If the features used during training are computed differently from the features used during serving — different bucketing, different normalisation, different time windows — the model’s predictions in production will differ from its validation performance. Features must be computed identically in training and serving, which is why the feature store is the central component.

FM11 — Observability Blindness: Recommendation quality degrades without raising errors. If the feedback loop collapses — no new interaction data is processed, model weights stop updating — recommendations become stale and engagement drops. The signal is falling engagement metrics, not system errors. Click-through rate, session length, and explicit feedback are the observability instruments.

FM3 — Unbounded Resource Consumption: The candidate generation index grows as the catalogue grows. If item embeddings are not expired for deleted or unlisted items, the ANN index consumes unbounded memory. Explicit lifecycle management — removing items from the index when they are unlisted — is required.

How It Evolves at Scale

At 10×: the candidate generation index grows 10×. ANN index quantisation (reducing embedding precision from 32-bit float to 8-bit integer) reduces memory by 4×. The scoring model may be distilled — a smaller student model trained to approximate a larger teacher model, matching 95% of the quality at 20% of the compute.

At 100×: real-time personalisation with bandits — reinforcement learning that explores the recommendation space while exploiting known user preferences. The feedback loop runs in near-real-time (seconds, not hours), enabling rapid adaptation to user mood and context.

Real-World Variants

Netflix uses a multi-stage pipeline with separate models for different recommendation surfaces (home page, because-you-watched, similar items). The home page ranking uses tens of features; the similar items surface uses primarily item-embedding similarity.

YouTube published the 2016 paper “Deep Neural Networks for YouTube Recommendations” — the canonical reference for multi-stage recommendation. Two neural networks: one for candidate generation (user embedding to item embedding), one for ranking (many features, including impression history).

Spotify emphasises audio feature similarity alongside collaborative filtering. Track embeddings are built from audio analysis (tempo, key, energy) combined with listening behaviour embeddings.

Amazon uses item-to-item collaborative filtering at massive scale. “Customers who bought X also bought Y” is a precomputed similarity table, updated continuously as purchases occur.

Exercises