Design a feed ranking system

The problem

A reader opens the app. Thousands of posts are eligible: from friends, from accounts they follow, from groups, from creators they have never seen who are popular with people like them. The screen shows about three. A feed ranker decides the order, in under a couple of hundred milliseconds, hundreds of millions of times a day.

The engineering is hard, but the defining difficulty is the objective. A ranker optimized for clicks learns to show clickbait, outrage and the same topic over and over, because those get clicks today. Readers then feel worse about the time they spent and come back less. "What should the feed optimize?" is a product question that the ML system has to turn into numbers.

Think of a newspaper editor choosing the front page. They pull candidate stories from every desk, judge each one on several things at once (will people read it, will they trust us more for it, is it important), then step back and make sure the page is not five stories about the same football match. A feed ranker does the same three steps, personalized for each reader: gather candidates, score, then compose the page.

Requirements

Functional

• Given a reader and a request (app open or scroll), return an ordered page of about 10 posts from all eligible sources.
• Respect hard rules: blocked accounts, content policy, posts already seen in this session, ads slots handled elsewhere.
• Mix sources: friends, followed accounts, groups, recommended content from outside the reader's network.
• Log every impression with its position and features, and every action (click, like, comment, share, hide, report, dwell time).

Non-functional

• Latency: P99 under 200 ms for the full ranking pipeline, of which the heavy model gets perhaps 50 to 80 ms.
• Freshness: a post from a close friend should be rankable within seconds of being published.
• Scale: tens of thousands of feed requests per second at peak, each scoring hundreds to thousands of candidates.
• Integrity: ranking must not amplify content that violates policy, even if it is engaging.

Out of scope

• Feed storage, fan-out on write vs read, and caching of timelines ( Design a news feed (Twitter/Instagram) ).
• Ad ranking and auctions ( Design an ad click-through-rate prediction system ).
• Content moderation classifiers themselves ( Design a spam / content moderation classifier ), which feed this system as signals.

Estimates

Assumed numbers for a large social app, not any company's figures:

# Back-of-envelope sizing for a feed ranker. Every input is an assumption; change them and re-run.
dau = 100_000_000
sessions_per_user = 8
pages_per_session = 2              # each page is a feed request for about 10 posts
peak_factor = 2
candidates = 1_000                 # posts retrieved from all sources per request
heavy_ranked = 300                 # survivors of the light ranker, scored by the big model
mflops_per_score = 20              # heavy multi-task model, per post
posts_per_page = 10

requests = dau * sessions_per_user * pages_per_session
peak_qps = requests / 86_400 * peak_factor
print(f"feed requests per day        {requests / 1e9:.1f} billion, peak {peak_qps:,.0f} per second")
print(f"light ranker scores / s      {peak_qps * candidates / 1e6:,.0f} million at peak")
print(f"heavy ranker scores / s      {peak_qps * heavy_ranked / 1e6:,.1f} million at peak")
print(f"heavy ranker compute         {peak_qps * heavy_ranked * mflops_per_score * 1e6 / 1e12:,.0f} TFLOP/s at peak")
impressions = requests * posts_per_page
print(f"impressions logged per day   {impressions / 1e9:,.0f} billion (training rows before sampling)")

feed requests per day        1.6 billion, peak 37,037 per second
light ranker scores / s      37 million at peak
heavy ranker scores / s      11.1 million at peak
heavy ranker compute         222 TFLOP/s at peak
impressions logged per day   16 billion (training rows before sampling)

• Requests: 100 million daily users, 8 sessions and 2 pages each, is 1.6 billion feed requests a day, about 37,000 per second at peak.
• The funnel is mandatory. Scoring 1,000 candidates per request with the heavy model would be 37 million heavy scores per second. A cheap light ranker cuts the list to 300 first, and even then the heavy model does 11 million scores a second, about 222 TFLOP/s at an assumed 20 MFLOPs per score: a fleet of accelerators, not a few CPUs.
• Training data is enormous. 16 billion impressions a day. Teams sample negatives (impressions without action) heavily and keep all positives.

Framing, objective and metrics

Several predictions, one score

There is no single label for "a good post to show". The model predicts several actions, each a separate head of a multi-task modelA model with a shared body and several output heads, each trained on a different label (click, like, share, hide), so related tasks share what they learn.:

• $P(\text{click})$, $P(\text{like})$, $P(\text{comment})$, $P(\text{share})$: positive engagement of increasing effort;
• $E[\text{dwell time}]$: time spent, a regression head;
• $P(\text{hide})$, $P(\text{report})$, $P(\text{unfollow})$: negative feedback.

A value modelA hand-set formula that combines a model's predicted probabilities into one ranking score, with weights chosen by the product team and tuned with experiments. combines them into the score the feed sorts by:

The weights are the product's values written as numbers. Worked example with $w_c=1,w_l=4,w_h=10$: a clickbait post with $P(\text{click})=0.5$, $P(\text{like})=0.05$, $P(\text{hide})=0.08$ scores $0.5+0.2-0.8=-0.1$. A thoughtful post with $0.25$, $0.3$, $0.01$ scores $0.25+1.2-0.1=1.35$ and wins easily, though it gets half the clicks.

Engagement vs long-term value

Short-term signals are easy to predict and easy to move. What the business needs is long-term valueAn outcome measured over weeks or months, such as retention or survey satisfaction, that short-term engagement metrics only partly predict.: readers who come back next week and say the time was well spent. Practical ways to bring it into the system:

• Weight effortful and explicit signals higher (likes, comments, shares, "show more like this") than passive ones (clicks).
• Penalize negative feedback heavily: one hide carries far more information than one missing click.
• Use surveys ("was this worth your time?") on a sample of impressions as an extra label and a head of the model.
• Pick weights by long experiments: run A/B tests for weeks, judge them on retention and survey scores, and accept a drop in clicks if those improve.
• Guardrail metrics that must not get worse: reports, hides, time on integrity-flagged content.

High-level design

Deep dives

Deep dive 1: Candidate generation, ranking, re-ranking

Candidate generation must be fast and have high recall: it is fine to bring back junk, not fine to miss the best post. Several sources run in parallel, each returning a few hundred: recent posts from the reader's network (sorted by a simple affinity), embedding retrievalRetrieving items whose learned embedding is near the user's embedding, using a two-tower model and an approximate nearest neighbour index. from a two-tower model over an approximate nearest neighbour index ( Two-tower retrieval and embedding-based retrieval ), and fresh or trending posts that have no history yet. Mixing sources is itself a design choice: give each a quota so one source cannot crowd out the rest.

Light ranking is a small model (a tiny neural network or boosted trees on a few dozen features) that orders 1,000 candidates cheaply so the heavy model sees 300. It is often trained to imitate the heavy model, since its only job is to not drop what the heavy model would have ranked high.

Heavy ranking is the big multi-task model: hundreds of features (reader history, post content embeddings, author, reader-author interaction counts, freshness, context) through a shared body and one head per action. Shared layers help the rarer tasks (shares, reports) learn from the common ones (clicks). When tasks conflict, mixtures of experts or per-task towers on top of the shared body reduce interference.

Re-ranking composes the final page from the scored list. A pure sort by score gives ten posts from the same topic or author. A greedy re-ranker such as MMRMaximal marginal relevance: a greedy re-ranker that picks, at each step, the item with the best trade-off between relevance and dissimilarity to items already picked. picks one post at a time, penalizing similarity to what is already on the page ( Cold start, diversity and feedback loops ). Business rules live here too: no more than two posts in a row from one author, at least one fresh post in the first five, policy demotions.

Deep dive 2: Choosing the objective

The sample below simulates 2,000 readers and 300 posts, a fifth of them clickbait (more clicks, fewer likes, more hides). Each reader's top 10 is chosen by three objectives.

# Feed ranking on synthetic posts: which objective to rank by, and how position bias distorts click logs.
import numpy as np

rng = np.random.default_rng(5)
users, posts = 2000, 300
bait = rng.random(posts) < 0.2                         # 20 percent of posts are clickbait
quality = rng.normal(0, 1, posts) - 1.2 * bait         # how much a reader is glad they saw it
affinity = rng.normal(0, 1, (users, posts))            # per-user interest
p_click = 1 / (1 + np.exp(-(-2 + 0.6 * affinity + 1.5 * bait + 0.3 * quality)))
p_like = 1 / (1 + np.exp(-(-3 + 0.6 * affinity + 1.2 * quality)))
p_hide = 1 / (1 + np.exp(-(-4 + 2.0 * bait - 0.8 * quality - 0.3 * affinity)))

def feed_stats(name, score):
    top = np.argsort(-score, axis=1)[:, :10]           # each user's top 10
    pick = lambda m: np.take_along_axis(m, top, axis=1).mean()
    print(f"{name:34s} clicks {pick(p_click):.3f}  likes {pick(p_like):.3f}  hides {pick(p_hide):.3f}"
          f"  clickbait share {bait[top].mean():4.0%}")

feed_stats("rank by P(click)", p_click)
feed_stats("rank by click + 4 like", p_click + 4 * p_like)
feed_stats("rank by click + 4 like - 10 hide", p_click + 4 * p_like - 10 * p_hide)

# Position bias: a reader examines slot k with probability 1/k, and clicks only what they examine.
slots = 10
exam = 1 / np.arange(1, slots + 1)
true_ctr = rng.uniform(0.05, 0.3, slots)               # ten posts, each shown in a fixed slot by the old ranker
shown = 100_000
clicks = rng.binomial(shown, exam * true_ctr)
naive = clicks / shown
ips = clicks / (shown * exam)                          # inverse propensity weighting
print("true CTR rank     ", np.argsort(-true_ctr))
print("naive CTR rank    ", np.argsort(-naive))
print("IPS-corrected rank", np.argsort(-ips))
print(f"mean abs error: naive {np.abs(naive - true_ctr).mean():.3f}, IPS {np.abs(ips - true_ctr).mean():.3f}")

rank by P(click)                   clicks 0.538  likes 0.099  hides 0.121  clickbait share  94%
rank by click + 4 like             clicks 0.321  likes 0.405  hides 0.005  clickbait share   2%
rank by click + 4 like - 10 hide   clicks 0.312  likes 0.407  hides 0.004  clickbait share   0%
true CTR rank      [5 7 6 1 9 2 3 0 8 4]
naive CTR rank     [0 1 2 5 3 6 7 9 4 8]
IPS-corrected rank [5 6 7 1 9 2 3 0 8 4]
mean abs error: naive 0.105, IPS 0.003

• Ranking by click probability fills 94 percent of slots with clickbait. Clicks are high (0.538 per post), but likes are low (0.099) and 12 percent of impressions end in a hide.
• Adding likes with weight 4 removes almost all clickbait (2 percent), cuts clicks by 40 percent (to 0.321) and quadruples likes (to 0.405). Hides fall from 0.121 to 0.005.
• Adding the hide penalty removes the rest of the clickbait at almost no further cost.

If the team is judged on clicks, the first row looks best. That is why the objective must be chosen against long-term outcomes, with weights set by experiments that measure retention and satisfaction, not by whichever row moves the dashboard.

Deep dive 3: Position bias

Readers click the first post more because it is first. Logged clicks therefore measure "good post times good position", and a model trained on raw clicks learns to keep showing whatever the old ranker put on top. This is position biasThe tendency of users to interact more with items shown higher in a list, regardless of their relevance, which biases click logs., and it creates a feedback loop.

Model it as examination times relevance: the reader examines slot $k$ with probability ${\theta }_k$ and clicks if the post appeals, so $P(\text{click})={\theta }_k\cdot P(\text{appeal})$. Corrections:

• Inverse propensity weighting: weight each logged click by $1/{\theta }_k$, so clicks from low positions count more. With ${\theta }_k=1/k$, a click at slot 5 counts 5 times.
• Position as a feature in training, a constant at serving: the model learns how much position adds; at serving you set position to 1 for every candidate so they are compared fairly.
• Estimate ${\theta }_k$ with small randomized experiments: swap two adjacent results for a sliver of traffic and compare click rates.

The second half of the sample shows ten posts that the old ranker always showed in the same slots. Naive click rates rank them almost by slot, with a mean error of 0.105. Dividing by the examination probability recovers the true ranking almost exactly, with a mean error of 0.003.

Try it

1. Start with all sliders at 0, ranking by click probability alone: the page gets 4.37 expected clicks but 7 of 10 posts are clickbait, with 0.99 expected likes and 1.86 hides.
2. Raise the like weight to 2: clickbait drops to 3 posts. At 4 it is gone: clicks fall to 2.78 while likes more than double to 2.33 and hides drop to 0.14. Which page would you rather scroll?
3. Set like weight 4 and hide penalty 10 (football fills 7 of 10 slots, 3 topics). Raise diversity to 1.0: football drops to 4 posts and the page covers 5 topics, for a small cost in predicted likes (2.34 to 2.24). Predicted engagement says diversity is a loss; long-term experiments are how you find out whether readers agree.

Going deeper

Evaluation and experiments

Offline, evaluate each head separately (AUC and log loss for the probability heads, calibration plots per head, because the value model multiplies them by weights and miscalibrated heads silently change the trade-off) and the ranking as a whole with NDCG on held-out sessions. Split by time.

Online, A/B tests decide. Feed experiments need care:

• Long durations (weeks), because novelty effects fade and retention moves slowly.
• Holdouts: a small group that keeps an old ranker for months, to measure the cumulative effect of many small launches.
• Network effects: a ranker that makes some readers post more changes what everyone else sees. Cluster-randomized experiments (by friend group or region) reduce this leakage.
• Metric hierarchy: a north-star such as daily active readers or survey satisfaction, supported by engagement metrics, protected by guardrails (reports, hides, integrity).

Monitoring and feedback loops

Monitor per-head calibration, score distributions, the share of each candidate source in final pages, content diversity, the fraction of fresh posts, and integrity metrics. Watch for feedback loopsWhen a model's own outputs shape the data it is retrained on, so its biases are reinforced over time.: the ranker decides what gets impressions, impressions decide what gets labels, and labels train the next ranker. A small exploration budget (randomly promoting some lower-ranked candidates) keeps the training data from collapsing onto whatever the model already likes.

Bottlenecks and failure modes

Trade-offs we made

Extensions

• Sequence-aware ranking: score the page as a whole (what goes well after what), not post by post.
• Creator-side objectives: give new creators enough distribution to find an audience.
• User controls: "show less of this topic" as both a feature and a hard rule.
• Reinforcement learning over sessions to optimize return visits directly ( RL in the real world ).

In the wild

Where you will see this: every social feed, video app, music app and news aggregator uses a retrieval, ranking and re-ranking funnel with a multi-task model, and most have publicly debated the gap between engagement and well-being that the objective choice creates. The same funnel powers search ( Design a search ranking system ) and ads ( Design an ad click-through-rate prediction system ).

Check yourself

Recap