Cold start, diversity and feedback loops

The big idea

A recommender learns from what users did with the items it showed them. That sentence hides two problems. First, a new user or a new item has no history, so the model knows nothing about them: the cold-start problemThe situation where a new user or item has no interaction history, so collaborative methods cannot score it.. Second, the system only learns about items it chose to show. An item it never shows gets no clicks, so it looks bad, so it is never shown. The model's own choices become its training data: a feedback loopA loop where a model's outputs shape the data it is later trained on, so early choices and biases reinforce themselves..

Picture a restaurant guide written only from the reviews of restaurants the guide already recommends. The places on page one get the visitors, so they get the reviews, so they stay on page one. A new restaurant around the corner could be the best in town and never be reviewed. Good recommenders deliberately send some traffic to the new and the uncertain, and deliberately mix the page so one taste does not take over.

How it works

Step 1: The loop

Three biases ride this loop:

• Popularity biasThe tendency of a recommender to over-recommend already popular items, because they have the most interactions to learn from and win most comparisons.. Popular items have the most data and the most confident scores, so they win ties and gather still more data. The long tail of the catalogue is rarely shown.
• Position bias. Items at the top get clicked partly for being at the top ( Learning to rank ). Training on raw clicks teaches the model to copy its own previous ranking.
• Bias toward the past. A model trained on old logs predicts the past. The YouTube recommendations paper found that, without care, it would be very difficult for new content to surface; feeding the age of each training example as an input feature removed that bias toward the past.

Over many rounds the loop can narrow what each user sees to more of what they already clicked, the effect popularly called a filter bubbleA popular term for a user seeing a narrowing range of content because personalization keeps showing more of what they engaged with before..

Step 2: Cold start for new users

With no history, use whatever you do have, roughly in this order:

• Context. Country, language, device, time of day, the page or search that brought them in. Even a two-tower user tower ( Two-tower retrieval and embedding-based retrieval ) can run on these features alone.
• Popularity, scoped. Trending in your region or category beats global top lists, and it is a strong baseline that is hard to beat early.
• Onboarding. Ask for a few topics, artists or genres at sign-up; each answer is a strong early signal.
• Session signals. After three clicks in this visit, a session-based model has something to work with, long before a nightly batch job sees the user.

Step 3: Cold start for new items

New items are the bigger long-term danger, because they are where the feedback loop bites.

• Content features. An item tower that reads text, images and metadata gives a new item a sensible vector before its first click. This is the main reason to prefer feature-based models over pure ID-based matrix factorization.
• Reserved exposure. Give new items a guaranteed share of impressions, for example a slot or two per page, to collect their first interactions.
• Optimism under uncertainty. Score an item by its estimated quality plus a bonus that shrinks as evidence grows. That is the upper confidence bound idea from Multi-armed bandits , and Thompson sampling does the same with probabilities.

Step 4: Diversity on the page

The ranker scores each item alone, so the top ten can be ten near-copies of one thing. Re-ranking fixes the page as a whole. Maximal marginal relevanceMaximal marginal relevance: a greedy re-ranker that picks, at each step, the item with the best trade-off between relevance and dissimilarity to items already picked. (MMR, Carbonell and Goldstein) builds the list one item at a time, each time picking

where $S$ is what is already on the page. With $\lambda =1$ it is the plain ranking; lower $\lambda$ trades relevance for variety. Simpler rules work too: at most two items per creator, per category or per seller.

Going deeper

Code: greedy vs exploring in a feedback loop

import numpy as np

# A feedback loop: each day the system shows 10 items, users click, and tomorrow's ranking is
# learned from those clicks. 300 items exist on day 0; 5 new items arrive every day.
def simulate(policy, days=200, seed=0):
    rng = np.random.default_rng(seed)
    p = list(rng.beta(2, 40, 300))                       # true click rate of each item, about 5 percent
    shows = list(rng.poisson(50, 300) * (rng.random(300) < 0.1))   # old log: only 10% of items were ever shown
    clicks = [rng.binomial(s, q) for s, q in zip(shows, p)]
    total = 0
    for day in range(days):
        for _ in range(5):                               # cold-start items: no history at all
            p.append(rng.beta(2, 40)); shows.append(0); clicks.append(0)
        P, S, C = np.array(p), np.array(shows), np.array(clicks)
        ctr = (C + 1) / (S + 20)                         # smoothed click rate (prior about 5 percent)
        if policy == "greedy":
            top = np.argsort(-ctr)[:10]
        elif policy == "explore":                        # 8 slots exploit, 2 slots try items never shown much
            top = list(np.argsort(-ctr)[:8])
            rest = [i for i in np.flatnonzero(S < 200) if i not in top]
            top += list(rng.choice(rest, 2, replace=False))
        else:                                            # ucb: optimism for items with few impressions
            top = np.argsort(-(ctr + np.sqrt(2 * np.log(day + 2) / (S + 20)) * 0.2))[:10]
        top = np.array(top)
        c = rng.binomial(100, P[top])                    # 100 impressions per slot per day
        for i, k in zip(top, c):
            shows[i] += 100; clicks[i] += k
        total += c.sum()
    S, P = np.array(shows), np.array(p)
    best = set(np.argsort(-P)[:10])
    share_top10 = np.sort(S)[::-1][:10].sum() / S.sum()
    return total, len(best & set(np.argsort(-S)[:10])), (S > 0).mean(), share_top10, P[np.argsort(-ctr)[:10]].mean()

print(f"{'policy':8s} {'clicks':>7s} {'best-10 found':>14s} {'items ever shown':>17s} {'top-10 share':>13s} {'true CTR of final top 10':>25s}")
for pol in ["greedy", "explore", "ucb"]:
    t, found, ever, share, final = simulate(pol)
    print(f"{pol:8s} {t:7d} {found:14d} {ever:17.0%} {share:13.0%} {final:25.3f}")

policy    clicks  best-10 found  items ever shown  top-10 share  true CTR of final top 10
greedy     18058              0                3%           99%                     0.090
explore    25403              3               27%           66%                     0.151
ucb        19442              8              100%           30%                     0.184

300 items with hidden click rates (about 5 percent on average), plus 5 brand-new items every day for 200 days. The starting log only covers 10 percent of items. Each day the system shows 10 items, 100 impressions each, and re-estimates click rates with a smoothed estimate, $(\text{clicks}+1)/(\text{shows}+20)$.

• Greedy shows only 3 percent of all items ever, 99 percent of its impressions go to just 10 items, and none of the true best 10 is among its most-shown 10. It earned 18,058 clicks.
• Exploring 2 of 10 slots on items with little history found 3 of the best 10, showed 27 percent of items and earned the most clicks, 25,403.
• A UCB bonus showed every item at least once, found 8 of the best 10 and ended with the best top 10 (true click rate 0.184), but its exploration cost clicks over this 200-day window: 19,442.

Exploration is a cost now for a better catalogue later. How much to pay is a product decision; the bandits lesson treats it formally.

Measuring the loop

Offline accuracy on logged data cannot see the items the old system never showed. Track coverage (share of the catalogue shown at all), concentration (share of impressions on the top 1 percent of items), how quickly new items get their first impressions, and the age of items being clicked. To evaluate a new policy from logs, the logs must record how likely each shown item was to be shown, so off-policy estimates can re-weight them; random exploration slots make that possible.

Trade-offs

Common mistakes

• Evaluating only on logged clicks, which can never reward items the old system did not show.
• Using an ID-only model, so every new item has an empty vector until it collects data.
• Treating "not clicked" as "disliked" when the item was never shown, or shown at rank 30.
• Adding exploration without logging which impressions were exploratory, so you cannot learn from them cleanly.
• Fixing diversity with random shuffling instead of a relevance-aware re-ranker or explicit caps.

Where you will see this

Every large feed, store and streaming service faces these problems, and the remedies in this lesson (reserved slots for new items, uncertainty bonuses, caps per source or creator, diversity re-ranking) are the standard toolbox. The YouTube recommendations paper (linked in Two-tower retrieval and embedding-based retrieval ) is a published example: it adds the age of each training example as a feature precisely so fresh videos are not buried by a model trained on the past.

Check yourself

Recap