Recommender fundamentals

The big idea

A streaming service has tens of thousands of titles; you will look at maybe twenty on the home screen. A shop has a million products; you will scroll past a few dozen. A recommender systemA system that predicts which items a user is likely to want and shows the best few, such as films, products, songs or posts. decides which few you see. Its job is to predict, for a user and an item they have not tried, how much they would like it, then show the items with the highest predictions.

Think of two ways a friend in a video shop might help. One asks what you liked and finds films with the same genre, director or actors: "you liked Alien, try Arrival". The other notices that people who liked the same films as you also loved something you have never heard of: "everyone with your taste raves about this one". The first is content-based filtering, the second collaborative filtering, and between them they cover the classic ideas behind most recommenders.

How it works

Step 1: The user-item matrix

Put users in rows and items in columns. Each cell holds what you know about that user and that item: a rating, a purchase, a play count. This is the user-item matrixA table with one row per user and one column per item, holding ratings or interactions; almost all cells are empty., and recommendation becomes filling in its empty cells.

Real matrices are far emptier than this toy. The classic MovieLens 100K data set has 100,000 ratings from 943 users on 1,682 films: 100,000 / (943 x 1,682), about 6.3 percent of cells filled, even though every user rated at least 20 films. On a shop with millions of products the filled share is a tiny fraction of a percent. This sparsityThe share of a matrix's cells that are empty; user-item matrices are usually more than 99 percent empty. is the central difficulty of the whole field.

Step 2: Explicit and implicit feedback

What goes in the cells?

• Explicit feedbackFeedback a user gives on purpose to express preference, such as star ratings or thumbs up and down. is a deliberate opinion: 4 stars, thumbs down. It is clear, but most people rarely give it.
• Implicit feedbackFeedback inferred from behaviour, such as clicks, plays, watch time and purchases, rather than stated on purpose. is behaviour: clicks, plays, minutes watched, purchases, add to cart. It is plentiful, since every user produces it all the time, which is why most production recommenders run on it.

Implicit feedback is not just "explicit feedback with more rows". Hu, Koren and Volinsky's paper on implicit-feedback collaborative filtering lists its traits. There is no negative feedback: a film you never played might be one you dislike or one you never saw. It is inherently noisy: you may buy a gift you would never choose for yourself, or leave a show running while you sleep. And its numbers measure confidence, not preference: watching a show ten times makes it more certain you like it, not that you like it ten times more. That shapes the models in the next lesson, Collaborative filtering and matrix factorization.

Step 3: Content-based filtering

Content-based filteringRecommending items whose features resemble those of items the user liked, using item attributes rather than other users' behaviour. describes each item with features: genre, price band, artist, words in the description, or an embedding of the image. It builds a profile of each user from the features of the items they liked, then recommends items whose features match the profile.

In the figure, Ana rated Alien 5, Arrival 4 and Notting Hill 1. Centre her ratings on 3 stars (+2, +1, -2) and average them by genre: her profile is +1.0 for sci-fi and -0.67 for romance. Interstellar is sci-fi, so it scores +1.0; the two unseen romances score -0.67.

It needs no other users, so it works for a brand-new item the moment its features exist. But it can only recommend more of what you already know. It will never suggest a documentary to a sci-fi fan, and it is only as good as the features.

Step 4: Collaborative filtering

Collaborative filteringRecommending items based on the behaviour of many users: people who agreed in the past tend to agree again, so similar users' choices predict yours. ignores item features entirely. It uses only the matrix: people who agreed in the past will probably agree again.

The user-based version works like this. Measure how similar every other user is to Ana, using the films both rated; cosine similarity of their mean-centred ratings works well ( Vectors and vector spaces ). Then predict Ana's rating of a film as her own average plus a similarity-weighted average of how far her neighbours' ratings of it sat above or below their own averages. The item-based version flips it around: two items are similar if the same users rated them alike, and you recommend items similar to those the user liked. Because item-to-item similarities depend only on the matrix, not on the user asking, they can be computed offline ahead of time and looked up at request time.

Collaborative filtering discovers links nobody wrote down, for example that fans of one band also love a podcast. But it needs history. A new item with no interactions cannot be recommended, and a new user cannot be matched to anyone. That is the cold-start problemThe problem of recommending for new users or new items that have little or no interaction history., the subject of the lesson Cold start, diversity and feedback loops later in this module.

Going deeper

Code: both methods on the toy matrix

import numpy as np

movies = ["Alien", "Arrival", "Interstellar", "Notting Hill", "Love Actually", "Mamma Mia"]
genres = np.array([[1, 0], [1, 0], [1, 0], [0, 1], [0, 1], [0, 1]])   # columns: sci-fi, romance
R = np.array([            # explicit ratings 1-5; 0 means "not rated"
    [5, 4, 0, 1, 0, 0],   # Ana (we recommend for her)
    [5, 5, 4, 0, 1, 2],   # Ben
    [4, 0, 5, 2, 1, 0],   # Cy
    [1, 0, 2, 5, 4, 5],   # Dee
    [0, 1, 1, 4, 5, 4],   # Eve
])
user, rated = 0, R[0] > 0
print(f"matrix density: {np.mean(R > 0):.0%} of cells known")

# Content-based: build Ana's taste from the genres of what she rated (centred on 3 stars).
profile = ((R[0, rated] - 3) @ genres[rated]) / rated.sum()
content = genres @ profile
print("content-based scores for Ana's unrated movies:")
for i in np.flatnonzero(~rated):
    print(f"  {movies[i]:14s} {content[i]:+.2f}")

# Collaborative filtering: weight other users' ratings by how similar their tastes are.
def centred(row):
    m = row[row > 0].mean()
    return np.where(row > 0, row - m, 0.0)
C = np.array([centred(r) for r in R])
sim = C @ C[user] / (np.linalg.norm(C, axis=1) * np.linalg.norm(C[user]))
print("similarity to Ana:", " ".join(f"{n}={s:+.2f}" for n, s in zip(["Ana", "Ben", "Cy", "Dee", "Eve"], sim)))
mean_ana = R[0, rated].mean()
print("collaborative predictions for Ana:")
for i in np.flatnonzero(~rated):
    who = [u for u in range(1, len(R)) if R[u, i] > 0 and sim[u] > 0]   # positive neighbours who rated i
    pred = mean_ana + sum(sim[u] * C[u, i] for u in who) / sum(sim[u] for u in who)
    print(f"  {movies[i]:14s} {pred:.2f} stars  (from {len(who)} similar user{'s' * (len(who) != 1)})")

print(f"MovieLens 100K: 100,000 ratings / (943 users x 1,682 movies) = {100_000 / (943 * 1682):.1%} filled")

matrix density: 73% of cells known
content-based scores for Ana's unrated movies:
  Interstellar   +1.00
  Love Actually  -0.67
  Mamma Mia      -0.67
similarity to Ana: Ana=+1.00 Ben=+0.35 Cy=+0.43 Dee=-0.72 Eve=-0.33
collaborative predictions for Ana:
  Interstellar   4.71 stars  (from 2 similar users)
  Love Actually  1.15 stars  (from 2 similar users)
  Mamma Mia      1.93 stars  (from 1 similar user)
MovieLens 100K: 100,000 ratings / (943 users x 1,682 movies) = 6.3% filled

The toy matrix is 73 percent filled, far denser than any real one, which keeps the arithmetic visible.

• Content-based gives Interstellar +1.00 and both unseen romances -0.67, exactly the profile computed above.
• Collaborative: Ben (+0.35) and Cy (+0.43) resemble Ana, while Dee (-0.72) and Eve (-0.33) have opposite taste. Using only the positive neighbours who rated each film, Ana gets 4.71 stars for Interstellar, 1.15 for Love Actually and 1.93 for Mamma Mia.
• Both methods agree here. Mamma Mia rests on only one neighbour, a reminder that collaborative predictions are weakest exactly where data is thin.

Baselines and hybrids

Always compare against popularity: recommend the most-played items the user has not seen. It is hard to beat for new users and is the usual fallback for cold start. Production systems are almost always hybrids: collaborative signals for users and items with history, content features for new ones, often combined inside one model. The rest of this module builds the machinery: factorizing the matrix, retrieving candidates from millions of items, and learning to rank the final list.

Trade-offs

Common mistakes

• Treating unobserved items as dislikes. In implicit data a missing cell usually means "never saw it".
• Reading play counts as ratings: the tenth play adds confidence, not ten times the liking.
• Shipping a clever model without checking that it beats "most popular".
• Computing user similarity on raw ratings, so a generous rater and a harsh rater with identical taste look different. Centre each user's ratings first.
• Evaluating on a random split when the product recommends the future: hold out each user's latest interactions instead.

Where you will see this

Every large consumer product runs recommenders: the rows on a streaming home screen, "customers who bought this also bought" in online shops, music playlists built for you, the order of posts in a social feed and the jobs or people suggested on professional networks. Most use implicit feedback such as plays, clicks and dwell time far more than star ratings, and most combine collaborative signals with content features.

Check yourself

Recap