Recommender Systems Explained: Collaborative Filtering in Python

Walk into a sweet shop in Chhappan Dukan that you visit every week. The owner sees you and says, "Fresh rabri today. You liked the malpua last time, and people who order malpua usually love this."

In one sentence he has used two kinds of knowledge. He knows what the dishes are like (rabri and malpua are both rich, milk-based sweets). And he knows who tends to like what (malpua fans often order rabri).

A recommender system does the same thing for millions of customers at once, on a food app, a shopping site or a streaming service.

What is a recommender system?

Map the sweet shop to the system:

• The customers are users and the dishes are items.
• Every past order or rating is an interaction, stored in a user-item matrix.
• "Rich, milk-based" is an item feature, the basis of content-based filtering.
• "People who order malpua love rabri" is collaborative filtering.

A recommender system is a model that estimates how much each user will like each item they have not yet seen, then shows the highest-scoring items as personalised suggestions.

It sits in the Advanced ML part of this series because it combines several earlier ideas: similarity from k-nearest neighbours, compression from PCA, and careful evaluation from model evaluation metrics. Code uses numpy, pandas and scikit-learn 1.7.

Explicit vs implicit feedback

Explicit feedback is a direct signal, such as a 4-star rating. It is clear but rare, because most people never rate anything. Implicit feedback is behaviour: clicks, orders, watch time, add-to-cart. It is plentiful but noisy, since a click is not always a like. Most real systems run mainly on implicit data.

Types of recommender systems

How does collaborative filtering work?

Collaborative filtering ignores what items are. It only looks at who interacted with what. If you and another user rated many dishes the same way, your unrated dishes can be predicted from theirs.

Collaborative filtering fills a missing cell using rows (similar users) or columns (similar items) that agree with the known ratings.

Item-based collaborative filtering is usually the better starting point. Item-to-item similarity changes slowly, while user tastes shift every week.

The steps are:

1. Build the user-item matrix and subtract each user's average rating, so a strict rater and a generous rater become comparable.
2. Compute cosine similarity between every pair of item columns.
3. To predict a user's rating for item j, take the items they did rate, weight each rating by its similarity to j and average.

import numpy as np
import pandas as pd

dishes = ["poha", "jalebi", "dal_bafla", "malpua", "rabri", "garadu"]
R = pd.DataFrame([[5, 4, np.nan, 1, 1, 4],
                  [4, 5, 5, 2, np.nan, 5],
                  [1, 2, 1, 5, 5, np.nan],
                  [2, np.nan, 1, 4, 5, 1],
                  [5, 5, 4, np.nan, 2, 4]],
                 index=["Aman", "Bhavna", "Chirag", "Divya", "Esha"], columns=dishes)

centred = R.sub(R.mean(axis=1), axis=0).fillna(0)       # remove each user's bias
M = centred.to_numpy()
norms = np.linalg.norm(M, axis=0)
sim = pd.DataFrame(M.T @ M / np.outer(norms, norms), index=dishes, columns=dishes)
print(sim.round(2))

def predict(user, item):
    rated = R.loc[user].dropna().index
    w = sim.loc[item, rated].clip(lower=0)               # use only positive neighbours
    dev = R.loc[user, rated] - R.loc[user].mean()
    return R.loc[user].mean() + (w * dev).sum() / (w.sum() + 1e-9)

print(f"\nAman on dal_bafla: {predict('Aman', 'dal_bafla'):.2f}")
print(f"Divya on jalebi:   {predict('Divya', 'jalebi'):.2f}")

Poha, jalebi, dal bafla and garadu form one taste group (Indore's breakfast and street-food crowd), and malpua and rabri form another. Aman belongs to the first group, so his predicted dal bafla rating is 4.3. Divya prefers milk sweets, so her jalebi prediction is about 1.5.

Content-based filtering with TF-IDF

Content-based filtering compares item descriptions instead of users. It works on day one for a new dish, as long as you can describe it. TF-IDF turns text into weighted word vectors, and cosine similarity finds the closest items.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

menu = {
    "poha":      "flattened rice savoury breakfast light onion peanut",
    "dal_bafla": "wheat dumpling ghee lentil savoury heavy malwa",
    "jalebi":    "fried sweet syrup crispy breakfast",
    "malpua":    "fried sweet pancake syrup rich milk",
    "rabri":     "sweet milk rich thickened dessert",
    "garadu":    "fried yam spicy savoury winter street snack",
    "khichdi":   "rice lentil light savoury comfort ghee",   # brand-new item, no ratings yet
}
names = list(menu)
tfidf = TfidfVectorizer().fit_transform(menu.values())
sim = cosine_similarity(tfidf)

for target in ["khichdi", "rabri"]:
    i = names.index(target)
    best = sim[i].argsort()[::-1][1:3]
    print(f"{target}: most similar ->", [(names[j], round(float(sim[i, j]), 2)) for j in best])

Khichdi has no orders yet, but its description already links it to poha and dal bafla. That is how content features solve the item cold-start problem. The text step is the same idea as the embeddings behind modern search, explained in embeddings explained.

What is matrix factorisation in recommender systems?

The user-item matrix on a real app is huge and mostly empty. A food app might have a million users and 50,000 dishes, and each user has ordered maybe 30 of them, so more than 99.9% of cells are blank.

Matrix factorisation assumes a few hidden factors explain taste, such as "sweet vs savoury" or "light vs heavy". It learns a short vector for every user and every item so that their dot product approximates each known rating.

The sparse ratings matrix R is approximated by U times V transposed. Each user and each item gets a short vector of hidden taste factors, and their dot product predicts the rating.

The code below learns two factors by gradient descent on known ratings only, holding a few ratings back to test.

import numpy as np

rng = np.random.default_rng(0)
n_users, n_items, k = 50, 12, 2
U_true = rng.normal(0, 1, (n_users, k))
V_true = rng.normal(0, 1, (n_items, k))
full = 3 + U_true @ V_true.T + rng.normal(0, 0.3, (n_users, n_items))

known = np.argwhere(rng.random((n_users, n_items)) < 0.4)    # only 40% of cells observed
rng.shuffle(known)
test, train = known[:40], known[40:]

U = rng.normal(0, 0.1, (n_users, k)); V = rng.normal(0, 0.1, (n_items, k))
mu, lr, reg = full[tuple(train.T)].mean(), 0.05, 0.02
for epoch in range(200):
    for u, i in train:
        err = full[u, i] - (mu + U[u] @ V[i])
        U[u], V[i] = U[u] + lr * (err * V[i] - reg * U[u]), V[i] + lr * (err * U[u] - reg * V[i])

def rmse(pairs, pred):
    return np.sqrt(np.mean([(full[u, i] - pred(u, i)) ** 2 for u, i in pairs]))

print(f"train cells: {len(train)}, test cells: {len(test)}")
print(f"test RMSE, predict the global mean: {rmse(test, lambda u, i: mu):.2f}")
print(f"test RMSE, matrix factorisation:    {rmse(test, lambda u, i: mu + U[u] @ V[i]):.2f}")

From only 40% of cells, two learned factors cut the error on hidden ratings to about a third of the mean-only baseline. The reg term is L2 regularisation, the same idea as in ridge and lasso regression: it stops rarely seen users from getting extreme factors.

Recommender systems in production

Real apps cannot score every item for every user on each visit, so they split the work:

1. Candidate retrieval: quickly pull a few hundred plausible items, using item similarity, embeddings in a vector index, or popularity in the user's area.
2. Ranking: score those candidates with a richer model, often gradient boosting, using features like time of day, price and distance.
3. Re-ranking: apply business rules such as removing out-of-stock items or adding variety.

The retrieval step uses the same nearest-neighbour search as vector databases.

How do you evaluate a recommender system?

Users see a short list, so what matters is whether good items appear near the top. Common offline metrics:

• Precision@k: of the top k suggestions, the share the user actually liked.
• Recall@k / hit rate@k: whether the held-out item appears in the top k.
• Coverage and diversity: how much of the catalogue ever gets recommended.

A simple, honest test is leave-one-out: hide each user's last interaction, recommend from the rest, and check whether the hidden item lands in the top 5.

import numpy as np

rng = np.random.default_rng(5)
n_users, n_items = 300, 40
group = rng.integers(0, 4, n_users)                     # 4 hidden taste groups
item_group = np.arange(n_items) % 4
p = np.where(group[:, None] == item_group[None, :], 0.35, 0.03)
p = p * np.linspace(1.5, 0.5, n_items)                  # some items are more popular overall
X = (rng.random((n_users, n_items)) < p).astype(float)

held = np.array([rng.choice(np.flatnonzero(r)) if r.sum() > 1 else -1 for r in X])
users = np.flatnonzero(held >= 0)
train = X.copy(); train[users, held[users]] = 0

norms = np.linalg.norm(train, axis=0) + 1e-9
item_sim = (train.T @ train) / np.outer(norms, norms)
np.fill_diagonal(item_sim, 0)

def hit_rate(scores):
    scores = np.where(train > 0, -np.inf, scores)        # never recommend what they already have
    top5 = np.argsort(-scores, axis=1)[:, :5]
    return np.mean([held[u] in top5[u] for u in users])

popularity = np.tile(train.sum(axis=0), (n_users, 1))
print(f"hit rate@5, most-popular list:  {hit_rate(popularity):.2f}")
print(f"hit rate@5, item-based CF:      {hit_rate(train @ item_sim):.2f}")

Popularity is a fair baseline, but item-based collaborative filtering roughly triples the hit rate because it uses each user's own history. Always beat a popularity baseline before shipping anything fancier, and confirm with an online A/B test, as covered in A/B testing.

Cold start and other common pitfalls

• New user cold start: no history yet. Show popular items in their area, or ask for two or three preferences at sign-up.
• New item cold start: no interactions yet. Use content features until clicks arrive.
• Popularity bias: collaborative models keep pushing hits, and niche items never get shown. Track catalogue coverage.
• Feedback loops: users click what you show, so the model learns from its own choices. Reserve some slots for exploration.
• Random splits on time-ordered data: test on interactions that happen after training ones, or scores look better than reality.
• Optimising clicks alone: a list that maximises clicks can hurt repeat orders. Pick a metric that matches the business goal.

Building and evaluating a full recommender makes a strong portfolio project, and it goes faster with a mentor checking your evaluation setup, for example in Debugshala's placement-focused ML course. To revise similarity and ranking concepts, solve recommender and similarity MCQs on Debugshala Test.

Key takeaways

• Recommenders predict a user's interest in unseen items and show the top few.
• Content-based filtering uses item features; collaborative filtering uses other users' behaviour; matrix factorisation learns hidden taste factors.
• Item-based collaborative filtering is a strong, explainable starting point.
• Production systems retrieve candidates fast, then rank them with a richer model.
• Evaluate with precision@k or hit rate on held-out interactions, and always beat a popularity baseline.
• Plan for cold start, popularity bias and feedback loops.