Framing ML problems for products

The big idea

Most failed ML projects do not fail because the model was weak. They fail because the model solved the wrong problem, optimized a number nobody cared about, or replaced a simple rule that was already good enough. Framing comes before modelling: what decision will the prediction change, who acts on it, and how will you know it helped?

Think of a doctor asked to "lower the hospital's readmission rate". The useful question is not "can I predict readmissions?" but "for which patients can we still do something, what would we do, and does doing it help?". A perfectly accurate predictor of readmissions for patients nobody can help is worthless. The same goes for products: a model is only worth building if its predictions drive an action, and that action moves the outcome you care about.

How it works

Step 1: Does it need ML at all?

Google's Rules of Machine Learning open with the same advice. Rule 1: don't be afraid to launch a product without machine learning; ML needs data, and a basic heuristic can get you a long way (the guide puts it at half of a hoped-for ML gain). Rule 2: first, design and implement metrics, tracking as much as possible in the current system. Rule 3: choose ML over a complex heuristic, because a complex heuristic is unmaintainable. The pattern is: simple rule, measure, then a simple model.

Step 2: Turn the goal into a prediction task

A business goal ("more bookings", "fewer fraud losses") has to become a supervised learning problem: an input available at decision time, a label you can observe, and a decision rule on top of the prediction. Many framings are possible, and the first one that comes to mind is often not the best. For "help guests pick a hotel" you could predict clicks, bookings, cancellations, review scores or a guest's flexibility about destination. Each has different data, label delay and link to the goal.

Questions that shake out a framing:

• When is the prediction made, and what is known then? Features from after that moment leak the answer ( Data leakage ).
• What exactly is the label? "Churned" in 30 days or 90? Purchase or purchase-not-refunded?
• How long until the label arrives? A label that arrives months later slows every iteration.
• What does a wrong prediction cost, each way? That sets the threshold and the metric ( Classification metrics ).

Step 3: Proxy metrics against business metrics

Models are trained on a proxy metricA metric that is easy to measure quickly, such as click rate or log loss, used in place of the outcome you actually care about because that outcome is slow, rare or hard to attribute.: something measurable per prediction, such as click-through rate or log loss. The company cares about a business metricThe outcome the product exists to move, such as revenue, retained customers or bookings, usually measured with an online experiment.: revenue, bookings, retention. The two usually move together, which is why proxies work, until a model finds a way to raise the proxy without the goal. Clickbait is the classic case: more clicks, fewer purchases.

So measure both, and judge a model by a randomized experiment on the business metric ( Offline vs online evaluation and A/B tests ), with the proxy as a diagnostic. If a better offline score does not move the business metric, that is information about the framing, not just about the model.

Going deeper

Code: a model that wins the proxy and loses the business

import numpy as np
from sklearn.linear_model import LogisticRegression

# A shop shows each visitor one of 4 banners. Logged data: 40,000 visits with a random banner.
# Each visitor has 3 features; click and purchase probabilities depend on visitor and banner.
rng = np.random.default_rng(0)
n, K = 40000, 4
X = rng.normal(size=(n, 3))
price = np.array([20, 35, 60, 120])                       # revenue per purchase, by banner
def probs(X, b):                                          # the 4th banner (index 3) is flashy: many clicks, few sales
    click = 1 / (1 + np.exp(-(-2.0 + [0.0, 0.2, 0.3, 1.2][b] + 0.8 * X[:, 0] * (b == 1) + 0.8 * X[:, 1] * (b == 2))))
    buy_given_click = 1 / (1 + np.exp(-(-1.0 + [0.5, 0.3, 0.2, -2.5][b] + 0.9 * X[:, 2] * (b >= 2))))
    return click, click * buy_given_click
shown = rng.integers(0, K, n)
click_p = np.array([probs(X, b)[0] for b in range(K)]).T; buy_p = np.array([probs(X, b)[1] for b in range(K)]).T
clicked = rng.random(n) < click_p[np.arange(n), shown]
bought = clicked & (rng.random(n) < buy_p[np.arange(n), shown] / click_p[np.arange(n), shown])

def fit(target):                                          # one model per banner on the logged data
    return [LogisticRegression().fit(X[shown == b], target[shown == b]) for b in range(K)]
ctr_models, buy_models = fit(clicked), fit(bought)

Xt = rng.normal(size=(20000, 3))                          # new visitors: evaluate each policy exactly
def evaluate(choice):
    idx = np.arange(len(Xt))
    cp = np.array([probs(Xt, b)[0] for b in range(K)]).T[idx, choice]
    bp = np.array([probs(Xt, b)[1] for b in range(K)]).T[idx, choice]
    return cp.mean(), (bp * price[choice]).mean()
policies = {
    "random banner": rng.integers(0, K, len(Xt)),
    "heuristic: best average revenue": np.full(len(Xt), np.argmax([bought[shown == b].mean() * price[b] for b in range(K)])),
    "model: maximize click rate": np.argmax([m.predict_proba(Xt)[:, 1] for m in ctr_models], axis=0),
    "model: maximize revenue": np.argmax([m.predict_proba(Xt)[:, 1] * price[b] for b, m in enumerate(buy_models)], axis=0),
}
for name, choice in policies.items():
    ctr, rev = evaluate(choice)
    print(f"{name:32s} click rate {ctr:5.1%}   revenue per visit {rev:5.2f}")

random banner                    click rate 19.5%   revenue per visit  2.03
heuristic: best average revenue  click rate 18.2%   revenue per visit  3.69
model: maximize click rate       click rate 33.1%   revenue per visit  2.79
model: maximize revenue          click rate 23.2%   revenue per visit  4.15

A shop shows each visitor one of four banners. Banner 4 is flashy: many clicks, few purchases. From 40,000 logged visits with random banners, the code builds four policies and scores them exactly on 20,000 new visitors.

• Random banner: 19.5 percent click rate, 2.03 revenue per visit.
• A one-line heuristic, always show the banner with the best average revenue in the logs: 3.69 per visit. That alone captures about 78 percent of the gain of the best model over random.
• A model that maximizes click rate wins the proxy by a mile, 33.1 percent, but earns only 2.79 per visit: it learned to show the flashy banner.
• A model that maximizes expected revenue (purchase probability times price) earns the most, 4.15 per visit, with a click rate of only 23.2 percent.

Same data, same model class; only the framing changed, and it decided whether the project helped.

When the setup itself is the lever

The Booking.com paper in the case study found that some of their most powerful improvements came not from better models in a fixed setup, but from changing the setup, for example turning a user-preference model built on click data into a natural language problem built on guest reviews. Revisit the framing whenever offline gains stop turning into business gains.

Trade-offs

Common mistakes

• Starting with "we should use ML for this" instead of a decision and an outcome.
• Skipping the heuristic baseline, so nobody knows whether the model is worth its cost.
• Treating offline metric gains as business wins without an online test.
• Choosing a label that arrives months after the prediction, then iterating once a quarter.
• Defining features with information that will not exist at prediction time.

In the wild

Check yourself

Recap