Feature stores and training-serving skew

The big idea

A model learns a mapping from features to outcomes. That mapping is only useful if, in production, the model receives the same features, computed the same way, from the same kind of moment in time, as it saw in training. Break any of the three and the model is answering a question it was never trained on, usually without any error message.

Think of grading a weather forecaster. To judge Monday's forecast fairly, you must give them only what was known on Sunday night: Sunday's satellite images, not Tuesday's. And when they forecast live, they must get the same instruments, in the same units, as the ones they practised on. A forecaster trained on Tuesday's images looks brilliant in practice and useless on the job. A feature storeA system that computes features from shared definitions and serves them consistently: full history with timestamps for building training sets, and the latest value per key for live predictions. is the archive and the instrument panel at once: one definition of each feature, a time-stamped history for training and a fast copy of the latest values for serving.

How it works

Step 1: One definition, two stores

Feast, the open-source feature store linked at the end, splits the work the same way. It describes an offline store "for historical feature extraction used in model training" and an online store "for serving features at low-latency in production". Its online store keeps only the latest values: "No historical values are stored." Values reach it through materializationA job that reads the latest feature values from the offline store or a data source and writes them into the online store. or are written directly through push sources, for features computed from a stream.

In code, the two paths become two calls against the same named features. This is Feast's quickstart, shown as a listing:

from feast import FeatureStore

store = FeatureStore(repo_path=".")
features = ["driver_hourly_stats:conv_rate", "driver_hourly_stats:avg_daily_trips"]

# Training: one row per (driver_id, event_timestamp), values as of each timestamp
training_df = store.get_historical_features(entity_df=entity_df, features=features).to_df()

# Serving: latest values for the drivers in this request
online = store.get_online_features(features=features, entity_rows=[{"driver_id": 1004}]).to_dict()

Note what a feature store is not. Feast says plainly that it is not a general-purpose data pipelining system and does not orchestrate workflow DAGs. Heavy aggregations are still computed by your pipelines, run by an orchestrator (the next lesson, Training pipelines and orchestration ). The store's job is to name, time-stamp, store and serve the results consistently.

Step 2: Point-in-time correct joins

A training example is a pair: what was known at some moment, and what happened afterwards. If a customer's churn label is "cancelled within 30 days of 1 March", the features must be the values as they stood on 1 March. Joining the label table with today's feature table gives every row values from its own future.

This is an as-of joinA join that, for each row with a timestamp, takes the most recent feature value at or before that timestamp, so no row sees values from its future. (pandas calls it merge_asof). Feature stores do it for you: the input to Feast's get_historical_features is an entity dataframe of keys plus an event_timestamp column, and for each row Feast "will scan backward in time from the entity dataframe timestamp up to a maximum of the TTL". The TTLTime to live: how old a feature value may be and still be used. Older values are treated as missing. matters in both directions: without one, a row could pick up a value that went stale months ago; with one, a row whose last value is too old gets a missing value, exactly as it would at serving time if the pipeline had stopped.

One subtlety: the time a value describes is not always the time it became available. A daily table of yesterday's totals that lands at 6 a.m. was not known at midnight. Join on the time the value was available, or add the pipeline's delay to the timestamp; Feast offers an option to also require the row's created timestamp to be at or before the entity timestamp, which blocks backfilled values from leaking into training.

Step 3: Where training-serving skew comes from

Training-serving skewA difference between how a model performs, or what it receives, in training and in serving, caused by differences in data handling, by data changing between training and serving, or by feedback loops. is the gap between what the model saw in training and what it gets in production. Google's Rules of Machine Learning list three causes: a discrepancy between how data is handled in the training and serving pipelines, a change in the data between training and serving, and a feedback loop between the model and the system. In practice the first cause takes several forms:

• Two implementations. The feature is SQL in the training pipeline and Java in the service. Windows, time zones, rounding and null handling drift apart.
• Time travel in training. A naive join gives training rows future values (Figure 2). Offline metrics soar; live metrics do not.
• Freshness. Training used values exactly as of each event; serving reads a cache refreshed nightly, so live values are up to a day old.
• Defaults. A missing value is 0 in training and -1, or the mean, in serving.

Step 4: Preventing it

• Define each feature once and generate both stores from it. Google's Rule 32: reuse code between your training pipeline and your serving pipeline whenever possible.
• Log features at serving time. Rule 29: "The best way to make sure that you train like you serve is to save the set of features used at serving time, and then pipe those features to a log to use them at training time." The guide reports that YouTube's home page switched to logging features at serving time, with significant quality improvements and less code. It needs patience: a new feature has no logged history until you start logging it.
• Use point-in-time joins with a TTL for anything built from history.
• Measure skew. Rule 37: the same example should get exactly the same prediction whether it is scored in the training pipeline or at serving time, so a discrepancy there probably indicates an engineering error, not a modelling one. Comparing logged serving features with the training pipeline's values for the same rows finds the culprit. ML system architecture overview 's monitoring loop is where this check lives.

Going deeper

Code: a leak and a skew, measured

import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
# Event log: 3,000 subscribers order over 180 days. 30% cancel; their orders thin out in the 60 days before.
rng = np.random.default_rng(4)
U = 3000
rate = rng.gamma(30, 0.01, U)                                   # orders per day while active (about 0.3)
cancel_day = np.where(rng.random(U) < 0.3, rng.uniform(40, 180, U), np.inf)
tickets = rng.poisson(np.where(np.isfinite(cancel_day), 2.0, 1.0))  # a static feature: support tickets
rows = []
for u in range(U):
    t = np.sort(rng.uniform(0, min(cancel_day[u], 180), rng.poisson(rate[u] * 180)))
    keep = rng.random(t.size) < np.clip((cancel_day[u] - t) / 60, 0.1, 1)
    rows.append(pd.DataFrame({"user": u, "ts": t[keep]}))
orders = pd.concat(rows)

def orders_30d(ex, window=30):                                  # ONE feature definition, evaluated as of a time
    o = orders.merge(ex.reset_index(), on="user")
    o = o[(o.ts <= o.as_of) & (o.ts > o.as_of - window)]
    return np.bincount(o["index"], minlength=len(ex))

# Training examples: (user, prediction day); label = cancels within the next 30 days.
ex = pd.DataFrame({"user": rng.integers(0, U, 8000), "as_of": rng.uniform(40, 150, 8000)})
ex = ex[cancel_day[ex.user] > ex.as_of].reset_index(drop=True)   # only users still active at that time
y = (cancel_day[ex.user] <= ex.as_of + 30).astype(int)
pit = orders_30d(ex)                                            # point-in-time: as of each row's own timestamp
latest = orders_30d(ex.assign(as_of=180.0))                     # naive join: today's snapshot for every row
skewed = orders_30d(ex, window=7)                               # serving code reimplemented: 7-day window
T = tickets[ex.user]; tr = np.arange(len(ex)) < 5000; te = ~tr

def run(name, train_f, live_f):
    m = LogisticRegression().fit(np.c_[train_f, T][tr], y[tr])
    p = m.predict_proba(np.c_[live_f, T][te])[:, 1]
    print(f"{name:36s} AUC {roc_auc_score(y[te], p):.3f}  mean predicted risk {p.mean():5.1%}"
          f"  weights orders {m.coef_[0][0]:+.2f} tickets {m.coef_[0][1]:+.2f}")

print(f"{len(ex)} examples, {y[te].mean():.1%} of test rows cancel within 30 days")
run("point-in-time: offline test", pit, pit)
run("latest snapshot: offline test", latest, latest)
run("latest snapshot: live, as served", latest, pit)
run("point-in-time: live with 7-day skew", pit, skewed)
print("first 5 test rows, training value:", pit[te][:5], " served value:", skewed[te][:5])

7099 examples, 6.8% of test rows cancel within 30 days
point-in-time: offline test          AUC 0.719  mean predicted risk  7.2%  weights orders -0.15 tickets +0.58
latest snapshot: offline test        AUC 0.945  mean predicted risk  7.1%  weights orders -1.26 tickets +0.07
latest snapshot: live, as served     AUC 0.669  mean predicted risk  0.2%  weights orders -1.26 tickets +0.07
point-in-time: live with 7-day skew  AUC 0.728  mean predicted risk 16.1%  weights orders -0.15 tickets +0.58
first 5 test rows, training value: [17 15 13 12  5]  served value: [7 2 4 5 2]

A subscription service wants to predict which users cancel in the next 30 days so it can send them an offer. Users who cancel order less in the two months before. The model has two features: orders_30d, the only time-dependent one, and a static count of support tickets. There are 7,099 training examples, each a (user, prediction day) pair, and 6.8 percent of the test rows cancel.

• Point-in-time features: offline AUC 0.719, and the average predicted risk, 7.2 percent, is close to the real rate. The model uses both features.
• Latest-snapshot features (today's value joined to every row): offline AUC jumps to 0.945. Every user who has already cancelled has 0 recent orders today, so the feature leaks the label. The model puts almost all its weight on orders (-1.26) and nearly ignores tickets (+0.07, down from +0.58).
• That leaky model, served with real point-in-time values: AUC 0.669, worse than the honest model, and an average predicted risk of 0.2 percent, so a threshold for offers would fire for almost nobody. The 0.945 was never real.
• Skew: the honest model, but the serving team re-implemented orders_30d with a 7-day window. Ranking happens to survive (AUC 0.728), but every user looks less active than in training and the average predicted risk is 16.1 percent, more than double the true rate. Every offer threshold, budget and alert tuned offline is now wrong.

The last line of output shows why nothing crashed: the training value and the served value for the same user (17 against 7, for example) are both plausible numbers. Skew does not throw exceptions; only comparing logged serving features with training features catches it.

Online features computed at request time

Some features only exist at request time: the distance between the user and a restaurant, or the amount of this transaction compared with the user's average. These cannot be precomputed into the online store. Feature stores handle them as on-demand transformations: a function registered with the store that runs on request data plus stored features, and that the store also applies when building training sets, so the same code runs on both paths. Feast's quickstart includes one: it adds a value sent with the request to a stored conversion rate.

Trade-offs

Common mistakes

• Joining labels with the current feature table instead of values as of each example's timestamp.
• Using the time a value describes rather than the time it became available, so daily tables leak a few hours of future.
• Reimplementing a feature in the serving language "because it is simple"; window boundaries and null handling will differ.
• Setting no TTL, so training rows silently use values months old that serving would treat as missing.
• Celebrating an offline jump in AUC without asking whether a new feature could have seen the label.

In the wild

Check yourself

Recap