ML system architecture overview

The big idea

In a notebook, a model is a few lines: load a table, call fit, print a score. In a product, that same model has to be retrained when new data arrives, fed the same features at 3 a.m. on a phone request as it saw in training, answered within a few milliseconds, and watched so someone notices when it quietly gets worse. The model is a small box; the system around it is most of the work.

Think of a restaurant. The recipe is the model. But a restaurant also needs suppliers delivering fresh ingredients (data pipelines), a prep station where ingredients are cut the same way every time (features), a test kitchen that tries new recipes and keeps notes (training and experiment tracking), a book of approved recipes (the model registry), the line that cooks orders fast during the rush (serving) and someone tasting plates and reading reviews (monitoring). A great recipe in a chaotic kitchen still produces bad dinners.

How it works

Step 1: Data pipelines

Everything starts with logged data: transactions, clicks, sensor readings, labels. Data pipelinesScheduled or streaming jobs that collect raw data, clean and validate it, and compute features and labels from it. turn that raw data into features and labels. Batch pipelines run every few hours or once a day over the data lake; streaming pipelines update aggregates within seconds from an event stream ( Stream processing ). Pipelines should validate what they produce, because a broken upstream table is the most common way a good model starts making bad predictions.

Step 2: The feature store

A feature storeA system that stores feature values computed from shared definitions, serving full history to training (offline store) and the latest values to live predictions (online store). holds computed features in two forms. The offline store keeps the full history, so a training set can ask "what was this restaurant's average prep time on the day of this order?". The online store keeps only the latest value per key in a fast database, so a live request can fetch it in a millisecond or two. The key idea is that both are filled from the same feature definitions; the feature store lesson later in this module ( Feature stores and training-serving skew ) goes into how.

Step 3: Training pipeline and model registry

A training pipeline is the notebook turned into a repeatable job: build a training set, train, evaluate against a baseline, and stop if the new model is worse. A model that passes goes into the model registryA versioned catalogue of trained models with the metadata needed to reproduce, compare and deploy them: training data reference, parameters, metrics and status. with everything needed to trust it later: which data and code produced it, its hyperparameters and its metrics. Deployment then means "serve version 7 of this model", not "copy this file from a laptop".

Step 4: Serving

Serving turns a registered model into predictions. Some products need predictions on request (an ETA when the user opens the app): an online service loads the model, fetches features from the online store, and answers within a latency budget. Others only need predictions on a schedule (tomorrow's demand per store): a batch job scores everything and writes the results to a table. Serving patterns get their own lesson later in this module ( Model serving patterns ).

Step 5: Monitoring and the loop back

Monitoring watches three things: the system (latency, errors), the inputs (are feature distributions still like training?) and the outputs (once real outcomes arrive, how accurate were the predictions?). When those drift, a retraining run starts and the loop closes. Without this loop, a model is a snapshot of the world on the day it was trained.

Offline and online paths

The platform has two speeds. The offline path (pipelines, offline store, training, registry) runs in hours and optimizes for throughput and reproducibility. The online path (online store, serving) runs in milliseconds and optimizes for latency and availability. The danger is the seam between them: if a feature is computed one way for training and another way for serving, the model sees inputs in production that it never saw in training. That mismatch is training-serving skewA difference between the data or features a model saw in training and what it receives in production, often from computing the same feature with two code paths., and sharing one definition across both paths is the main defence against it.

Going deeper

Code: a whole platform in under 40 lines

import numpy as np
from sklearn.linear_model import LinearRegression
# 1. Data pipeline: a log of 60 days of orders at 50 restaurants (prep minutes, delivery minutes).
rng = np.random.default_rng(2)
R, D = 50, 60
base = rng.uniform(8, 30, R)                                   # each restaurant's typical prep time
def day_orders(slow=0.0):                                      # 40 orders per restaurant per day
    r = np.repeat(np.arange(R), 40); dist = rng.uniform(1, 8, r.size)
    prep = base[r] * (1 + slow) + rng.normal(0, 4, r.size)
    return r, dist, prep, prep + 3 * dist + rng.normal(0, 3, r.size)
log = [day_orders() for d in range(D)]
# 2. One feature definition, used by the offline AND the online store: 7-day mean prep time.
def avg_prep_7d(days):
    r = np.concatenate([o[0] for o in days]); p = np.concatenate([o[2] for o in days])
    return np.bincount(r, p, R) / np.bincount(r, minlength=R)
# 3. Offline store + training: each day's orders joined with the feature as of the day before.
X, y = [], []
for d in range(7, D):
    f = avg_prep_7d(log[d - 7:d]); r, dist, _, total = log[d]
    X.append(np.c_[f[r], dist]); y.append(total)
X, y = np.vstack(X), np.concatenate(y); cut = int(0.8 * len(y))
model = LinearRegression().fit(X[:cut], y[:cut])
mae = np.abs(model.predict(X[cut:]) - y[cut:]).mean()
baseline = np.abs(y[:cut].mean() - y[cut:]).mean()
# 4. Registry with a validation gate: register only if the model beats the baseline.
registry = {"eta-v1": {"features": ["avg_prep_7d", "distance"], "train_days": "7-59", "mae": round(float(mae), 2)}}
print(f"offline MAE {mae:.2f} min, baseline {baseline:.2f}, registered: {mae < baseline}", registry)
# 5-7. Serve 4 more weeks: look up the online store, predict, log; join labels when they arrive.
# From week 3 every kitchen is 50% slower. The online store is refreshed nightly, unless the job broke.
for refresh in [True, False]:
    served, online, weekly = list(log), avg_prep_7d(log[-7:]), []
    for day in range(28):
        r, dist, prep, total = day_orders(slow=0.5 if day >= 14 else 0.0)
        pred = model.predict(np.c_[online[r], dist])
        weekly.append(np.abs(pred - total).mean())               # monitoring: live MAE per day
        served.append((r, dist, prep, total))
        if refresh: online = avg_prep_7d(served[-7:])            # nightly batch job
    print("nightly refresh" if refresh else "refresh broken ", "live MAE by week:",
          [round(float(np.mean(weekly[w:w + 7])), 2) for w in range(0, 28, 7)])

offline MAE 4.03 min, baseline 7.93, registered: True {'eta-v1': {'features': ['avg_prep_7d', 'distance'], 'train_days': '7-59', 'mae': 4.03}}
nightly refresh live MAE by week: [3.96, 3.96, 6.6, 3.97]
refresh broken  live MAE by week: [3.99, 4.03, 9.73, 9.65]

The code builds every box of Figure 1 in miniature for a delivery-time model at 50 restaurants. One function, avg_prep_7d, is the single feature definition. The offline path joins each day's orders with the feature as of the day before, trains a linear model and registers it only because it beats a predict-the-average baseline: 4.03 minutes mean absolute error against 7.93. The online path serves four more weeks, looking up the online store for each order and logging the error once the delivery completes.

From week 3 every kitchen becomes 50 percent slower. With the nightly job refreshing the online store, live error jumps to 6.60 in the week of the change and is back to 3.97 a week later, because the feature catches up and the model already knows how prep time maps to delivery time. With the refresh job broken, the error sits at 9.73 and 9.65. Nothing in the model changed: a stale pipeline alone more than doubled the error, and only the monitoring loop would tell you.

Where the effort goes

The NIPS 2015 paper "Hidden technical debt in machine learning systems" (Sculley and colleagues at Google) drew the picture this lesson is built around: only a small fraction of a real-world ML system is the ML code; the surrounding infrastructure is vast and complex. It named the ways that infrastructure goes wrong:

• Glue code. Supporting code written to get data into and out of general-purpose packages; the paper notes a mature system might be at most 5 percent ML code and at least 95 percent glue code.
• Pipeline jungles. Data preparation that grows organically with new sources and joins until it is hard to test or change.
• Entanglement (CACE). Changing anything changes everything: inputs are mixed inside the model, so no feature or setting can be changed in isolation.
• Undeclared consumers. Other systems silently start using a model's outputs, so changing the model breaks them.
• Hidden feedback loops. Two systems influence each other indirectly through the world, for example one picks the products on a page and another picks the reviews shown next to them. The direct version, a model shaping its own future training data, is the subject of Cold start, diversity and feedback loops .

A platform does not remove this debt, but it gives every team the same pipelines, the same feature definitions and the same registry, so the debt is paid once instead of per project.

Trade-offs

Common mistakes

• Treating the model as the product and leaving data pipelines, serving and monitoring as afterthoughts.
• Computing the same feature twice, once in a training notebook and once in the serving code.
• Deploying files by hand with no registry, so nobody can say which model is live or how it was trained.
• Monitoring only latency and errors, not inputs and prediction quality.
• Building a full platform before there is a second model to share it with.

In the wild

Check yourself

Recap