Monitoring, drift and retraining

The big idea

A trained model is a snapshot of the world on the day its training data ended. The world keeps moving: new customers arrive, prices change, a competitor launches, fraudsters learn the rules. The model does not crash when this happens. It keeps answering with confidence, just less and less correctly. Monitoring exists to notice that before the business does.

Think of a paper city map. On the day it is printed it is perfect. A year later a few streets have been renamed and a new district has been built. The map still looks authoritative, and nothing on it tells you which parts are out of date. You only find out when you take a wrong turn, unless you regularly compare the map with the streets.

How it works

Step 1: What drifts

Write a model's world as two parts: how inputs are distributed, $P(x)$, and how outcomes depend on inputs, $P(y\mid x)$. Either can change.

• Data driftA change in the distribution of a model's inputs, P(x), such as more mobile users or a new price range, while the relationship between inputs and outcomes may stay the same. (also called covariate shift): the inputs change. A shop expands to a new country and the age and basket-size distributions shift. The rules linking inputs to outcomes may still hold, but the model is now working in regions it saw little of in training.
• Concept driftA change in the relationship between inputs and outcomes, P(y given x): the same input now leads to a different outcome, so the model's learned mapping is wrong.: the relationship changes. The same transaction pattern that used to be safe is now how a new fraud ring operates. Inputs can look identical while the model's answers become wrong.
• Upstream breakage: not the world, but the data pipeline. A unit changes from dollars to cents, a join starts returning nulls. It looks like drift and is the most common cause of sudden jumps.

Step 2: The monitoring loop and label delay

The hard part is label delayThe time between a prediction and the moment its true outcome is known, such as weeks for a chargeback or months for a loan default.. A click label arrives in seconds, a chargeback in weeks, a loan default in months or years. Until labels arrive, you watch what you can see: the inputs, the prediction distribution (has the share of "high risk" scores doubled?), and fast proxies such as user complaints or manual review overturns. Michelangelo, from ML system architecture overview , does exactly this join: predictions are logged and later joined with the observed outcomes from the data pipeline to measure live accuracy.

Step 3: Measuring drift with PSI

The population stability indexPopulation stability index: a number that measures how much a variable's distribution has moved between a reference sample and a current sample, summed over bins. (PSI) comes from credit scoring. Split the training values of a feature into bins (often 10, at the training deciles, so each holds 10 percent). Count the share of live values in each bin. Then sum, over bins, the difference times the log ratio:

where $e_i$ is the expected (training) share and $a_i$ the actual (live) share in bin $i$. Each term is positive whichever way the share moved. Worked example: one bin goes from 10 percent to 15 percent, contributing $(0.15-0.10)\ln 1.5=0.05\times 0.405=0.020$; another goes from 10 to 5 percent, contributing $(-0.05)\ln 0.5=0.035$.

The usual rule of thumb, attributed to Lewis in a 2023 review of population stability tests: below 0.1, no substantial change; 0.1 to 0.25, a small change; 0.25 or more, a substantial change. The same review, and a 2018 dissertation on PSI's statistical properties, warn that these cut-offs ignore the sample size and the number of bins, so treat them as a starting point for alerts, not as a test.

Step 4: The KS test, and why sample size matters

The KS testKolmogorov-Smirnov test: compares two distributions by the largest vertical gap between their cumulative distribution functions, and gives a p-value for the hypothesis that they are the same. compares the cumulative distributions directly: its statistic $D$ is the largest vertical gap between the training and live cumulative curves, and it comes with a p-value. It needs no bins, but it answers a different question: "is there any difference at all?". With tens of thousands of rows, a shift too small to matter gets a microscopic p-value. With a few hundred rows, PSI itself is noisy. The lab below shows both effects.

In practice, monitor PSI (or a similar distance) per feature with thresholds tuned on your own history, look at the features the model relies on most, and treat statistical tests as a hint, not a verdict.

Try it

• At the start the live mean is shifted by 0.3 standard deviations with 2,000 rows. PSI is 0.079, under the 0.1 rule of thumb, while the KS p-value is about 5e-22. Which bins contribute most? (The two outer bins, where the tails moved.)
• Set the shift to 0 and the spread to 1, then switch between 200 and 20,000 rows. With no drift at all, 200 rows give a PSI of 0.047, and 20,000 rows give 0.000. Small samples make PSI noisy.
• Set the shift to 0.05 with 20,000 rows: PSI 0.003, but the KS p-value is about 6e-9. KS calls a shift of one twentieth of a standard deviation "significant" because the sample is large.
• Leave the mean at 0 and set the spread to x1.50. PSI rises to 0.185 with the red bars in bins 1 and 10: drift does not have to move the average.

Going deeper

Code: PSI, KS, and the drift you cannot see

import numpy as np
from scipy import stats
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(7)

def psi(expected, actual, bins=10):
    """Population stability index over decile bins of the training (expected) data."""
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1)); edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.clip(np.histogram(actual, edges)[0] / len(actual), 1e-4, None)
    return np.sum((a - e) * np.log(a / e))

train_x = rng.normal(0, 1, 50_000)                         # one feature as seen in training
print("live data                      n        PSI    KS p-value")
for name, shift, scale in [("same distribution", 0, 1), ("mean +0.05 sd", 0.05, 1),
                           ("mean +0.5 sd", 0.5, 1), ("spread x1.5", 0, 1.5)]:
    for n in [500, 50_000]:
        live = rng.normal(shift, scale, n)
        p = stats.ks_2samp(train_x, live).pvalue
        print(f"{name:22s} {n:9,d}  {psi(train_x, live):7.3f}  " + (f"{p:10.2g}" if p > 0 else "  < 1e-300"))

# Concept drift: inputs look the same, but the relationship between input and label flips for a segment.
def world(n, flipped):
    X = rng.normal(0, 1, (n, 2)); logit = 2 * X[:, 0] - X[:, 1]
    if flipped: logit = np.where(X[:, 1] > 0.5, -logit, logit)   # behaviour changes for one segment
    return X, (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
X0, y0 = world(20_000, False); model = LogisticRegression().fit(X0, y0)
for week, flipped in [(1, False), (2, False), (3, True), (4, True)]:
    X, y = world(5_000, flipped)
    print(f"week {week}: PSI x1 {psi(X0[:, 0], X[:, 0]):.3f}, x2 {psi(X0[:, 1], X[:, 1]):.3f}"
          f"   accuracy once labels arrive {model.score(X, y):.3f}")

live data                      n        PSI    KS p-value
same distribution            500    0.020        0.23
same distribution         50,000    0.000        0.59
mean +0.05 sd                500    0.006        0.96
mean +0.05 sd             50,000    0.003     1.9e-15
mean +0.5 sd                 500    0.350     7.1e-27
mean +0.5 sd              50,000    0.246    < 1e-300
spread x1.5                  500    0.253     1.1e-07
spread x1.5               50,000    0.197      2e-218
week 1: PSI x1 0.003, x2 0.001   accuracy once labels arrive 0.791
week 2: PSI x1 0.002, x2 0.002   accuracy once labels arrive 0.804
week 3: PSI x1 0.002, x2 0.002   accuracy once labels arrive 0.597
week 4: PSI x1 0.003, x2 0.002   accuracy once labels arrive 0.608

The first table compares a training sample of 50,000 values with live samples of 500 and 50,000, using scipy's two-sample KS test. It reproduces the lab's lessons: with no drift, 500 rows give PSI 0.020 by chance; a 0.05 standard deviation shift has a PSI of 0.003 but a KS p-value of 1.9e-15 at 50,000 rows; a 0.5 standard deviation shift has a PSI of 0.246 with 50,000 rows and 0.350 with 500.

The second part is the important one. A model is trained on two features. In week 3 the relationship between inputs and outcome flips for one segment of users (those with a high second feature), while the inputs keep exactly the same distribution. Every PSI stays at 0.003 or below, so input monitoring sees nothing. Accuracy, once labels arrive, falls from 0.791 and 0.804 in weeks 1 and 2 to 0.597 in week 3. Concept drift is only visible in outcomes, which is why label delay matters so much.

Retraining triggers

• On a schedule ( Training pipelines and orchestration ): simple and predictable; catches slow drift; wastes compute when nothing changed and is late for sudden changes.
• On input drift: fast, but noisy, and drift in a feature the model barely uses may not matter. Use it to alert a person or to check performance, not to retrain blindly.
• On a performance drop: the most direct signal, but it waits for labels, and it needs enough labelled volume per window to be stable.

Retraining is not always the cure. After concept drift, retraining helps only once enough labels from the new regime exist; retraining on old labels just rebuilds the old model. After an upstream bug, the fix is the pipeline, not the model; retraining on broken data teaches the model the bug.

Trade-offs

Common mistakes

• Monitoring only latency and error rates, so a silently decaying model looks healthy.
• Alerting on every feature with a KS p-value below 0.05; with large traffic everything is "significant" all the time.
• Treating stable inputs as proof the model still works; concept drift leaves inputs unchanged.
• Retraining automatically on data from a broken pipeline.
• Measuring performance only in aggregate, so a collapse in one segment is averaged away.

Where you will see this

PSI comes from credit risk, where banks use it to check whether the population a scorecard is applied to still resembles the one it was built on. Production ML platforms log every prediction for the same reason: Michelangelo joins predictions with observed outcomes to track live accuracy, and model-monitoring features in the major cloud ML services compare live feature distributions with a training baseline. Fraud teams, whose labels arrive with chargebacks weeks later, lean on drift and review-queue signals in between, a theme that returns in the fraud detection case study (ML-17.01).

Check yourself

Recap