Experiment tracking and reproducibility

The big idea

A model that scored 0.88 last Tuesday is useless if nobody can say how it was made. Which version of the code? Which snapshot of the data? Which learning rate, which random seed? Training ML models is a stream of experiments, often hundreds of them, and memory and file names like model_final_v2_really.pkl do not scale.

Experiment trackingRecording, for every training run, the code version, data version, parameters, metrics and output files, so runs can be compared, searched and reproduced. is the lab notebook of ML. A chemist writes down every reagent, quantity and temperature before looking at the result, so any experiment can be repeated and any surprising result traced. A tracker does the same for training runs, automatically, and adds a searchable table on top.

How it works

Step 1: Log parameters, metrics and artifacts

A tracker organizes work into runs (one execution of a training script) grouped into experiments (all runs for one task). For each run you log:

• Parameters: inputs you chose, such as learning rate, tree depth, feature set.
• Metrics: numbers that measure the result, such as validation loss, often logged per epoch so you can plot curves.
• Artifacts: output files, such as the trained model, plots and evaluation reports.

MLflow, an open-source tracker, looks like this. By default it writes to a local mlruns directory, and mlflow server opens a web UI to compare runs:

import mlflow

mlflow.set_experiment("churn")
with mlflow.start_run():
    mlflow.log_param("max_depth", 6)
    model = train(max_depth=6)                 # your training code
    mlflow.log_metric("val_auc", evaluate(model))
    mlflow.log_artifact("confusion_matrix.png")

mlflow.autolog() captures parameters, metrics and the model automatically for libraries such as scikit-learn, XGBoost and PyTorch. Weights and Biases (W&B), a hosted tracker, works the same way: wandb.init starts a run, run.config holds the hyperparameters and run.log records metrics over time.

Step 2: Version the code and the data

Parameters are not enough if the code or data changed underneath them. The code version is easy: record the git commit of the training code with every run (MLflow tags each run with the git commit automatically when it is started from a git repository). Data is harder, because datasets are too big for git. Options:

• Record a content hash of the exact files used; any change to the data changes the hash.
• Use immutable, dated snapshots in the data lake, and log the snapshot path.
• Use a data-versioning tool such as DVC, which keeps large files in a cache and remote storage and commits a small .dvc placeholder file with the file's hash to git, so a git commit pins both code and data.

Step 3: Fix the seeds, and know their limits

Training uses randomness everywhere: weight initialization, data shuffling, dropout, train/test splits, the bootstrap samples of a random forest. Fixing every random seedThe starting value of a pseudo-random number generator; the same seed produces the same sequence of random numbers. makes a run repeatable on the same machine and library versions. In NumPy that means a seeded generator; in scikit-learn, random_state; in PyTorch, torch.manual_seed(0).

Even then, PyTorch's own documentation warns that completely reproducible results are not guaranteed across releases, individual commits or platforms, and that results may differ between CPU and GPU even with identical seeds. For one platform you can go further: PyTorch's deterministic-algorithms switch (use_deterministic_algorithms) uses deterministic implementations where they exist and raises an error otherwise; deterministic operations are often slower. So log the library versions and hardware too.

Try it

import hashlib, json
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

def fingerprint(*arrays):                       # data version: a hash of the exact bytes used
    h = hashlib.sha256()
    for a in arrays: h.update(np.ascontiguousarray(a).tobytes())
    return h.hexdigest()[:10]

runs = []                                       # a minimal tracker: one record per training run
def train_run(X, y, params, code="v1"):
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=params["split_seed"])
    model = RandomForestClassifier(n_estimators=params["trees"], max_features=params["max_features"],
                                   random_state=params["seed"]).fit(Xtr, ytr)
    acc = model.score(Xte, yte)
    runs.append({"run": len(runs) + 1, "code": code, "data": fingerprint(X, y), "params": params, "acc": round(acc, 4)})
    return acc

X, y = make_classification(n_samples=600, n_features=20, n_informative=5, flip_y=0.1, random_state=0)
base = {"trees": 50, "max_features": "sqrt", "seed": 0, "split_seed": 0}
train_run(X, y, base); train_run(X, y, base)                   # runs 1, 2: same everything
for s in [1, 2, 3, 4]: train_run(X, y, {**base, "seed": s})    # runs 3-6: only the model seed changes
for s in range(5): train_run(X, y, {**base, "max_features": 0.5, "seed": s})   # runs 7-11: the change
X2, y2 = X.copy(), y.copy(); y2[:30] = 1 - y2[:30]             # run 12: 30 labels changed upstream
train_run(X2, y2, base)

for r in runs:
    print(f"run {r['run']:2d}  acc {r['acc']:.4f}  data {r['data']}  {json.dumps(r['params'])}")
a = np.array([r["acc"] for r in runs[1:6]]); b = np.array([r["acc"] for r in runs[6:11]])
print(f"runs 1 and 2 identical: {runs[0]['acc'] == runs[1]['acc']}")
print(f"sqrt features, 5 seeds: {a.mean():.4f} +- {a.std():.4f}   (range {a.min():.4f} to {a.max():.4f})")
print(f"0.5 features,  5 seeds: {b.mean():.4f} +- {b.std():.4f}   (range {b.min():.4f} to {b.max():.4f})")
print(f"run 12 data fingerprint changed: {runs[11]['data'] != runs[0]['data']}, acc {runs[11]['acc']}")

run  1  acc 0.8778  data b0e0463fd4  {"trees": 50, "max_features": "sqrt", "seed": 0, "split_seed": 0}
run  2  acc 0.8778  data b0e0463fd4  {"trees": 50, "max_features": "sqrt", "seed": 0, "split_seed": 0}
run  3  acc 0.8722  data b0e0463fd4  {"trees": 50, "max_features": "sqrt", "seed": 1, "split_seed": 0}
run  4  acc 0.8667  data b0e0463fd4  {"trees": 50, "max_features": "sqrt", "seed": 2, "split_seed": 0}
run  5  acc 0.8444  data b0e0463fd4  {"trees": 50, "max_features": "sqrt", "seed": 3, "split_seed": 0}
run  6  acc 0.8500  data b0e0463fd4  {"trees": 50, "max_features": "sqrt", "seed": 4, "split_seed": 0}
run  7  acc 0.8444  data b0e0463fd4  {"trees": 50, "max_features": 0.5, "seed": 0, "split_seed": 0}
run  8  acc 0.8833  data b0e0463fd4  {"trees": 50, "max_features": 0.5, "seed": 1, "split_seed": 0}
run  9  acc 0.8722  data b0e0463fd4  {"trees": 50, "max_features": 0.5, "seed": 2, "split_seed": 0}
run 10  acc 0.8500  data b0e0463fd4  {"trees": 50, "max_features": 0.5, "seed": 3, "split_seed": 0}
run 11  acc 0.8667  data b0e0463fd4  {"trees": 50, "max_features": 0.5, "seed": 4, "split_seed": 0}
run 12  acc 0.8333  data a8f97ea84a  {"trees": 50, "max_features": "sqrt", "seed": 0, "split_seed": 0}
runs 1 and 2 identical: True
sqrt features, 5 seeds: 0.8622 +- 0.0129   (range 0.8444 to 0.8778)
0.5 features,  5 seeds: 0.8633 +- 0.0143   (range 0.8444 to 0.8833)
run 12 data fingerprint changed: True, acc 0.8333

The code is a tracker in a dozen lines: each run appends a record with a data fingerprint (a hash of the exact arrays), the parameters and the test accuracy of a random forest on 600 examples. Read the table it prints:

• Runs 1 and 2 use the same data, parameters and seeds and get exactly the same accuracy, 0.8778. That is what fixed seeds buy you.
• Runs 1 to 6 change only the model's seed, and accuracy ranges from 0.8444 to 0.8778. That spread is noise, not skill.
• Runs 7 to 11 test a change (max_features 0.5 instead of sqrt). Compare run 1 with run 7 and the change looks 3.3 points worse; compare run 5 with run 8 and it looks 3.9 points better. Averaged over five seeds it is 0.8633 against 0.8622: no real difference.
• Run 12 has identical parameters to run 1, but 30 labels were changed upstream. Accuracy drops to 0.8333, and only the changed data fingerprint explains why.

Trade-offs

Common mistakes

• Logging parameters and metrics but not the data version, so a changed upstream table goes unnoticed.
• Comparing two single runs and calling a difference smaller than seed-to-seed noise an improvement.
• Seeding one library (NumPy) and forgetting another (the framework, the data loader, the split).
• Training from uncommitted code, so the logged commit does not match what actually ran.
• Keeping only the best run, which hides how many configurations were tried to find it.

Where you will see this

Open-source trackers such as MLflow, hosted ones such as Weights and Biases, and the experiment features of the big cloud ML platforms all use the same model: runs grouped into experiments, each with parameters, metrics and artifacts. In-house platforms do the same. Uber's Michelangelo, from the previous lesson, stores a versioned record for every trained model with who trained it, the full model configuration, references to the training and test data, accuracy metrics and the learned parameters, so a deployed model can always be traced back to the run that made it.

Check yourself

Recap