- Learn
- Machine Learning
- Model Evaluation and Selection
ML-M05
Model Evaluation and Selection
How to know whether a model is really good: bias and variance, cross-validation, tuning, learning curves, baselines and A/B tests.
Training a model is the easy part. Knowing whether it is any good, and whether it will stay good on data it has never seen, is where most real projects succeed or fail. This module gives you the tools to answer that honestly.
You will learn to diagnose underfitting and overfitting from training and validation error, estimate performance reliably with cross-validation, tune hyperparameters without fooling yourself, read learning curves to decide between more data and a better model, and finally test a model where it matters: on live users, with a properly designed A/B test.
Lessons
- ML-05.01 Bias, variance, underfitting and overfitting Underfitting means a model is too simple to capture the pattern and does badly on both training and validation data (high bias). Beginner 14 min
- ML-05.02 Cross-validation Cross-validation splits data into k folds, trains on k minus 1 and validates on the remaining fold, rotating so every fold is used once, and averages the results. Intermediate 12 min
- ML-05.03 Hyperparameter tuning Hyperparameters, such as tree depth or learning rate, are set before training rather than learned. Intermediate 12 min
- ML-05.04 Learning curves and baselines Always start with a simple baseline, such as predicting the average or a one-feature rule, so you know whether a complex model is actually helping. Intermediate 10 min
- ML-05.05 Offline vs online evaluation and A/B tests Offline metrics measured on historical data do not always translate into business impact. Intermediate 14 min