Machine Learning ML-05.03
Hyperparameter tuning
Grid search, random search and Bayesian optimization: how to find good settings fast, stop bad trials early, and never tune on the test set.
The big idea
A model has two kinds of numbers. ParametersValues learned from data during training, such as the weights of a linear model or the split points of a tree. are learned during training: the weights of logistic regression, the split points of a tree. HyperparametersSettings chosen before training that control how a model learns, such as the learning rate, tree depth, regularization strength or k in kNN. are chosen before training and control how learning happens: the learning rate, the depth of a tree, the regularization strength C, the number of neighbours k. Training cannot learn them for you, because they define what training does.
Tuning is a search: try settings, score each one on validation data, keep the best. It is like adjusting an oven's temperature and baking time. You could test every combination in 10-degree and 5-minute steps, or try a dozen random combinations, or bake one cake, look at it, and decide what to try next. Those are grid search, random search and Bayesian optimization.
How it works
Step 1: Grid search
Grid searchTrying every combination of a fixed list of values for each hyperparameter. picks a few values for each hyperparameter and tries every combination. It is simple and exhaustive, and its cost multiplies: 5 values for each of 4 hyperparameters is 5^4 = 625 training runs, times 5 cross-validation folds. Grid search is fine for one or two hyperparameters and hopeless beyond that.
Step 2: Random search
Random searchSampling hyperparameter combinations at random from chosen ranges or distributions, for a fixed budget of trials. samples each setting at random from a range, for a fixed budget of trials. It sounds sloppier but usually wins, for a simple reason: in most problems only one or two hyperparameters really matter, and you do not know which in advance. A 3 x 3 grid spends nine trials but tests only three values of each hyperparameter. Nine random trials test nine different values of each.
- 3 x 3 grid: three distinct values Nine trials, but only three learning rates (1/6, 1/2 and 5/6), because each is repeated for three values of an unimportant second setting. All three miss the narrow peak.
- Nine random trials: nine distinct values The same budget samples nine different learning rates. One of them lands at 0.64, right on the peak.
Two practical rules: sample scale-type hyperparameters (learning rate, C, regularization strength) on a log scale, from 0.0001 to 1 rather than uniformly, and set the ranges wide at first, then narrow them around what works.
Step 3: Bayesian optimization
Grid and random search never learn from their own results: trial 50 is chosen as blindly as trial 1. Bayesian optimizationA tuning strategy that fits a cheap model of score as a function of the hyperparameters and uses it to choose the most promising setting to try next. keeps a cheap surrogate model of "score as a function of hyperparameters", updated after every trial, and picks the next trial where the surrogate predicts a high score or is very uncertain. It balances exploiting known good regions with exploring unknown ones.
Tools such as Optuna, Hyperopt and the tuners built into cloud ML platforms implement this loop; Optuna's default sampler uses a tree-structured Parzen estimator, one kind of surrogate. Bayesian optimization shines when each trial is expensive (hours of GPU time) and the budget is tens of trials. When trials are cheap and you can run hundreds in parallel, random search is nearly as good and much simpler.
Step 4: Stop bad trials early
Many trials are obviously bad after a fraction of training: the loss diverges or lags far behind the leaders. Early stopping in tuningStopping an unpromising tuning trial before it finishes, based on its intermediate validation score, to spend the budget on better candidates. (also called pruning) cuts them off. Successive halving starts many trials with a small budget (few epochs or a data subset), keeps the best half, doubles their budget, and repeats. Hyperband runs several such brackets. These methods often cut tuning cost several-fold, because most of the budget goes to the candidates that deserve it.
Step 5: Tune on validation data, never on the test set
Every time you look at a score and change something, that data leaks a little into your choices. Tuning on the test set turns it into training data and makes the final number optimistic. The workflow:
- Split off a test set and lock it away.
- Tune with cross-validation (or a validation set) on the rest.
- Retrain the chosen setting on all non-test data.
- Evaluate once on the test set and report that number.
If you need an honest estimate on small data, use nested cross-validation ( Cross-validation ).
Try it
Try this:
- With a budget of 9, compare the tick marks: the grid tests three learning rates, random search tests nine. Which finds the green band?
- Raise the budget to 16 and then 25. The grid tests 4, then 5, distinct values. How often does it hit the band compared with random search? Press "New random draw" a few times.
- Imagine adding a third, irrelevant hyperparameter. With a grid of 3 values each the budget triples to 27 for no gain; random search spends the same 9 trials and still tests 9 learning rates.
Going deeper
Code: grid and random search on the same budget
import numpy as np
from scipy.stats import loguniform
from sklearn.datasets import load_digits
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, train_test_split
from sklearn.svm import SVC
X, y = load_digits(return_X_y=True) # 8x8 handwritten digits, built in
X = X / 16.0
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
# Same budget, 16 settings each, 3-fold CV on the training data only
grid = GridSearchCV(SVC(), {"C": [0.1, 1, 10, 100], "gamma": [0.001, 0.01, 0.1, 1]}, cv=3).fit(Xtr, ytr)
rand = RandomizedSearchCV(SVC(), {"C": loguniform(0.1, 100), "gamma": loguniform(0.001, 1)},
n_iter=16, cv=3, random_state=0).fit(Xtr, ytr)
for name, s in [("grid", grid), ("random", rand)]:
p = s.best_params_
print(f"{name:<7} best CV {s.best_score_:.4f} C {p['C']:.3g} gamma {p['gamma']:.3g} "
f"distinct gamma values tried: {len(set(np.round(s.cv_results_['param_gamma'].data.astype(float), 6)))}")
# The test set is touched once, at the very end
print(f"\nfinal test accuracy (random search winner): {rand.score(Xte, yte):.4f}")
# Which hyperparameter mattered? Spread of CV score across gamma vs across C in the grid
res = grid.cv_results_
scores = res["mean_test_score"].reshape(4, 4) # rows: C, cols: gamma
print("\nmean CV score by gamma:", scores.mean(axis=0).round(3))
print("mean CV score by C: ", scores.mean(axis=1).round(3))Output
grid best CV 0.9833 C 1 gamma 0.1 distinct gamma values tried: 4
random best CV 0.9865 C 2.42 gamma 0.22 distinct gamma values tried: 16
final test accuracy (random search winner): 0.9944
mean CV score by gamma: [0.625 0.799 0.973 0.821]
mean CV score by C: [0.471 0.804 0.968 0.976] Both searches get 16 trials of an RBF SVM on handwritten digits. The grid tries 4 values of gamma; random search, sampling C and gamma on a log scale, tries 16, and finds a slightly better cross-validated score (0.9865 against 0.9833). The test set is used once, at the end. The last two lines show why tuning matters at all: averaged over the grid, a poor gamma gives 0.625 and a good one 0.973.
What to tune first
Not all hyperparameters deserve equal effort. Typically the learning rate for neural networks and boosting, the regularization strength for linear models, and depth and number of trees for tree ensembles matter most. Tune those first with wide log ranges, then refine. Fix the random seed while comparing settings, or average over seeds, so you compare settings rather than luck.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Manual tuning | Early exploration with strong intuition; one or two knobs | Many knobs; it is slow and hard to reproduce |
| Grid search | One or two hyperparameters with few sensible values | Three or more hyperparameters: cost explodes |
| Random search | Many hyperparameters, cheap parallel trials | Very expensive trials where every one counts |
| Bayesian optimization | Expensive trials and a budget of tens of trials | Massively parallel cheap trials; extra machinery |
| Successive halving / Hyperband | Trials whose quality shows early (learning curves) | Settings that only pay off late in training |
Common mistakes
- Tuning on the test set, then reporting the test score as if it were unbiased.
- Searching learning rates or C on a linear scale instead of a log scale.
- Using a grid over many hyperparameters and running out of budget.
- Leaving the search ranges too narrow, so the best value sits on the edge of the range. If it does, widen it.
- Comparing settings with different random seeds and mistaking noise for improvement.
In the wild
Automated tuning is built into most ML platforms: cloud services offer managed hyperparameter tuning jobs with Bayesian search and early stopping, and open-source libraries such as Optuna, Ray Tune and KerasTuner provide the same strategies. Large training runs use these methods on small proxy models first, then transfer the best settings, because a single full-size trial can cost days of compute.
Check yourself
-
Which of these is a hyperparameter rather than a parameter?
Show answer
B. The maximum depth of a decision tree Max depth is set before training and controls how the tree grows. Weights, biases and split thresholds are learned from data.
-
With 25 trials, a 5 x 5 grid tests how many distinct values of the first hyperparameter? And random search?
Show answer
B. 5 and 25 The grid repeats each of its 5 values 5 times. Random search samples a new value on every trial.
-
Each trial trains a large model for 6 hours and you can afford 30 trials. Which strategy fits best?
Show answer
B. Bayesian optimization, ideally with early stopping of weak trials When trials are expensive and few, learning from previous results to choose the next one pays off, and pruning bad trials saves even more.
-
Your best learning rate is 0.1, the largest value in your search range. What should you do?
Show answer
B. Extend the range upward; the optimum may lie beyond it A best value on the boundary suggests the range was too narrow. Widen it and search again.
Recap
- Hyperparameters are set before training and control how the model learns; parameters are learned.
- Grid search is exhaustive but its cost multiplies with each hyperparameter.
- Random search tests far more distinct values per hyperparameter for the same budget and usually wins.
- Bayesian optimization learns from past trials; early stopping cuts weak trials and saves budget.
- Tune on validation data or cross-validation; touch the test set once, at the end.
Next up
- From notebook to production learning path