XGBoost, LightGBM and CatBoost in practice

The big idea

You now know gradient boosting is a loop of small trees fitting residuals. You will almost never write that loop yourself. Three open-source libraries do it with decades of engineering on top: XGBoostAn open-source gradient boosting library known for regularized objectives, sparsity-aware splits and scalable distributed training., LightGBMA gradient boosting library from Microsoft that grows trees leaf-wise and uses histogram-based splits, built for speed on large datasets. and CatBoostA gradient boosting library from Yandex with built-in handling of categorical features through ordered target statistics and symmetric trees.. scikit-learn's HistGradientBoostingClassifier, used in this lesson's code, follows the LightGBM design.

Like three brands of professional camera, they take the same picture with different ergonomics. What matters is knowing which handful of settings to touch, how to stop at the right number of trees, how to feed in categories, and which mistakes quietly ruin results.

How it works

Step 1: Bin the features into histograms

Exact split search sorts every feature at every node, which is slow on millions of rows. All three libraries first binReplacing each continuous feature by the index of the bucket (bin) it falls in, typically 255 buckets, so split search scans bins instead of every distinct value. each feature into at most about 255 buckets. A split search then scans 255 bin boundaries instead of millions of values, using gradient sums per bin. The loss of precision almost never matters, and training becomes many times faster and lighter on memory.

Step 2: Know how each library grows trees

• XGBoost grows depth-wise by default, limited by max_depth (default 6), and adds explicit L1 and L2 penalties on leaf values to its objective.
• LightGBM grows leaf-wise, limited by num_leaves (default 31). Set max_depth or min_child_samples too on small datasets.
• CatBoost grows symmetric (oblivious) treesTrees in which every node at the same depth uses the same split, so the tree is a lookup table of 2^depth cells; fast to evaluate and naturally regularized., where every node at one level uses the same split. They are fast to evaluate and resistant to overfitting.

Step 3: Turn the few knobs that matter

The same handful of knobs appears in every library, under different names (XGBoost / LightGBM / CatBoost):

• Number of trees: n_estimators / n_estimators / iterations.
• Learning rate: learning_rate in all three.
• Tree size: max_depth / num_leaves / depth.
• Minimum leaf size: min_child_weight / min_child_samples / min_data_in_leaf.
• Row sampling: subsample / bagging_fraction / subsample.
• Column sampling: colsample_bytree / feature_fraction / rsm.
• L2 penalty on leaf values: reg_lambda / lambda_l2 / l2_leaf_reg.

A practical recipe: fix the learning rate (0.05 to 0.1), set a large number of trees and let early stopping choose; tune tree size and minimum leaf size; then row and column subsampling (0.6 to 0.9); then the L2 penalty. Lower the learning rate at the end for a small final gain. Random or Bayesian search over these ( Hyperparameter tuning ) beats hand tuning.

Step 4: Always use early stopping

Pass a validation set and a patience (say 50 rounds). Training stops when validation loss has not improved for that many rounds, and the model keeps the best round. In the code below, learning rate 0.3 stops after 66 trees (test AUC 0.7260), 0.1 after 93 (0.7350) and 0.03 after 314 (0.7407): smaller steps, more trees, slightly better model.

Step 5: Give categories the right treatment

Converting a category such as a merchant id to an integer pretends merchant 17 is "between" 16 and 18. Better options:

• Native categorical splits (LightGBM, CatBoost, XGBoost with enable_categorical, scikit-learn's histogram GBM): the split groups categories into two sets. In the code, treating the 200-merchant id as categorical lifts test AUC from 0.7395 to 0.7881.
• Target encoding: replace each category by the average label for that category. Powerful for very high cardinality, and dangerous, as the next figure shows.

The fix is to compute each row's encoding without its own label: out-of-fold means (as in the code), or CatBoost's ordered target statisticsCatBoost's target encoding that, for each row, uses only the labels of rows that come before it in a random ordering, so no row sees its own label., which use only rows earlier in a random permutation. Smoothing towards the global mean helps rare categories.

Going deeper

Code: early stopping, categoricals and a leak

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import KFold, train_test_split

# Synthetic "transactions": numbers plus a merchant id with 200 categories
rng = np.random.default_rng(0)
n = 20_000
merchant = rng.integers(0, 200, n)
risk = rng.normal(0, 1, 200)                       # each merchant has its own fraud risk
amount = rng.lognormal(3, 1, n)
hour = rng.integers(0, 24, n)
logit = -5 + 1.5 * risk[merchant] + 0.6 * np.log(amount) + 1.0 * (hour < 6)
y = rng.random(n) < 1 / (1 + np.exp(-logit))
X = pd.DataFrame({"merchant": merchant, "amount": amount, "hour": hour})
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
print(f"fraud rate {y.mean():.3f}")

# 1. Early stopping picks the number of trees on an internal validation split
for lr in [0.3, 0.1, 0.03]:
    m = HistGradientBoostingClassifier(learning_rate=lr, max_iter=2000, early_stopping=True,
                                       validation_fraction=0.15, n_iter_no_change=30, random_state=0).fit(Xtr, ytr)
    print(f"lr {lr:<4}  trees used {m.n_iter_:>4}  test AUC {roc_auc_score(yte, m.predict_proba(Xte)[:, 1]):.4f}")

# 2. Treating merchant id as a number vs as a native categorical feature
for cat in [None, ["merchant"]]:
    Xc_tr, Xc_te = Xtr.copy(), Xte.copy()
    if cat:
        Xc_tr["merchant"] = Xc_tr["merchant"].astype("category")
        Xc_te["merchant"] = pd.Categorical(Xc_te["merchant"], categories=Xc_tr["merchant"].cat.categories)
    m = HistGradientBoostingClassifier(learning_rate=0.1, categorical_features="from_dtype", random_state=0).fit(Xc_tr, ytr)
    print(f"merchant as {'category' if cat else 'integer '}  test AUC {roc_auc_score(yte, m.predict_proba(Xc_te)[:, 1]):.4f}")

# 3. Target encoding pitfall: a device id with 5,000 values and NO real signal.
#    Encoding it with means that include each row's own label leaks the label.
dev_tr = rng.integers(0, 5000, len(Xtr))
dev_te = rng.integers(0, 5000, len(Xte))
df = pd.DataFrame({"d": dev_tr, "y": ytr.astype(float)})
prior = df.y.mean()
naive = df.d.map(df.groupby("d").y.mean()).values
oof = np.zeros(len(df))
for a, b in KFold(5, shuffle=True, random_state=0).split(df):
    oof[b] = df.d.iloc[b].map(df.iloc[a].groupby("d").y.mean()).fillna(prior).values
test_enc = pd.Series(dev_te).map(df.groupby("d").y.mean()).fillna(prior).values
cat_tr = Xtr.assign(merchant=Xtr["merchant"].astype("category"))
cat_te = Xte.assign(merchant=pd.Categorical(Xte["merchant"], categories=cat_tr["merchant"].cat.categories))
for name, enc in [("naive target encoding ", naive), ("out-of-fold encoding  ", oof)]:
    m = HistGradientBoostingClassifier(categorical_features="from_dtype", random_state=0).fit(cat_tr.assign(device=enc), ytr)
    a = roc_auc_score(ytr, m.predict_proba(cat_tr.assign(device=enc))[:, 1])
    b = roc_auc_score(yte, m.predict_proba(cat_te.assign(device=test_enc))[:, 1])
    print(f"{name}: train AUC {a:.4f}, test AUC {b:.4f}")

fraud rate 0.097
lr 0.3   trees used   66  test AUC 0.7260
lr 0.1   trees used   93  test AUC 0.7350
lr 0.03  trees used  314  test AUC 0.7407
merchant as integer   test AUC 0.7395
merchant as category  test AUC 0.7881
naive target encoding : train AUC 0.9851, test AUC 0.5371
out-of-fold encoding  : train AUC 0.9178, test AUC 0.7815

The same experiments in the three libraries look almost identical: XGBClassifier(..., early_stopping_rounds=50).fit(X, y, eval_set=[(Xv, yv)]), LGBMClassifier().fit(X, y, eval_set=..., callbacks=[lightgbm.early_stopping(50)]), and CatBoostClassifier(cat_features=[...]).fit(X, y, eval_set=..., early_stopping_rounds=50).

GPU training and scale

All three train on GPUs (device="cuda" in XGBoost, device_type="gpu" in LightGBM, task_type="GPU" in CatBoost), which helps mostly on millions of rows and deep or many trees. On a few hundred thousand rows a multi-core CPU is often just as fast. All three also support distributed training across machines.

Pitfalls that cost real projects

• Early stopping on the test set: the stopping round is a tuned hyperparameter; use a separate validation split.
• Class imbalance: prefer reporting PR-AUC and calibrating, or use scale_pos_weight carefully; weights distort predicted probabilities ( Probability calibration ).
• Category mismatch at serving: unseen categories, different encodings or a different category order between training and serving silently change predictions. Version the encoders with the model.
• Monotonic constraints: when the business requires that, say, a higher income never lowers a credit score, all three libraries can enforce monotonic constraints per feature.

Trade-offs

Common mistakes

• Training a fixed 1,000 trees without early stopping.
• Feeding category ids as plain integers.
• Target encoding with in-sample means, then celebrating a near-perfect training score.
• Tuning ten hyperparameters at once by grid search instead of a staged random search.
• Leaving LightGBM's num_leaves high on a dataset of a few thousand rows.

In the wild

Check yourself

Recap