Machine Learning ML-02.05
Feature engineering
Turn raw records into model inputs: scaling, one-hot, target and hash encoding, binning, interactions, date parts, text counts and window aggregates.
The big idea
A card transaction arrives: 400 euros, an electronics shop, 23:55 on a Friday, customer "ana". A fraud model cannot do anything with "ana" or a timestamp string. It needs numbers, and not just any numbers: numbers that make the pattern easy to see. "This is 14.5 times her usual amount, at night" is a far more useful input than the raw fields, even though it is computed entirely from them.
Feature engineeringTurning raw data into model inputs (features) that make the patterns you care about easy for a model to find. is that translation. Think of a doctor reading a chart. Raw measurements matter, but what really helps is derived information: the change since the last visit, the reading relative to the patient's normal, the ratio of two values. Good features carry that kind of domain knowledge into the model, and they often matter more than the choice of algorithm.
How it works
Step 1: Scale numeric features
Many models care about the size of numbers. k-nearest neighbours measures distances, so a feature in the thousands drowns out one between 0 and 1. Gradient descent on linear models and neural networks also converges faster when features have similar ranges. Three common scalersRescaling a numeric feature to a common range or spread, such as mean 0 and standard deviation 1, so no feature dominates by its units alone.:
- Standard scaling: subtract the mean, divide by the standard deviation, . The default for most models.
- Min-max scaling: squeeze into 0 to 1, . Useful when a bounded range matters, such as pixel values. One extreme value squashes everything else together.
- Robust scaling: subtract the median, divide by the interquartile range. The bulk of the data keeps a sensible scale even when there are big outliers.
Tree-based models split on thresholds, so they do not need scaling at all. For heavily skewed values such as money or counts, a log transform before scaling often helps more than the scaler choice.
Step 2: Encode categories
- One-hot encodingTurning a category with k values into k columns of 0s and 1s, with a 1 in the column for the row's category.: one 0/1 column per category. No fake order, works with every model. With thousands of categories it creates thousands of mostly-zero columns.
- Ordinal encoding: map ordered categories to 0, 1, 2. Right for true ordinal data (poor, good, excellent); wrong for plain categories, because it invents an order.
- Target encodingReplacing each category with the average target value for that category, smoothed toward the overall average, computed without using each row's own label.: replace each category with the average label for that category, such as the churn rate per city. One column, however many categories. Two dangers: rare categories get extreme values from a handful of rows, so you smooth toward the overall average; and computing the average with a row's own label leaks the answer, so it must be fitted on training data only, ideally with cross-fitting.
- Feature hashingMapping category values to a fixed number of columns with a hash function. Handles huge or unbounded vocabularies; unrelated values sometimes collide.: run each value through a hash function into a fixed number of columns. Handles millions of values, including ones never seen in training, in constant memory. The price is occasional collisions, where two unrelated values share a column.
The smoothed target encoding below uses
where is the count for category , its mean label, the overall mean and the smoothing strength. Nantes has only 2 rows, both churned. Unsmoothed, it would get an encoding of 1.00; smoothed with , it gets , pulled back toward the 0.35 average because two rows are weak evidence.
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler, OneHotEncoder
from sklearn.feature_extraction import FeatureHasher
income = np.array([[28], [35], [41], [52], [60], [950]]) # one huge value (in k)
for scaler in (StandardScaler(), MinMaxScaler(), RobustScaler()):
z = scaler.fit_transform(income).ravel().round(2)
print(f"{type(scaler).__name__:<15}", z)
city = np.array([["Lyon"], ["Paris"], ["Lyon"], ["Nantes"], ["Paris"], ["Paris"]])
oh = OneHotEncoder(handle_unknown="ignore").fit(city)
print("\none-hot columns:", oh.get_feature_names_out().tolist())
print("Lyon ->", oh.transform([["Lyon"]]).toarray()[0], " Lille (unseen) ->", oh.transform([["Lille"]]).toarray()[0])
# Target encoding: replace a category by its average label, smoothed toward the
# overall average so a city seen twice cannot get an extreme value (m = smoothing).
df = pd.DataFrame({"city": ["Lyon"] * 40 + ["Paris"] * 50 + ["Nantes"] * 2,
"churned": [1] * 10 + [0] * 30 + [1] * 20 + [0] * 30 + [1, 1]})
prior, m = df["churned"].mean(), 10
g = df.groupby("city")["churned"].agg(["count", "mean"])
g["encoded"] = (g["count"] * g["mean"] + m * prior) / (g["count"] + m)
print("\n" + g.round(2).to_string())
print(f"overall churn rate {prior:.2f}")
# Hashing: millions of possible values squeezed into a fixed number of columns
h = FeatureHasher(n_features=8, input_type="string")
print("\nhashed user ids:\n", h.transform([["user_9812"], ["user_17"], ["user_5550331"]]).toarray())Output
StandardScaler [-0.49 -0.47 -0.45 -0.42 -0.4 2.23]
MinMaxScaler [0. 0.01 0.01 0.03 0.03 1. ]
RobustScaler [-0.86 -0.53 -0.26 0.26 0.63 42.02]
one-hot columns: ['x0_Lyon', 'x0_Nantes', 'x0_Paris']
Lyon -> [1. 0. 0.] Lille (unseen) -> [0. 0. 0.]
count mean encoded
city
Lyon 40 0.25 0.27
Nantes 2 1.00 0.46
Paris 50 0.40 0.39
overall churn rate 0.35
hashed user ids:
[[ 0. 0. 0. 0. 1. 0. 0. 0.]
[ 1. 0. 0. 0. 0. 0. 0. 0.]
[ 0. -1. 0. 0. 0. 0. 0. 0.]] Look at the scalers' output: one income of 950 makes standard scaling squash the other five into a narrow band around -0.45, and min-max puts them all between 0 and 0.03. Robust scaling keeps them spread from -0.86 to 0.63 and lets the outlier sit far away on its own.
Step 3: Binning and interactions
BinningTurning a numeric feature into ranges, such as age 18 to 25, 26 to 40 and so on, then treating each range as a category. turns a number into ranges: age 18 to 25, 26 to 40, 41 to 65, 65 and over. It helps linear models capture effects that are not straight lines, such as risk that is high for the young and the old. It throws away detail within each bin, so tree models rarely need it.
An interactionA feature made by combining two or more features, such as their product or ratio, to capture an effect that depends on both together. feature combines two features: price per square metre (price divided by size), or "night and electronics". Linear models can only add up the effects of individual features, so an effect that exists only for a combination must be handed to them as its own feature. Trees and neural networks can find some interactions themselves, but a well-chosen ratio still often helps.
Step 4: Date and time features
A raw timestamp is a huge, ever-growing number that means little to a model. Break it into parts that repeat: hour of day, day of week, month, is weekend, is holiday, days since the account was opened. For a value that wraps around, like hour, 23 and 0 are neighbours but look far apart as numbers. Encoding the hour as a point on a circle, and , fixes that.
Step 5: Text features
The simplest text features are counts: bag of wordsRepresenting text as counts of each word in a vocabulary, ignoring word order. counts each word, and TF-IDF downweights words that appear everywhere. They are covered in Bag of words and TF-IDF . Simple hand-made features help too: length, number of exclamation marks, whether it contains a link. Modern systems often use embeddingsA dense vector of numbers produced by a trained model that represents the meaning of a text, image or item. from a pretrained model, a vector that captures meaning, as features for a downstream model.
Step 6: Aggregates over time windows
For behaviour data, the strongest features usually describe a history: number of purchases in the last 7 days, average basket size over 30 days, minutes since the last login, this amount compared with the user's own average. They turn "what happened" into "how unusual is this for this person".
import pandas as pd
tx = pd.DataFrame({
"user": ["ana", "ana", "ana", "ben", "ana", "ben"],
"ts": pd.to_datetime(["2026-03-01 09:10", "2026-03-03 22:40", "2026-03-06 23:55",
"2026-03-06 12:00", "2026-03-09 08:30", "2026-03-10 13:15"]),
"amount": [20.0, 35.0, 400.0, 15.0, 22.0, 18.0],
}).sort_values("ts")
# Date and time parts a model can use
tx["hour"] = tx["ts"].dt.hour
tx["weekday"] = tx["ts"].dt.dayofweek # 0 = Monday
tx["is_night"] = tx["hour"].isin([23, 0, 1, 2, 3, 4, 5]).astype(int)
# Window aggregates per user over the previous 7 days, NOT including this row,
# so each feature uses only what was known before the transaction happened.
def past_7d(g):
w = g.set_index("ts")["amount"].rolling("7D", closed="left") # [t - 7 days, t)
return pd.DataFrame({"n_7d": w.count().fillna(0).values,
"sum_7d": w.sum().fillna(0).values}, index=g.index)
tx = tx.join(tx.groupby("user", group_keys=False).apply(past_7d))
tx["amount_vs_avg"] = (tx["amount"] / (tx["sum_7d"] / tx["n_7d"])).round(1)
print(tx.drop(columns="ts").to_string(index=False))Output
user amount hour weekday is_night n_7d sum_7d amount_vs_avg
ana 20.0 9 6 0 0.0 0.0 NaN
ana 35.0 22 1 0 1.0 20.0 1.8
ben 15.0 12 4 0 0.0 0.0 NaN
ana 400.0 23 4 1 2.0 55.0 14.5
ana 22.0 8 0 0 2.0 435.0 0.1
ben 18.0 13 1 0 1.0 15.0 1.2 The 400 euro purchase gets amount_vs_avg = 14.5: 400 divided by her average of 27.5 over the previous 7 days. The window is closed="left", which means it covers the 7 days before each transaction and never includes the transaction itself. Getting that boundary wrong is one of the most common sources of leakage: a feature that includes the current row, or rows from after it, cannot be computed at prediction time.
Going deeper
Fit on training data only. Every feature step that learns something from data, the mean for scaling, the categories for one-hot, the averages for target encoding, is part of the model. Fit it on the training set and reuse the fitted version for validation, test and serving. A scikit-learn Pipeline with a ColumnTransformer does this for you and saves the steps with the model.
Unseen categories. In production a new city or product will appear. Decide in advance: one-hot with handle_unknown="ignore" (all zeros, as Lille gets above), hashing (it always has a column), or a dedicated "other" bucket for rare and unseen values.
Online and offline consistency. A 7-day aggregate is easy to compute in a batch job over a table and much harder to compute in under 50 ms when a transaction arrives. If the two computations differ even slightly, you get training-serving skew. Feature stores, covered in Feature stores and training-serving skew , exist to compute such features once and serve them consistently.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| One-hot | Few to a few hundred categories, any model | Thousands of categories: wide, sparse, slow |
| Target encoding | High-cardinality categories with plenty of rows each | Done without smoothing or cross-fitting, it leaks and overfits rare categories |
| Hashing | Huge or open-ended vocabularies, streaming data | You need to explain which value drove a prediction; collisions blur meaning |
| Standard vs robust scaling | Standard for roughly bell-shaped data; robust when outliers are present | Neither is needed for tree models |
| Window aggregates | Behaviour data where history defines what is normal | You cannot compute them identically and fast enough at serving time |
Common mistakes
- Fitting scalers or encoders on the full dataset before splitting, which leaks test statistics.
- Ordinal-encoding plain categories (Lyon = 0, Paris = 1, Nantes = 2), inventing an order.
- Target-encoding with each row's own label included, which hands the model the answer.
- Window features that include the current event or anything after it.
- Feeding a raw timestamp as a number, or hour of day without handling that 23 and 0 are neighbours.
In the wild
Fraud and risk teams rely heavily on velocity features: counts and sums of a card's or account's activity over the last minutes, hours and days, compared with that account's own history. Recommender and ads systems use enormous categorical features such as user and item ids, and handle them with hashing or learned embeddings. Feature stores became a standard part of ML platforms at large tech companies precisely because these windowed features must be computed the same way for training and for real-time serving.
Check yourself
-
You train a k-nearest-neighbours model on income (thousands) and number of children (0 to 5) without scaling. What happens?
Show answer
B. Distances are dominated by income, so the children feature barely matters k-NN measures distance. A difference of 10,000 in income swamps a difference of 3 children unless both are put on comparable scales.
-
A city seen in only 3 training rows, all churned, gets target encoding 1.0. What is the fix?
Show answer
B. Smooth toward the overall average, so rare categories get moderate values Three rows are weak evidence. Smoothing blends the category mean with the global mean, weighted by how many rows the category has.
-
Which window feature is safe to use for predicting whether a transaction is fraud?
Show answer
B. Number of transactions in the 7 days before this one, excluding it Only information available before the moment of prediction is allowed. The others use the current event, future events or the outcome itself.
-
Your linear model misses that house prices rise steeply with size in the city centre but barely in the suburbs. What feature helps most?
Show answer
B. An interaction such as size x is_centre A linear model adds up separate effects. An effect that depends on two features together needs an interaction term.
Recap
- Feature engineering turns raw fields into numbers that make patterns easy to learn.
- Scale numeric features for distance- and gradient-based models; standard is the default, robust handles outliers, trees need none.
- Encode categories with one-hot, ordinal only for true orders, smoothed target encoding or hashing for many values.
- Bins, interactions, date parts, text counts and window aggregates add domain knowledge a model would struggle to find.
- Fit every feature step on training data, and make sure each feature uses only information available at prediction time.
Next up
- From notebook to production learning path