Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Data: the most important part

Machine Learning ML-02.05

Feature engineering

Turn raw records into model inputs: scaling, one-hot, target and hash encoding, binning, interactions, date parts, text counts and window aggregates.

Intermediate 18 min Last reviewed

The big idea

A card transaction arrives: 400 euros, an electronics shop, 23:55 on a Friday, customer "ana". A fraud model cannot do anything with "ana" or a timestamp string. It needs numbers, and not just any numbers: numbers that make the pattern easy to see. "This is 14.5 times her usual amount, at night" is a far more useful input than the raw fields, even though it is computed entirely from them.

Feature engineeringTurning raw data into model inputs (features) that make the patterns you care about easy for a model to find. is that translation. Think of a doctor reading a chart. Raw measurements matter, but what really helps is derived information: the change since the last visit, the reading relative to the patient's normal, the ratio of two values. Good features carry that kind of domain knowledge into the model, and they often matter more than the choice of algorithm.

How it works

A raw transaction becomes a feature vector On the left, four raw fields of one card transaction: amount 400, merchant type electronics, timestamp Friday 23:55, and the user's history. Arrows labelled log, one-hot, date parts and 7-day window turn them into a column of numeric features on the right: log amount 2.60, two one-hot merchant columns, hour 23, is night 1, weekday 4, transactions in the last 7 days 2, and amount versus 7-day average 14.5, which glows as the most telling feature. raw transaction feature vector log one-hot date parts 7-day window amount 400.00 EUR merchant electronics time 2026-03-06 23:55 user history ana, past 7 days log_amount = 2.60 merch_electronics = 1 merch_grocery = 0 hour = 23 is_night = 1 weekday = 4 n_7d = 2 amount_vs_avg_7d = 14.5 Figure 1. One raw row becomes a numeric feature vector. The glowing feature, the amount compared with the user's own recent average, is computed from history and is often the most predictive.

Step 1: Scale numeric features

Many models care about the size of numbers. k-nearest neighbours measures distances, so a feature in the thousands drowns out one between 0 and 1. Gradient descent on linear models and neural networks also converges faster when features have similar ranges. Three common scalersRescaling a numeric feature to a common range or spread, such as mean 0 and standard deviation 1, so no feature dominates by its units alone.:

  • Standard scaling: subtract the mean, divide by the standard deviation, . The default for most models.
  • Min-max scaling: squeeze into 0 to 1, . Useful when a bounded range matters, such as pixel values. One extreme value squashes everything else together.
  • Robust scaling: subtract the median, divide by the interquartile range. The bulk of the data keeps a sensible scale even when there are big outliers.

Tree-based models split on thresholds, so they do not need scaling at all. For heavily skewed values such as money or counts, a log transform before scaling often helps more than the scaler choice.

Step 2: Encode categories

  • One-hot encodingTurning a category with k values into k columns of 0s and 1s, with a 1 in the column for the row's category.: one 0/1 column per category. No fake order, works with every model. With thousands of categories it creates thousands of mostly-zero columns.
  • Ordinal encoding: map ordered categories to 0, 1, 2. Right for true ordinal data (poor, good, excellent); wrong for plain categories, because it invents an order.
  • Target encodingReplacing each category with the average target value for that category, smoothed toward the overall average, computed without using each row's own label.: replace each category with the average label for that category, such as the churn rate per city. One column, however many categories. Two dangers: rare categories get extreme values from a handful of rows, so you smooth toward the overall average; and computing the average with a row's own label leaks the answer, so it must be fitted on training data only, ideally with cross-fitting.
  • Feature hashingMapping category values to a fixed number of columns with a hash function. Handles huge or unbounded vocabularies; unrelated values sometimes collide.: run each value through a hash function into a fixed number of columns. Handles millions of values, including ones never seen in training, in constant memory. The price is occasional collisions, where two unrelated values share a column.

The smoothed target encoding below uses

where is the count for category , its mean label, the overall mean and the smoothing strength. Nantes has only 2 rows, both churned. Unsmoothed, it would get an encoding of 1.00; smoothed with , it gets , pulled back toward the 0.35 average because two rows are weak evidence.

encode.pyruns with python3 + numpy + pandas + scikit-learn
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler, OneHotEncoder
from sklearn.feature_extraction import FeatureHasher

income = np.array([[28], [35], [41], [52], [60], [950]])   # one huge value (in k)
for scaler in (StandardScaler(), MinMaxScaler(), RobustScaler()):
    z = scaler.fit_transform(income).ravel().round(2)
    print(f"{type(scaler).__name__:<15}", z)

city = np.array([["Lyon"], ["Paris"], ["Lyon"], ["Nantes"], ["Paris"], ["Paris"]])
oh = OneHotEncoder(handle_unknown="ignore").fit(city)
print("\none-hot columns:", oh.get_feature_names_out().tolist())
print("Lyon ->", oh.transform([["Lyon"]]).toarray()[0], " Lille (unseen) ->", oh.transform([["Lille"]]).toarray()[0])

# Target encoding: replace a category by its average label, smoothed toward the
# overall average so a city seen twice cannot get an extreme value (m = smoothing).
df = pd.DataFrame({"city": ["Lyon"] * 40 + ["Paris"] * 50 + ["Nantes"] * 2,
                   "churned": [1] * 10 + [0] * 30 + [1] * 20 + [0] * 30 + [1, 1]})
prior, m = df["churned"].mean(), 10
g = df.groupby("city")["churned"].agg(["count", "mean"])
g["encoded"] = (g["count"] * g["mean"] + m * prior) / (g["count"] + m)
print("\n" + g.round(2).to_string())
print(f"overall churn rate {prior:.2f}")

# Hashing: millions of possible values squeezed into a fixed number of columns
h = FeatureHasher(n_features=8, input_type="string")
print("\nhashed user ids:\n", h.transform([["user_9812"], ["user_17"], ["user_5550331"]]).toarray())

Output

StandardScaler  [-0.49 -0.47 -0.45 -0.42 -0.4   2.23]
MinMaxScaler    [0.   0.01 0.01 0.03 0.03 1.  ]
RobustScaler    [-0.86 -0.53 -0.26  0.26  0.63 42.02]

one-hot columns: ['x0_Lyon', 'x0_Nantes', 'x0_Paris']
Lyon -> [1. 0. 0.]  Lille (unseen) -> [0. 0. 0.]

        count  mean  encoded
city                        
Lyon       40  0.25     0.27
Nantes      2  1.00     0.46
Paris      50  0.40     0.39
overall churn rate 0.35

hashed user ids:
 [[ 0.  0.  0.  0.  1.  0.  0.  0.]
 [ 1.  0.  0.  0.  0.  0.  0.  0.]
 [ 0. -1.  0.  0.  0.  0.  0.  0.]]

Look at the scalers' output: one income of 950 makes standard scaling squash the other five into a narrow band around -0.45, and min-max puts them all between 0 and 0.03. Robust scaling keeps them spread from -0.86 to 0.63 and lets the outlier sit far away on its own.

Step 3: Binning and interactions

BinningTurning a numeric feature into ranges, such as age 18 to 25, 26 to 40 and so on, then treating each range as a category. turns a number into ranges: age 18 to 25, 26 to 40, 41 to 65, 65 and over. It helps linear models capture effects that are not straight lines, such as risk that is high for the young and the old. It throws away detail within each bin, so tree models rarely need it.

An interactionA feature made by combining two or more features, such as their product or ratio, to capture an effect that depends on both together. feature combines two features: price per square metre (price divided by size), or "night and electronics". Linear models can only add up the effects of individual features, so an effect that exists only for a combination must be handed to them as its own feature. Trees and neural networks can find some interactions themselves, but a well-chosen ratio still often helps.

Step 4: Date and time features

A raw timestamp is a huge, ever-growing number that means little to a model. Break it into parts that repeat: hour of day, day of week, month, is weekend, is holiday, days since the account was opened. For a value that wraps around, like hour, 23 and 0 are neighbours but look far apart as numbers. Encoding the hour as a point on a circle, and , fixes that.

Step 5: Text features

The simplest text features are counts: bag of wordsRepresenting text as counts of each word in a vocabulary, ignoring word order. counts each word, and TF-IDF downweights words that appear everywhere. They are covered in Bag of words and TF-IDF . Simple hand-made features help too: length, number of exclamation marks, whether it contains a link. Modern systems often use embeddingsA dense vector of numbers produced by a trained model that represents the meaning of a text, image or item. from a pretrained model, a vector that captures meaning, as features for a downstream model.

Step 6: Aggregates over time windows

For behaviour data, the strongest features usually describe a history: number of purchases in the last 7 days, average basket size over 30 days, minutes since the last login, this amount compared with the user's own average. They turn "what happened" into "how unusual is this for this person".

time.pyruns with python3 + pandas
import pandas as pd

tx = pd.DataFrame({
    "user": ["ana", "ana", "ana", "ben", "ana", "ben"],
    "ts": pd.to_datetime(["2026-03-01 09:10", "2026-03-03 22:40", "2026-03-06 23:55",
                          "2026-03-06 12:00", "2026-03-09 08:30", "2026-03-10 13:15"]),
    "amount": [20.0, 35.0, 400.0, 15.0, 22.0, 18.0],
}).sort_values("ts")

# Date and time parts a model can use
tx["hour"] = tx["ts"].dt.hour
tx["weekday"] = tx["ts"].dt.dayofweek            # 0 = Monday
tx["is_night"] = tx["hour"].isin([23, 0, 1, 2, 3, 4, 5]).astype(int)

# Window aggregates per user over the previous 7 days, NOT including this row,
# so each feature uses only what was known before the transaction happened.
def past_7d(g):
    w = g.set_index("ts")["amount"].rolling("7D", closed="left")  # [t - 7 days, t)
    return pd.DataFrame({"n_7d": w.count().fillna(0).values,
                         "sum_7d": w.sum().fillna(0).values}, index=g.index)

tx = tx.join(tx.groupby("user", group_keys=False).apply(past_7d))
tx["amount_vs_avg"] = (tx["amount"] / (tx["sum_7d"] / tx["n_7d"])).round(1)
print(tx.drop(columns="ts").to_string(index=False))

Output

user  amount  hour  weekday  is_night  n_7d  sum_7d  amount_vs_avg
 ana    20.0     9        6         0   0.0     0.0            NaN
 ana    35.0    22        1         0   1.0    20.0            1.8
 ben    15.0    12        4         0   0.0     0.0            NaN
 ana   400.0    23        4         1   2.0    55.0           14.5
 ana    22.0     8        0         0   2.0   435.0            0.1
 ben    18.0    13        1         0   1.0    15.0            1.2

The 400 euro purchase gets amount_vs_avg = 14.5: 400 divided by her average of 27.5 over the previous 7 days. The window is closed="left", which means it covers the 7 days before each transaction and never includes the transaction itself. Getting that boundary wrong is one of the most common sources of leakage: a feature that includes the current row, or rows from after it, cannot be computed at prediction time.

Going deeper

Fit on training data only. Every feature step that learns something from data, the mean for scaling, the categories for one-hot, the averages for target encoding, is part of the model. Fit it on the training set and reuse the fitted version for validation, test and serving. A scikit-learn Pipeline with a ColumnTransformer does this for you and saves the steps with the model.

Unseen categories. In production a new city or product will appear. Decide in advance: one-hot with handle_unknown="ignore" (all zeros, as Lille gets above), hashing (it always has a column), or a dedicated "other" bucket for rare and unseen values.

Online and offline consistency. A 7-day aggregate is easy to compute in a batch job over a table and much harder to compute in under 50 ms when a transaction arrives. If the two computations differ even slightly, you get training-serving skew. Feature stores, covered in Feature stores and training-serving skew , exist to compute such features once and serve them consistently.

Trade-offs

OptionWins whenLoses when
One-hot Few to a few hundred categories, any model Thousands of categories: wide, sparse, slow
Target encoding High-cardinality categories with plenty of rows each Done without smoothing or cross-fitting, it leaks and overfits rare categories
Hashing Huge or open-ended vocabularies, streaming data You need to explain which value drove a prediction; collisions blur meaning
Standard vs robust scaling Standard for roughly bell-shaped data; robust when outliers are present Neither is needed for tree models
Window aggregates Behaviour data where history defines what is normal You cannot compute them identically and fast enough at serving time

Common mistakes

  • Fitting scalers or encoders on the full dataset before splitting, which leaks test statistics.
  • Ordinal-encoding plain categories (Lyon = 0, Paris = 1, Nantes = 2), inventing an order.
  • Target-encoding with each row's own label included, which hands the model the answer.
  • Window features that include the current event or anything after it.
  • Feeding a raw timestamp as a number, or hour of day without handling that 23 and 0 are neighbours.

In the wild

Fraud and risk teams rely heavily on velocity features: counts and sums of a card's or account's activity over the last minutes, hours and days, compared with that account's own history. Recommender and ads systems use enormous categorical features such as user and item ids, and handle them with hashing or learned embeddings. Feature stores became a standard part of ML platforms at large tech companies precisely because these windowed features must be computed the same way for training and for real-time serving.

Check yourself

  1. You train a k-nearest-neighbours model on income (thousands) and number of children (0 to 5) without scaling. What happens?

    Show answer

    B. Distances are dominated by income, so the children feature barely matters k-NN measures distance. A difference of 10,000 in income swamps a difference of 3 children unless both are put on comparable scales.

  2. A city seen in only 3 training rows, all churned, gets target encoding 1.0. What is the fix?

    Show answer

    B. Smooth toward the overall average, so rare categories get moderate values Three rows are weak evidence. Smoothing blends the category mean with the global mean, weighted by how many rows the category has.

  3. Which window feature is safe to use for predicting whether a transaction is fraud?

    Show answer

    B. Number of transactions in the 7 days before this one, excluding it Only information available before the moment of prediction is allowed. The others use the current event, future events or the outcome itself.

  4. Your linear model misses that house prices rise steeply with size in the city centre but barely in the suburbs. What feature helps most?

    Show answer

    B. An interaction such as size x is_centre A linear model adds up separate effects. An effect that depends on two features together needs an interaction term.

Recap

  • Feature engineering turns raw fields into numbers that make patterns easy to learn.
  • Scale numeric features for distance- and gradient-based models; standard is the default, robust handles outliers, trees need none.
  • Encode categories with one-hot, ordinal only for true orders, smoothed target encoding or hashing for many values.
  • Bins, interactions, date parts, text counts and window aggregates add domain knowledge a model would struggle to find.
  • Fit every feature step on training data, and make sure each feature uses only information available at prediction time.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up