Anomaly detection

The big idea

A card is used in two countries ten minutes apart. A pump's vibration creeps up a little each day. A payment provider's failure rate in one country jumps at 3 a.m. Each is an anomalyA data point that differs markedly from the bulk of the data, often signalling an error, fraud, a fault or a rare event. Also called an outlier.: rare, different, and usually what you most want to know about.

You almost never have labels for these. Fraud is a tiny fraction of transactions, and tomorrow's failure may look like nothing you have seen. So anomaly detectionFinding data points that do not fit the pattern of the rest, usually without labelled examples of anomalies. flips the problem: learn what normal looks like, then flag whatever does not fit it.

Think of a night guard who has walked the same building for a year. He cannot list every possible burglary, but he notices a door that is usually shut standing open. Every method in this lesson is a different way of building that sense of "usual", and each has its own blind spot.

How it works

Step 1: Statistical thresholds

The simplest model of normal is a bell curve per feature. Compute each value's z-scoreHow many standard deviations a value lies from the mean: z = (x - mean) / std., z = (x - mean) / std, and flag |z| above 3. For truly normal data that flags about 3 values in 1000, and in the code exactly 1 of 1000 normal rows crosses it on one reading.

Two weaknesses follow from the model. First, the mean and standard deviation are themselves dragged by the outliers you want to catch; a robust version uses the median and the median absolute deviation instead. Second, it looks at one feature at a time, so a row whose every value is ordinary but whose combination is impossible (a tall person who weighs 40 kg) passes unnoticed. For time series, compare each value to a rolling window and a seasonal baseline, or the Monday morning peak will alarm every week.

Step 2: Isolation forest

An isolation forestAn ensemble of random trees that isolates points with random splits; points that are isolated after few splits on average are scored as anomalies. turns the guard's intuition into geometry. Build a tree by picking a random feature and a random split value between its min and max, again and again, until every point sits alone in a leaf. A point far from the crowd gets separated after a cut or two. A point deep in the crowd needs many. Average the path length over a hundred random trees and you get a score: short paths mean anomalous.

In the code, on a 2D crowd of 255 points plus one at (6, 6), the average path length is 2.1 for the far point and 13.9 for a point at the centre. Isolation forests are fast (each tree uses a small subsample, 256 rows by default), need no distance calculations and handle many features, which makes them a strong default for tabular data.

Step 3: One-class SVM

A one-class SVMA support vector machine trained on normal data only. It learns a boundary that encloses most of the data in a kernel feature space; points outside are flagged. learns a boundary around the normal data, usually with an RBF kernel ( Support Vector Machines and kernels ), so the boundary can be curved. Its parameter nu is roughly the share of training points allowed outside the boundary. It works well on small to medium data with a smooth notion of distance, but it scales poorly to large datasets and is sensitive to feature scaling and gamma.

Step 4: Autoencoder reconstruction error

An autoencoderA neural network trained to reproduce its input through a narrow middle layer, forcing it to learn a compressed representation of the data. is a network that squeezes its input through a narrow bottleneck and rebuilds it. Train it on normal data only, and it learns the usual patterns: which readings move together. A normal row comes back almost unchanged. A row that breaks those patterns cannot be rebuilt well, so the reconstruction errorHow far a model's rebuilt version of an input is from the input itself, often the mean squared difference. Used as an anomaly score. is the anomaly score. A linear autoencoder with a 3-number bottleneck is essentially PCA with 3 components ( Principal Component Analysis ); deep ones capture curved patterns in images, logs and sensor streams.

Step 5: Pick a threshold

Every method outputs a score, not a verdict. Where you put the line decides how many alerts you get. If anomalies are 1 percent of data and you can review 50 alerts a day, set the threshold so roughly 50 rows cross it, then measure how many were real. When you have even a few labelled incidents, use them to compare methods by precision among the top alerts.

Going deeper

Code: two kinds of anomaly, four detectors

import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.svm import OneClassSVM
from sklearn.neural_network import MLPRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score

# 1000 normal rows: 10 sensor readings driven by 3 hidden factors (so they move together)
rng = np.random.default_rng(0)
W = rng.normal(0, 1, (3, 10))
normal = rng.normal(0, 1, (1000, 3)) @ W + rng.normal(0, 0.3, (1000, 10))
# 10 "loud" anomalies: one reading far too high
loud = rng.normal(0, 1, (10, 3)) @ W + rng.normal(0, 0.3, (10, 10))
loud[np.arange(10), rng.integers(0, 10, 10)] += 9
# 10 "quiet" anomalies: every reading in a normal range, but they ignore the shared factors
quiet = rng.normal(0, 1, (10, 10)) * normal.std(0)
X = StandardScaler().fit_transform(np.vstack([normal, loud, quiet]))
y = np.r_[np.zeros(1000), np.ones(10), np.full(10, 2)]  # 0 normal, 1 loud, 2 quiet

def report(name, score):
    # score: higher = more anomalous. Flag the top 20 rows (we expect 20 anomalies)
    top = np.argsort(score)[-20:]
    print(f"{name:<22} AUC {roc_auc_score(y > 0, score):.3f}   in top 20: "
          f"{np.sum(y[top] == 1):>2} loud, {np.sum(y[top] == 2):>2} quiet")

report("max |z-score|", np.abs(X).max(1))
report("isolation forest", -IsolationForest(random_state=0).fit(X).score_samples(X))
report("one-class SVM", -OneClassSVM(nu=0.02, gamma=0.1).fit(X).score_samples(X))
# Autoencoder: squeeze 10 -> 3 -> 10, train only on rows we believe are normal
ae = MLPRegressor(hidden_layer_sizes=(3,), activation="identity", max_iter=3000, random_state=0).fit(X[:1000], X[:1000])
report("autoencoder error", ((ae.predict(X) - X) ** 2).mean(1))

# The 3-sigma rule on one reading: how many false alarms per 1000 normal rows?
z = X[:1000, 0]
print(f"\nnormal rows with |z| > 3 on reading 0: {np.sum(np.abs(z) > 3)} of 1000")

# Why isolation works: average path length (number of random splits) to isolate a point
P = np.vstack([rng.normal(0, 1, (255, 2)), [[6, 6]]])
f = IsolationForest(random_state=0).fit(P)
c = 2 * (np.log(255) + 0.5772) - 2 * 255 / 256  # average depth of an unsuccessful BST search
depth = -c * np.log2(-f.score_samples([[0, 0], [6, 6]]))
print(f"average path length to isolate: centre point {depth[0]:.1f}, far point {depth[1]:.1f}")

max |z-score|          AUC 0.739   in top 20: 10 loud,  1 quiet
isolation forest       AUC 0.768   in top 20:  2 loud,  0 quiet
one-class SVM          AUC 0.962   in top 20:  4 loud,  1 quiet
autoencoder error      AUC 1.000   in top 20: 10 loud,  9 quiet

normal rows with |z| > 3 on reading 0: 1 of 1000
average path length to isolate: centre point 13.9, far point 2.1

The data has 1000 normal rows of 10 sensor readings driven by 3 hidden factors, so readings move together. Ten "loud" anomalies have one reading pushed far too high. Ten "quiet" anomalies have every reading in a normal range but ignore the shared factors: they break the correlations, not the ranges. Each detector flags its top 20 rows.

• Max |z-score| catches all 10 loud anomalies and only 1 quiet one (AUC 0.739). Per-feature thresholds cannot see broken combinations.
• Isolation forest does poorly here (AUC 0.768, 2 loud and 0 quiet). With 10 features, only about 1 random split in 10 lands on the spiked reading, and the quiet rows sit inside every feature's range, so random axis-aligned cuts struggle to isolate either kind. It shines when anomalies are far out on several features at once.
• One-class SVM ranks well overall (AUC 0.962) but its top 20 still mixes in normal rows.
• Autoencoder with a 3-unit bottleneck, trained on normal rows, finds everything: AUC 1.000, 10 loud and 9 quiet in the top 20. It learned the 3-factor pattern, and both kinds of anomaly break it.

The lesson is not "autoencoders win". It is that each method encodes a definition of normal, and an anomaly is only found if it violates that definition. This data was built with exactly the low-dimensional structure an autoencoder models. Know what kinds of anomaly you care about, and test methods on a few planted or historical ones.

Contamination and evaluation

Training data often contains some anomalies. Isolation forests and robust statistics tolerate a little contamination; autoencoders can learn to rebuild anomalies too if they are common in training. Evaluating is hard without labels: use past incidents, plant synthetic anomalies, and track how many alerts a human confirms. With anomalies at 1 percent, accuracy is useless (predicting "normal" always scores 99 percent); use precision and recall on the flagged set ( Classification metrics ).

Trade-offs

Common mistakes

• Fitting mean and standard deviation on data full of outliers, so the threshold hides them.
• Ignoring seasonality, so every Monday peak or holiday dip raises an alarm.
• Judging a detector by accuracy on data that is 99 percent normal.
• Training an autoencoder on data that already contains many anomalies, so it learns to rebuild them.
• Forgetting to scale features before distance- or kernel-based methods like the one-class SVM.

In the wild

Check yourself

Recap