Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Data: the most important part

Machine Learning ML-02.03

Exploratory data analysis

Look hard at data before modelling: summary stats, histograms, box plots, scatter plots and correlation, and the bugs and leaks they reveal.

Beginner 15 min Last reviewed

The big idea

You get a file of 300 house sales and a request to predict prices. The tempting move is to load it and call fit. Do not. First spend twenty minutes looking at it. Exploratory data analysisLooking at a dataset's summary numbers and plots before modelling, to understand it and to find bugs, oddities and leaks. (EDA) is that looking: counting, summarising and plotting every column and the relationships between them.

It is the equivalent of a mechanic walking around a car before opening the hood. Most problems are visible from the outside if you bother to look: a flat tyre, a puddle of oil. In data, the flat tyres are typos, unit mix-ups, missing values and columns that secretly contain the answer. Find them now and you save days of debugging a model that is learning from broken data.

How it works

Step 1: Summary statistics for every column

Start with numbers. For each column: how many values, how many missing, mean, median, standard deviation, min and max. Read them with suspicion:

  • min and max: is a house size of 893 square metres plausible next to a median of 72? Is any age negative?
  • mean versus median: when the mean sits well above the median, a long right tail or a few huge values are pulling it up.
  • missing count: 24 missing out of 300 is 8 percent. Is it random, or are they all from one source?
  • distinct values for categorical columns: does "Lyon", "lyon" and "LYON " appear as three cities?
eda.pyruns with python3 + numpy + pandas
import numpy as np
import pandas as pd

rng = np.random.default_rng(7)
n = 300
size = np.exp(4.3 + 0.35 * rng.normal(size=n)).round()
dist = np.minimum(40, np.exp(1.2 + 0.7 * rng.normal(size=n))).round(1)
price = np.exp(np.log(3.2 * size) - 0.25 * np.log1p(dist) + 0.22 * rng.normal(size=n)).round()
df = pd.DataFrame({"size_m2": size, "dist_km": dist, "price_k": price,
                   "tax_value_after_sale_k": (price * rng.uniform(0.97, 1.01, n)).round()})
df.loc[rng.choice(n, 24, replace=False), "dist_km"] = np.nan   # planted: missing values
df.loc[[17, 141, 233], "size_m2"] *= 10.76                     # planted: square feet typos

pd.set_option("display.width", 100)
print(df.describe().round(1).loc[["count", "mean", "50%", "std", "min", "max"]])
print("\nmissing per column:\n" + df.isna().sum().to_string())
print("\nskew:\n" + df.skew().round(2).to_string())
print("\ncorrelation with price:\n" + df.corr()["price_k"].round(2).to_string())

# The three biggest sizes are about 10x the rest: a unit error, not mansions.
print("\nlargest sizes:", df["size_m2"].nlargest(4).round().tolist())
clean = df[df["size_m2"] < 500]
print("size-price r, all rows:", round(df["size_m2"].corr(df["price_k"]), 2),
      " without the 3 typos:", round(clean["size_m2"].corr(clean["price_k"]), 2))

Output

       size_m2  dist_km  price_k  tax_value_after_sale_k
count    300.0    276.0    300.0                   300.0
mean      80.8      3.7    169.3                   167.8
50%       72.0      3.0    151.5                   151.0
std       71.5      2.7     70.2                    70.0
min       24.0      0.6     59.0                    59.0
max      893.1     19.6    484.0                   487.0

missing per column:
size_m2                    0
dist_km                   24
price_k                    0
tax_value_after_sale_k     0

skew:
size_m2                   8.64
dist_km                   2.17
price_k                   1.27
tax_value_after_sale_k    1.29

correlation with price:
size_m2                   0.23
dist_km                  -0.28
price_k                   1.00
tax_value_after_sale_k    1.00

largest sizes: [893.0, 678.0, 656.0, 162.0]
size-price r, all rows: 0.23  without the 3 typos: 0.79

In five lines of pandas you already have three findings: dist_km has 24 missing values, size_m2 has a maximum about 10 times larger than everything else, and one column correlates with price at 1.00.

Step 2: Plot the distribution of each column

A histogramA chart that splits a numeric range into bins and shows how many values fall in each, revealing the shape of a distribution. shows the shape of one column: where values pile up, how spread out they are, whether there is one hump or two. Money, distances and counts are usually skewedA distribution with a long tail on one side. Right-skewed data has a few very large values that pull the mean above the median.: most values are modest and a few are huge. Plotting those on a log scale often turns the long tail into a tidy bell, which is a hint that a log transform may help a model later.

A box plotA compact plot of a distribution: a box from the first to the third quartile with a line at the median, whiskers for the typical range, and dots for outliers. shows the same column compactly. The box spans the middle 50 percent of values, from the first quartile (Q1) to the third (Q3); the width of that box is the interquartile rangeInterquartile range: Q3 minus Q1, the spread of the middle half of the data. Robust to extreme values. (IQR). Points more than 1.5 IQR outside the box are drawn individually as candidate outliersA value far from the rest of the data. It may be an error to fix or a rare but real event to keep.. Box plots are great for comparing many columns or groups side by side.

Step 3: Plot relationships between columns

A scatter plotA plot of one numeric column against another, one dot per row, showing how they move together. puts one column on each axis, one dot per row. It shows whether two things rise together, how noisy the relationship is, and whether it is a line, a curve or something stranger. A scatter matrixA grid of scatter plots, one for every pair of columns, for scanning many relationships at once. (pair plot) draws every pair at once, so with six columns you scan fifteen relationships in one picture.

Step 4: Correlation, and its limits

CorrelationPearson correlation r: a number from -1 to 1 measuring how well two columns fit a straight line. 0 means no linear relationship, not no relationship. squeezes a scatter plot into one number, Pearson's , from -1 (perfect falling line) through 0 (no straight-line pattern) to +1 (perfect rising line):

It is useful and easy to misread:

Zero correlation does not mean no relationship Twenty-one points lying exactly on a U-shaped parabola, y equals x squared over ten. The Pearson correlation is exactly zero because the left half falls and the right half rises. -10-505100510xy r = 0.00, yet y depends completely on x Figure 1. A perfect U-shaped relationship has a correlation of exactly zero. Correlation only measures straight-line association, so always look at the scatter plot too.
  • It only sees straight lines. The U shape above has even though is completely determined by .
  • A few points can dominate it. In the dataset above, size and price have with all rows and once the three square-feet typos are removed. Three bad rows out of 300 hid a strong relationship.
  • It is not causation. Ice cream sales and drownings correlate because both rise in summer.
  • Near-perfect correlation with the target is a red flag, not a gift. See the next step.

Step 5: Spot leakage hints

When a feature predicts the target almost perfectly, ask one question: would I actually have this value at the moment I need to make the prediction? In the housing data, tax_value_after_sale_k correlates with price at 1.00 because it is computed from the sale price after the sale. A model trained with it would look brilliant offline and be useless in production, where the sale has not happened yet. That is data leakageWhen information that would not be available at prediction time ends up in the training features, making offline results look far better than reality., covered fully in Data leakage . EDA is where you usually catch it first.

Try it

The lab holds a similar synthetic dataset of 300 house sales with its own planted problems: skewed prices and distances, three sizes typed in square feet, missing ages, and one column computed after the sale.

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Histogram of size, then switch the box plot on. Which red points are real big homes and which are unit errors? Look at how far away the three furthest ones are.
  2. Histogram of distance to centre on a linear, then a log scale. Watch the mean and median lines: which one is the long tail pulling?
  3. Open the correlation matrix. Find the red-boxed pair, then scatter size against sale price and explain why that correlation is so much lower than you would expect.

Trade-offs

OptionWins whenLoses when
Summary statistics Quick first pass over many columns, finding impossible values and missing counts Shape matters: very different distributions can share a mean and sd
Histogram Seeing the shape of one column: skew, multiple humps, spikes at suspicious values Comparing many groups at once; bin width can hide or invent bumps
Box plot Comparing distributions across groups, flagging outlier candidates Two-humped data: the box hides that there are two clusters
Correlation matrix Scanning many pairs for strong linear links and leakage suspects Curved relationships, outlier-driven numbers, categorical columns

Common mistakes

  • Skipping EDA and discovering the unit error after a week of model tuning.
  • Deleting every point a box plot flags. Some outliers are your most important real cases.
  • Reading as "unrelated", or a high as "causes".
  • Celebrating a feature that correlates 0.99 with the target instead of asking whether it leaks.
  • Exploring the full dataset including the test set and then making choices based on it. Do EDA on training data once you have split (see Train, validation and test splits ).

In the wild

Data science teams typically start every new dataset with a profiling pass: summary statistics, missing-value counts and distribution plots for every column, often generated automatically by profiling tools. Kaggle competition notebooks almost always open with EDA, and many winning insights, including leaks in competition data, were found there rather than in modelling. In production, the same checks run continuously as data validation, catching broken upstream pipelines before they reach a model.

Check yourself

  1. A price column has mean 420k and median 260k. What does that suggest?

    Show answer

    B. A long right tail or some very large values pull the mean up The mean is sensitive to extreme values and the median is not. A mean far above the median is the signature of right skew.

  2. Two columns have Pearson r = 0.02. What can you conclude?

    Show answer

    B. There is no straight-line relationship, but there could still be a strong curved one Pearson r only measures linear association. A U shape or any symmetric curve can have r near 0 while being perfectly predictable.

  3. While predicting whether a loan will default, you find a feature 'days_in_collections' with correlation 0.97 to the label. What should you do first?

    Show answer

    B. Check when that value becomes known; it is probably recorded after default, so it leaks the answer Collections happen after a loan goes bad. At prediction time the value would not exist, so it is target leakage.

  4. A box plot of house sizes flags 13 outliers. Which is the best next step?

    Show answer

    B. Look at each one: values about 10x the rest suggest a unit error to fix; large but plausible homes should stay Outlier flags are candidates, not verdicts. Fix clear errors, keep genuine rare cases, and decide with domain knowledge.

Recap

  • EDA means looking at summaries and plots of every column before modelling.
  • Mean far from median signals skew; impossible min or max values signal bugs.
  • Histograms show shape, box plots compare spreads and flag outlier candidates, scatter plots show relationships.
  • Correlation measures only straight-line association and can be wrecked by a few bad rows.
  • A feature that predicts the target almost perfectly is a leakage suspect until proven otherwise.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up