Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Classification

Machine Learning ML-04.01

Logistic regression

Turn a linear score into a probability with the sigmoid, read weights as log-odds, draw the decision boundary and train with cross-entropy.

Beginner 15 min Last reviewed

The big idea

Linear regression predicts a number. Many questions have a yes or no answer instead: will this customer churn, is this email spam, will this student pass? You could fit a line to 0s and 1s, but a line happily predicts 1.7 or -0.4, which is not a probability, and one extreme point can drag it around.

Logistic regressionA linear classifier that passes a weighted sum of the features through the sigmoid function to output the probability of the positive class. keeps the linear score and adds one step: it squashes the score into the range 0 to 1 with an S-shaped curve. Think of a dimmer switch. Turning the knob (the score) far left keeps the light off, far right keeps it fully on, and only near the middle does a small turn change much. The output is a probability, and a thresholdThe probability above which a classifier predicts the positive class. 0.5 by default, but it should be chosen from the cost of each kind of error. turns it into a decision.

Despite its name, logistic regression is a classifier. It is fast, hard to break, easy to explain, and it is the first model to try on almost any binary problem.

How it works

Step 1: A linear score

Exactly as in linear regression, compute a weighted sum of the features plus a bias:

The score z can be any number. Large positive means "looks like class 1", large negative means "looks like class 0", and 0 means "cannot tell".

Step 2: The sigmoid squashes the score into a probability

The sigmoidThe function 1 / (1 + e^(-z)). It maps any real number to a value between 0 and 1, with sigmoid(0) = 0.5. function turns the score into a probability:

Some values: sigma(-4) = 0.018, sigma(-2) = 0.119, sigma(0) = 0.5, sigma(2) = 0.881, sigma(4) = 0.982. Every step of 2 in the score moves the probability a lot near the middle and very little at the ends.

The sigmoid maps scores to probabilities An S-shaped glowing curve rising from 0 on the left to 1 on the right, crossing 0.5 at score 0. Three gold points from the exam model in the code below: 2 hours of study has score -3.87 and probability 0.02; 5 hours has score -0.39 and probability 0.40; 8 hours has score 3.09 and probability 0.96. A dashed horizontal line marks the 0.5 threshold. -6-4-2024600.250.50.751score z (log-odds)probability of passing 2 hours: 0.02 5 hours: 0.40 8 hours: 0.96 threshold 0.5 Figure 1. The sigmoid turns any score into a probability. Three students from the exam model below: scores far from 0 give confident answers, a score near 0 gives an uncertain one.

Step 3: Weights speak in log-odds

Run the sigmoid backwards and you get the meaning of the score. The oddsThe ratio p / (1 - p): how many times more likely an event is to happen than not. Probability 0.8 is odds of 4. of passing are p / (1 - p), and the score is their logarithm:

So the score is the log-oddsThe natural logarithm of the odds, log(p / (1 - p)). Logistic regression is linear in the log-odds., and logistic regression is a linear model of the log-odds. That gives every weight a clean reading: increasing feature x_1 by one unit adds w_1 to the log-odds, which multiplies the odds by e^(w_1).

Worked example from the code below: the model learns log-odds = 1.16 x hours - 6.19. Since e^1.16 = 3.19, each extra hour of study multiplies the odds of passing by about 3.2. At 5 hours the log-odds are 1.16 x 5 - 6.19 = -0.39, so the probability is sigma(-0.39) = 0.40.

Step 4: The decision boundary is a straight line

With the default threshold 0.5, the model predicts class 1 exactly when z > 0, because sigma(0) = 0.5. The set of points where z = 0 is the decision boundaryThe surface in feature space where a classifier switches from predicting one class to the other.. For one feature it is a single cut point: 1.16 x hours - 6.19 = 0 gives hours = 5.33. For two features it is a straight line, for three a flat plane.

Fitting the sigmoid to yes or no outcomes Forty simulated students shown as dots at height 0 (failed, cyan) or 1 (passed, purple) against hours studied. A glowing sigmoid curve rises from near 0 at 0 hours to near 1 at 10 hours. A gold dashed vertical line at 5.33 hours marks where the predicted probability crosses 0.5; the region to its right is lightly shaded as predicted pass. 01234567891000.250.50.751hours studiedpassed (1) or not (0) boundary 5.33 h Figure 2. The data are only 0s and 1s, but the fitted curve gives a probability for every number of hours. Left of the gold boundary the model predicts fail, right of it pass. Students near the boundary are genuinely uncertain.

Step 5: Training with binary cross-entropy

Squared error works badly here: it gives a bumpy loss surface after the sigmoid and barely punishes confident mistakes. Logistic regression instead minimises binary cross-entropyThe loss -[y log p + (1 - y) log(1 - p)] averaged over examples. Also called log loss. It punishes confident wrong predictions very heavily. (log loss):

Only one term is active per example. For a student who passed (y = 1) the loss is -log p: predicting 0.9 costs 0.11, predicting 0.6 costs 0.51, predicting 0.1 costs 2.30, and predicting 0.01 costs 4.61. Confident and wrong is punished hard, which is exactly what you want from a probability model. This is the cross-entropy from Information theory for ML .

The loss is convex, so gradient descent ( Gradient descent, the engine of ML ) finds the single best set of weights. Its gradient is pleasingly simple: for each weight, the average of (p - y) times the feature, the same shape as linear regression with the prediction error p - y.

Probabilities vs labels

Keep the two outputs apart. predict_proba gives the probability; predict applies the 0.5 threshold and gives a label. The probability is the more useful output: you can rank customers by churn risk, pick a different threshold when errors have different costs ( Imbalanced data ), or feed it into a business calculation. Two predictions of 0.51 and 0.99 are both "class 1" as labels but mean very different things.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Drag one purple point deep into the cyan region. The boundary tilts toward it, but only a little: one point cannot overrule eleven. Watch the cross-entropy jump.
  2. Drag the two clusters far apart. The dashed 0.1 and 0.9 lines squeeze toward the boundary: the model becomes more confident because the classes are easy to separate.
  3. Move the threshold slider from 0.5 to 0.8. The boundary slides toward the purple side: you predict class 1 only when you are more sure, so fewer points are labelled purple.

Going deeper

Code: fitting it with scikit-learn

logistic.pyruns with python3 + numpy + scikit-learn
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

print("z      :", np.array([-4, -2, 0, 2, 4]))
print("sigmoid:", sigmoid(np.array([-4, -2, 0, 2, 4])).round(3))

# Hours studied -> passed the exam (1) or not (0). 200 students, seeded.
rng = np.random.default_rng(0)
hours = rng.uniform(0, 10, 200)
p_true = sigmoid(1.2 * (hours - 5))          # true chance of passing
passed = (rng.random(200) < p_true).astype(int)

model = LogisticRegression().fit(hours.reshape(-1, 1), passed)
w, b = model.coef_[0, 0], model.intercept_[0]
print(f"\nlearned: log-odds = {w:.2f} * hours {b:+.2f}")
print(f"boundary (p = 0.5) at hours = {-b / w:.2f}")
print(f"each extra hour multiplies the odds by {np.exp(w):.2f}")

for h in [2, 5, 8]:
    p = model.predict_proba([[h]])[0, 1]
    print(f"{h} hours: log-odds {w * h + b:+.2f}  probability {p:.2f}  label {int(p >= 0.5)}")

# Binary cross-entropy: confident and wrong is punished hard
for y, p in [(1, 0.9), (1, 0.6), (1, 0.1), (1, 0.01)]:
    print(f"true {y}, predicted {p:.2f}: loss {-np.log(p):.2f}")
print(f"\nmean BCE on training data: {log_loss(passed, model.predict_proba(hours.reshape(-1, 1))):.3f}")

Output

z      : [-4 -2  0  2  4]
sigmoid: [0.018 0.119 0.5   0.881 0.982]

learned: log-odds = 1.16 * hours -6.19
boundary (p = 0.5) at hours = 5.33
each extra hour multiplies the odds by 3.19
2 hours: log-odds -3.87  probability 0.02  label 0
5 hours: log-odds -0.39  probability 0.40  label 0
8 hours: log-odds +3.09  probability 0.96  label 1
true 1, predicted 0.90: loss 0.11
true 1, predicted 0.60: loss 0.51
true 1, predicted 0.10: loss 2.30
true 1, predicted 0.01: loss 4.61

mean BCE on training data: 0.261

The data were generated with a true log-odds slope of 1.2 per hour and a boundary at 5 hours; the model recovers 1.16 and 5.33 from 200 noisy students. The cross-entropy table shows how steeply the loss grows as a confident prediction turns out wrong.

Regularization and separable data

If the two classes can be separated perfectly, the loss keeps falling as the weights grow: a steeper sigmoid makes every training probability closer to 0 or 1. Without a brake the weights drift toward infinity. That is why scikit-learn's LogisticRegression applies L2 regularization by default (the C parameter is the inverse of its strength, see Regularization: ridge, lasso, elastic net ). It keeps weights finite and probabilities sensible.

Curved boundaries need new features

The boundary is linear in the features you give it. If class 1 forms a ring around class 0, no straight line works. Add features such as x1^2 and x2^2 (as with polynomial regression in Polynomial regression and basis functions ) and the boundary becomes a curve in the original space. Scale features first, both for regularization and for the solver.

Trade-offs

OptionWins whenLoses when
Logistic regression You need a fast, explainable baseline with real probabilities The true boundary is strongly curved and you cannot engineer the features
Linear regression on 0/1 labels Never, for classification Outputs below 0 and above 1; outliers drag the cut point
Tree ensembles Tabular data with interactions and nonlinear effects You need simple, stable coefficients to explain to regulators
Neural network Images, text, huge datasets Small tabular data; more tuning, less transparency

Common mistakes

  • Treating predict as the only output and throwing away the probabilities.
  • Reading a weight as "the change in probability per unit". It is the change in log-odds; the effect on probability depends on where you start.
  • Comparing weights across features measured on different scales. Standardize first if you want to compare them.
  • Forgetting that the default L2 regularization is on, and being surprised that weights are smaller than expected.
  • Expecting a straight-line model to separate classes that need a curved boundary.

In the wild

Logistic regression is the standard model wherever decisions must be explained and probabilities matter. Credit scorecards in lending are classically built on it, because each weight translates into points for a feature. Medical risk scores, click-through prediction in online advertising, and churn models start with it, and teams use it as the baseline that any fancier model must beat. Softmax, its multi-class cousin, is the final layer of most neural network classifiers ( Multi-class and multi-label classification ).

Check yourself

  1. A logistic regression outputs a score z = 0 for a customer. What is the predicted probability?

    Show answer

    B. 0.5 sigma(0) = 1 / (1 + e^0) = 0.5. A score of 0 sits exactly on the decision boundary.

  2. A churn model has weight 0.7 on 'support tickets last month'. What does one extra ticket do?

    Show answer

    B. Multiplies the odds of churn by e^0.7, about 2 Weights act on the log-odds. Adding 0.7 to the log-odds multiplies the odds by e^0.7 = 2.01. The change in probability depends on the starting point.

  3. Two models predict 0.9 and 0.6 for a case whose true label is 1. Which has the lower cross-entropy, and by how much?

    Show answer

    A. The 0.9 model: 0.11 against 0.51 The loss for a true 1 is -log p: -log 0.9 = 0.11 and -log 0.6 = 0.51. Cross-entropy rewards confidence when it is right, even if both labels are correct.

  4. Your two classes are perfectly separable and unregularized logistic regression reports huge weights. Why?

    Show answer

    B. The loss keeps shrinking as the sigmoid gets steeper, so weights grow without limit On separable data every increase in the weights pushes training probabilities closer to 0 or 1 and lowers the loss. Regularization stops this.

Recap

  • Logistic regression is a linear score passed through the sigmoid, giving a probability between 0 and 1.
  • The score is the log-odds; each weight multiplies the odds by e to the weight per unit of its feature.
  • The decision boundary, where the score is 0, is a straight line or plane in feature space.
  • It is trained by minimising binary cross-entropy, a convex loss that punishes confident mistakes.
  • Keep the probabilities: thresholds, rankings and costs all need them, not just the labels.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up