Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Mathematics for ML (only what you need, visually)

Machine Learning ML-01.02

Matrices and linear transformations

Understand matrices as machines that rotate, stretch and shear space, and master the shape rules behind every neural network layer.

Beginner 14 min Last reviewed

The big idea

A matrixA rectangular grid of numbers with rows and columns. Multiplying a vector by it transforms the vector. is a grid of numbers. You can read it as a table, but the far more useful reading is: a matrix is a machine that takes a vector in and puts a transformed vector out. Feed it every point of the plane and the whole plane gets rotated, stretched, squashed or slanted.

Here is why that matters for machine learning: every layer of a neural network is a matrix multiplication followed by a simple bend (the activation function). The network learns by adjusting the numbers in those matrices, which means it learns how to reshape space so that, say, cat photos and dog photos end up on opposite sides of a line.

Picture a sheet of graph paper printed on stretchy rubber. Pin the origin to the table, then grab two corners and pull. Straight lines stay straight, parallel lines stay parallel, and the origin stays put. Every such pull is a matrix, and every matrix is such a pull.

How it works

Step 1: A matrix is decided by where it sends two arrows

Take the two basic arrows of the plane: (one step right) and (one step up). Any point is some amount of plus some amount of . So if you know where a transformation sends and , you know where it sends everything. That is all a 2 by 2 matrix stores: its columns are where and land.

That particular matrix is a shearA transformation that slides each row sideways in proportion to its height, turning squares into parallelograms.: it slides each row sideways in proportion to its height, so squares become parallelograms.

A shear matrix warping the grid The grid lines are drawn after applying the matrix with columns 1, 0 and 1, 1. Vertical lines lean to the right, horizontal lines stay horizontal, and the unit square becomes a parallelogram. The basis vector i stays at 1, 0 and j moves to 1, 1. i j Figure 1. The shear [[1, 1], [0, 1]]: i stays put, j moves to (1, 1), and every grid line follows. The gold square became a parallelogram with the same area.

Step 2: Matrix times vector

To transform a vector, take each row of the matrix and dot it with the vector:

Worked example with the shear: . The point moved right by its height.

A few useful machines:

  • Rotate by angle :
  • Scale x by 2 and y by one half:
  • Do nothing: the identity matrixThe matrix with ones on the diagonal and zeros elsewhere. Multiplying by it leaves any vector unchanged.

Step 3: Matrix times matrix means "do one, then the other"

Multiplying two matrices composes their transformations. means "rotate with , then scale with ": read right to left, the same order the vector meets them. Order matters: rotating then stretching is not the same as stretching then rotating, so in general .

Step 4: Shapes must line up

Matrices do not have to be square. An matrix (m rows, n columns) takes an n-long vector and returns an m-long one. That is exactly a neural network layer: 3 inputs in, 4 outputs out means a weight matrix.

Shapes in a matrix-vector product A weight matrix of shape 4 by 3 times an input vector of length 3 gives an output vector of length 4. The inner dimensions, 3 and 3, must match; the outer ones give the result's shape. inner 3s match W shape (4, 3) x shape (3,) W @ x shape (4,) @ Figure 2. (4 x 3) times (3,) gives (4,). The inner sizes must match; the outer sizes become the result.

Matrix times matrix follows the same rule: . When shapes do not match, NumPy and PyTorch raise an error, and reading that error carefully is half of debugging ML code. NumPy's broadcasting rules from Your toolkit: Python, NumPy, pandas, scikit-learn, PyTorch stretch a smaller array across a larger one, which is how a bias vector gets added to every row of a batch.

Step 5: Undoing a transformation

The inverseThe matrix that undoes another: A times its inverse is the identity. It exists only when the determinant is not zero. undoes : . Not every matrix has one. If a matrix squashes the whole plane onto a line, information is lost and there is no way back. The determinantA number that says how a matrix scales area (or volume). Zero means space is flattened and the matrix has no inverse; negative means it is flipped. tells you: it is the factor by which areas scale, and a determinant of 0 means everything was flattened, so there is no inverse.

matrices.pyruns with python3 + numpy
import numpy as np

theta = np.radians(90)
R = np.array([[np.cos(theta), -np.sin(theta)],
              [np.sin(theta),  np.cos(theta)]])      # rotate 90 degrees
v = np.array([1.0, 0.0])
print("R @ v =", (R @ v).round(3))                    # x-axis turns into the y-axis

S = np.array([[2.0, 0.0], [0.0, 0.5]])               # stretch x, squash y
print("S @ R @ v =", (S @ R @ v).round(3))           # rotate first, then scale

W = np.random.default_rng(0).normal(size=(4, 3))      # a layer: 3 inputs -> 4 outputs
x = np.ones(3)
print("W shape", W.shape, "x shape", x.shape, "W @ x shape", (W @ x).shape)

A = np.array([[2.0, 1.0], [1.0, 3.0]])
A_inv = np.linalg.inv(A)
print("A @ A_inv =\n", (A @ A_inv).round(6) + 0.0)
print("det of singular [[1,2],[2,4]] =", np.linalg.det(np.array([[1.0, 2.0], [2.0, 4.0]])).round(6))

Output

R @ v = [0. 1.]
S @ R @ v = [0.  0.5]
W shape (4, 3) x shape (3,) W @ x shape (4,)
A @ A_inv =
 [[1. 0.]
 [0. 1.]]
det of singular [[1,2],[2,4]] = 0.0

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Press "shear" and watch move to while stays still. The determinant stays 1: areas do not change.
  2. Press "rotate". Then set a to 2. You just composed a rotation with a stretch.
  3. Press "squash". The determinant is 0 and the whole plane collapses onto a line. Try to imagine undoing that: you cannot, which is why there is no inverse.

Trade-offs

OptionWins whenLoses when
Solve with an explicit inverse Small, well-conditioned systems in a teaching example Real code: it is slower and less accurate than a solver
np.linalg.solve or least squares Solving Ax = b in practice, including non-square systems You need the inverse itself for some other purpose
Many small matrix products Readability when prototyping Performance: batch them into one large product so the GPU stays busy

Common mistakes

  • Multiplying in the wrong order. and are different transformations, and often different shapes.
  • Confusing * (element-wise) with @ (matrix multiplication) in NumPy and PyTorch.
  • Mixing up (n,), (n, 1) and (1, n). They broadcast differently. Print .shape constantly.
  • Inverting a matrix with a determinant near zero and getting huge, meaningless numbers.

In the wild

Matrix multiplication is the single most performed operation in modern AI. Every dense layer, every attention head and every embedding lookup in a large language model reduces to it, which is why GPUs and specialized AI chips are essentially matrix-multiplication machines. When people quote a model's size in parameters, most of those parameters are entries in weight matrices.

Check yourself

  1. A matrix has columns [2, 0] and [0, 3]. What does it do to the point [1, 1]?

    Show answer

    A. Sends it to [2, 3] The columns are where i and j land: [1, 1] = i + j goes to [2, 0] + [0, 3] = [2, 3]. The matrix stretches x by 2 and y by 3.

  2. A layer's weight matrix W has shape (128, 512). What input vector length does it accept, and what length does it output?

    Show answer

    B. 512 in, 128 out An (m, n) matrix takes an n-long vector and returns an m-long one: 512 in, 128 out.

  3. The determinant of a 2 by 2 matrix is 0. What does that tell you?

    Show answer

    C. It flattens the plane onto a line or point, so it has no inverse Determinant is the area scale factor. Zero means areas collapse to nothing, information is lost, and the transformation cannot be undone.

  4. You want to rotate a vector, then scale it. With rotation R and scaling S, which product do you compute?

    Show answer

    B. S R v Read right to left: the vector meets R first, then S. Matrix products generally do not commute, so the order matters.

Recap

  • A matrix is a transformation: its columns are where the basis vectors i and j land.
  • Matrix times vector dots each row with the vector; matrix times matrix composes transformations, right to left.
  • An (m, n) matrix maps n-long vectors to m-long ones, which is exactly a neural network layer.
  • The identity does nothing; the inverse undoes a transformation and exists only when the determinant is not 0.
  • Track shapes everywhere: most ML bugs are shape bugs.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up