The perceptron and the artificial neuron

The big idea

Should you go to the beach today? You weigh a few facts. Sunny: that counts a lot in favour. A friend is coming: a little in favour. Heavy traffic: against. You add up the votes, and if the total clears some bar, you go.

That is an artificial neuronA simple computing unit that multiplies each input by a weight, adds the results and a bias, and passes the total through an activation function.. It multiplies each input by a weightA number that sets how much one input counts in a neuron's decision; positive weights push towards firing, negative ones against., adds them together with a biasA constant added to a neuron's weighted sum. It shifts the threshold, letting the neuron fire even when all inputs are zero, or making it harder to fire., and passes the total through an activation functionThe function a neuron applies to its weighted sum, such as a step, sigmoid or ReLU. It decides the neuron's output. that decides the output. Every neural network, up to the largest language models, is built from many of these.

The perceptronThe original learning neuron (1958): a weighted sum with a step activation, trained by nudging weights after each mistake. It can only separate classes with a straight line. adds a way to learn the weights: every time it gets an example wrong, nudge the weights towards the right answer. Simple, and in the late 1950s it was built as a machine.

How it works

Step 1: The weighted sum and the step

With inputs x1 to xn, weights w1 to wn and bias b, the neuron computes:

In Figure 1, the weighted sum is 0.9 x 1 + 0.3 x 0 + (-0.6) x 1 = 0.3; adding the bias 0.1 gives z = 0.4, positive, so the neuron fires. If the third input's weight were -0.8, z would be 0.9 - 0.8 + 0.1 = 0.2, still positive; at -1.1 it would be -0.1 and the neuron would stay silent. The bias is a movable threshold: a large negative bias means "only fire on strong evidence".

Step 2: The biological inspiration, and where it ends

The design borrows from brain cells. A biological neuron receives signals from thousands of others through synapses of different strengths, and fires a spike when its combined input crosses a threshold. Weights play the role of synapse strengths, the step plays the role of the firing threshold.

The analogy stops there. Real neurons fire spikes over time, use chemistry, adapt in ways no one fully understands, and do not learn by any rule we know to be like the methods in this track. Artificial neurons are a mathematical convenience inspired by biology, not a model of it. Treat "neural" as a historical name.

Step 3: The perceptron learning rule

Start with any weights. Show the perceptron one example at a time. If its output is right, change nothing. If it is wrong:

where t is the target (0 or 1), y-hat the output, and eta the learning rate. When the neuron said 0 but should have said 1, (t - y-hat) = +1 and each weight grows by its input: next time this example produces a larger z. When it said 1 but should have said 0, the weights shrink the same way. Repeat passes over the data (epochs) until a pass has no mistakes.

Step 4: A straight line, and the XOR wall

The neuron fires where w . x + b > 0. The boundary w . x + b = 0 is a straight line in 2D (a flat plane in more dimensions), the same kind of boundary as logistic regression. So a perceptron can only separate classes that a straight line can separate: data that is linearly separableTwo classes are linearly separable when a single straight line (or flat hyperplane) can put all of one class on one side and all of the other on the other side.. The perceptron convergence theorem guarantees that on such data the rule finds a separating line in a finite number of mistakes.

XOR breaks it. XOR outputs 1 when exactly one input is 1: (0, 1) and (1, 0) are class 1; (0, 0) and (1, 1) are class 0. The two classes sit on opposite corners of a square, and no single line puts both 1s on one side and both 0s on the other. The perceptron keeps updating forever. A 1969 book by Minsky and Papert made this limit famous and helped cool interest in neural networks for over a decade.

The fix is a second layer. One hidden neuron computes OR, another NAND, and an output neuron ANDs them. That network computes XOR, and stacking layers is the subject of Multi-layer networks .

Try it

Try this:

1. On "two blobs", press Step a few times. Watch which point gets the yellow ring and how the line and the pink arrow swing towards fixing it. How many updates until no point is outlined in red?
2. Switch to AND and press Epoch until it converges. Then try XOR. Press Epoch ten more times. Does the number of misclassified points ever reach 0?
3. Change the learning rate between 0.1 and 1.0 on "two blobs" and reset each time. With all weights starting at the same values, does it change the final line much, or only the size of each step?

Going deeper

Code: AND, OR, XOR and a two-layer fix

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
GATES = {"AND": [0, 0, 0, 1], "OR": [0, 1, 1, 1], "XOR": [0, 1, 1, 0]}

def neuron(x, w, b):
    # weighted sum + bias, then a step activation
    return int(x @ w + b > 0)

def train(y, lr=1.0, max_epochs=20):
    w, b = np.zeros(2), 0.0
    for epoch in range(1, max_epochs + 1):
        mistakes = 0
        for x, t in zip(X, y):
            err = t - neuron(x, w, b)   # +1, 0 or -1
            if err:
                w += lr * err * x        # the perceptron learning rule
                b += lr * err
                mistakes += 1
        if mistakes == 0:
            return w, b, epoch
    return w, b, None

for name, y in GATES.items():
    w, b, ep = train(y)
    out = [neuron(x, w, b) for x in X]
    status = f"converged in {ep} epochs" if ep else "did NOT converge in 20 epochs"
    print(f"{name:<4} w={w} b={b:+.0f}  outputs {out} target {y}  {status}")

# XOR with two layers: hidden units compute OR and NAND, the output ANDs them
h_or, h_nand = (np.array([1, 1]), -0.5), (np.array([-1, -1]), 1.5)
for x in X:
    h = np.array([neuron(x, *h_or), neuron(x, *h_nand)])
    print(f"x={x}  hidden (OR, NAND)={h}  XOR={neuron(h, np.array([1, 1]), -1.5)}")

AND  w=[2. 1.] b=-2  outputs [0, 0, 0, 1] target [0, 0, 0, 1]  converged in 6 epochs
OR   w=[1. 1.] b=+0  outputs [0, 1, 1, 1] target [0, 1, 1, 1]  converged in 4 epochs
XOR  w=[-1.  0.] b=+1  outputs [1, 1, 0, 0] target [0, 1, 1, 0]  did NOT converge in 20 epochs
x=[0 0]  hidden (OR, NAND)=[0 1]  XOR=0
x=[0 1]  hidden (OR, NAND)=[1 1]  XOR=1
x=[1 0]  hidden (OR, NAND)=[1 1]  XOR=1
x=[1 1]  hidden (OR, NAND)=[1 0]  XOR=0

Starting from zero weights, the rule learns AND in 6 epochs (w = [2, 1], b = -2) and OR in 4 (w = [1, 1], b = 0). On XOR it is still wrong on two of four inputs after 20 epochs. The last block hand-sets a two-layer network: hidden units OR and NAND, output AND, and it gets all four XOR cases right.

From step to smooth

The step function has a slope of zero everywhere except at the jump, where it has no slope at all. That is fine for the perceptron rule, which only looks at mistakes, but it rules out gradient descent. Replace the step with a sigmoid and the neuron becomes logistic regression ( Logistic regression ), trainable by gradients. That is the move that makes multi-layer networks trainable with backpropagation ( Backpropagation ). Smooth activations are covered in Activation functions .

Trade-offs

Common mistakes

• Forgetting the bias, which forces the boundary through the origin.
• Expecting a single neuron to learn XOR or any pattern that needs a curved boundary.
• Reading "neural network" as a claim that the model works like a brain.
• Running the perceptron rule on overlapping classes and waiting for it to converge; it never will.

In the wild

The perceptron itself is rarely used today, but its neuron is everywhere: every unit in a modern network is a weighted sum plus bias followed by an activation, just with a smoother activation and trained by gradients. Online linear classifiers that update on each mistake, descendants of the perceptron rule, are still used for fast streaming problems such as spam filtering. And the XOR lesson, that some patterns need more than one layer, is the reason deep networks exist.

Check yourself

Recap