Backpropagation by hand

This is the chapter. Everything else was setup; this is the algorithm that makes a network learn. The loss tells us how wrong we are — backpropagation tells us, for every single weight, which direction to nudge it to make the loss smaller. It does that with one idea from calculus: the chain rule, applied backwards through the network.

The forward pass went X → z1 → a1 → z2 → a2 → loss. Backprop walks that chain in reverse, carrying the gradient of the loss with it, handing each layer its share of the blame.

Work backwards, one step at a time

Output layer. Here's the first gift. When a sigmoid output meets a cross-entropy loss, the messy derivatives cancel and collapse into something beautiful:

From that, the gradients for the output weights follow by the chain rule — each is the incoming activation matched against the error:

• dW2 = a1ᵀ · dz2 — how much each hidden unit's output contributed to the error.
• db2 = sum(dz2) — the bias just collects the error.

Hidden layer. Now push the error back through W2 to the hidden units, then back through the ReLU. ReLU's derivative is the simplest there is: 1 where its input was positive, 0 where it was clamped off. So the gradient only flows through units that were "on":

• da1 = dz2 · W2ᵀ — spread the error back onto each hidden unit.
• dz1 = da1 * (z1 > 0) — kill the gradient wherever ReLU was off.
• dW1 = Xᵀ · dz1 and db1 = sum(dz1) — same pattern as the output layer.

The whole thing in code

Every bullet above is one line. Note it takes z1, a1, a2 from the forward pass — that's why we returned them. Add to net.py:

def backward(X, y, z1, a1, a2):
  n = X.shape[0]

  # output layer: sigmoid + cross-entropy collapse to (a2 - y)
  dz2 = (a2 - y) / n
  dW2 = a1.T @ dz2
  db2 = dz2.sum(0, keepdims=True)

  # hidden layer: push error back through W2, then through ReLU
  da1 = dz2 @ W2.T
  dz1 = da1 * (z1 > 0)          # ReLU derivative
  dW1 = X.T @ dz1
  db1 = dz1.sum(0, keepdims=True)

  return dW1, db1, dW2, db2

That's it. Four gradients — one per weight matrix and bias — computed in six lines, no framework in sight. Read it against the forward pass: it's the exact same chain of matrix multiplies, run in reverse, with the error in place of the input.

This function is the whole trick. When you wrote loss.backward() in the PyTorch course, this is what ran under the hood — derived and executed automatically for a much bigger network. You just did it by hand. Now let's use these gradients to actually train.