Take a step, repeat

Backprop handed us the gradient — the downhill direction for every weight. Now we actually move. The update rule is almost anticlimactically simple: nudge each weight a little bit against its gradient. Do that, recompute, and do it again — millions of times. That's gradient descent, and it's all the "learning" there is.

Scroll to step a weight down the loss.

The update rule

For every weight, all at once:

Batches, epochs, and "stochastic"

One detail from practice: you don't compute the loss over the entire dataset before each step — that's too slow. Instead you use a batch (a small random sample), take a step, grab the next batch, and so on. A full pass over the data is an epoch, and models train for many epochs. Because each step uses a random batch, it's called stochastic gradient descent (SGD) — the noise even helps it escape bad spots.

The whole loop, in five lines

for batch in data:                       # many batches, many epochs
  preds = model(batch.inputs)          # forward pass
  loss  = loss_fn(preds, batch.labels) # how wrong?

  grads = loss.backward()              # backprop: ∂loss/∂w for every weight
  for w, g in zip(model.weights, grads):
      w -= learning_rate * g           # take a step downhill

That's it — forward, backward, step, repeat. Everything from a tiny classifier to GPT is trained by this loop. To close, let's see just how universal it is. Next: where backprop shows up.