Downhill on the loss

Forget millions of weights for a moment and picture just one. If you plot the loss as you slide that single weight left and right, you get a curve — a little valley. Stand anywhere on it and one question tells you everything: which way is downhill? The answer is the slope, and the slope is the gradient.

Scroll to slide one weight and read its slope.

Slope = the derivative = the gradient

That slope has a formal name — the derivative of the loss with respect to the weight, written ∂L/∂w. It answers precisely: if I increase this weight a tiny bit, does the loss go up or down, and how fast?

• Positive slope → increasing w increases loss → decrease w.
• Negative slope → increasing w decreases loss → increase w.
• Zero slope → you're at a flat spot (often the minimum) → stop.

In every case: step opposite the slope and the loss drops.

The catch that backprop solves

So we need ∂L/∂w for every weight. For the weights in the last layer that's easy — they're right next to the loss. But a weight buried deep in the first layer affects the loss only indirectly, through every layer that comes after it. How do you compute its slope without redoing enormous amounts of work?

The answer is a piece of calculus you already used in school without knowing its power: the chain rule. Next.