A guess, and how wrong it is

Every course on this site says the same magic phrase — "the weights are learned" — and then moves on. This course is the part everyone skips: how. How does a pile of random numbers turn itself into a network that recognizes cats, writes code, or folds proteins? The answer is one algorithm, backpropagation, and it's more elegant than it sounds.

It starts where every learning step starts: the network makes a guess, and we measure how wrong it was.

Scroll to watch a wrong guess become a number to fix.

The loss: one number for "how wrong"

Learning needs a target to aim at. The loss function provides it: it compares the network's prediction to the correct answer and boils the difference down to a single number. High loss = bad guesses; zero loss = perfect. It's the same loss you've met in the other courses — cross-entropy for classification, squared error for numbers.

Training has exactly one goal: make that number small.

The real problem: which way?

Here's the catch that makes it hard. A big model has millions or billions of weights. We can't try them one at a time — that would take longer than the age of the universe. And we can't just guess. We need, for every single weight simultaneously, the answer to: would nudging this weight up or down make the loss smaller, and by how much?

That "how the loss changes as I nudge a weight" is called the gradient, and computing it — cheaply, for every weight at once — is the whole game. Next, we'll see what a gradient actually is, one weight at a time.