Words need context

Our ChatGPT course showed what a language model does — predict the next word — but skipped the trick that makes it actually work. That trick is attention, and it's the single most important idea in modern AI. This course opens the box.

Start with the problem attention solves. Take the word "it" in a sentence. On its own it means nothing — to understand it, you have to connect it to another word, maybe several words back. Every word is like this: its meaning depends on the company it keeps.

Scroll to watch a word reach out for its meaning.

Why older models struggled

Before attention, language models (RNNs) read a sentence one word at a time, squeezing everything seen so far into a single running memory. Two problems:

• Information faded. By the time the model reached "it," the earlier word "cat" had been passed through many steps and blurred — like a game of telephone.
• It was sequential. Word 10 couldn't be processed until word 9 was done, so training couldn't be parallelized across a sentence.

Attention fixes both at once: it lets any word connect directly to any other word, no matter how far apart, in a single step — and it does this for all words simultaneously.

The plan

So we want a mechanism where each word can look at every other word, decide which ones matter, and pull in their meaning. Over the next chapters we'll build exactly that, piece by piece:

• Query, key, value — the operation one word uses to gather from the rest.
• Self-attention — running it for every word at once.
• Multi-head — doing it several ways in parallel.
• The Transformer — stacking it into the architecture behind every LLM.

Next: the core move — query, key, value.