Many heads, many patterns

One attention matrix has to pick a single way of deciding what's relevant. But language has many kinds of relationship at once — grammar, word order, who-did-what, which pronoun points where. The fix is simple and powerful: run several attention mechanisms in parallel, each with its own learned query/key/value matrices. These are heads, and using a bunch of them is multi-head attention.

Scroll to meet a few heads, then see them combine.

Division of labor

Each head sees the same words but through different learned lenses, so it can specialize. In a trained model you'll find heads that consistently do interpretable jobs — attend to the previous token, match opening and closing brackets, resolve pronouns, track subject-verb agreement — while others do fuzzier things we can't easily name. None of this is hand-assigned; it emerges from training.

The last piece

We now have the complete attention operation: multi-head self-attention. But a mechanism isn't an architecture. To get a model that actually generates text, we need to wrap attention with a few supporting parts and stack it deep. Next: the Transformer.