Everyone looks at everyone

Last chapter, one word gathered meaning from the rest. Now do it for every word at once. Each word runs its own query against all the keys, and the result is a grid of weights — the attention matrix — that says, for every pair of words, how much one attends to the other. This is self-attention, and it's the beating heart of a transformer.

Scroll to build the matrix, row by row.

Why "self"?

It's called self-attention because the queries, keys and values all come from the same sequence — the sentence attends to itself. (In translation, "cross-attention" lets one sentence attend to another; same math, different source.) Every word simultaneously plays all three roles — asking, answering, and contributing — for every other word.

Reading the matrix

That grid is genuinely interpretable. Look down a column and you see which words lean on a given word; look across a row and you see what one word is "thinking about." Researchers find heads that reliably light up for grammar, for the previous word, for matching quotes and brackets — patterns nobody programmed, all learned from text.

But a single attention matrix can only capture one notion of relevance at a time. Real transformers run several in parallel, each free to specialize. Next: multi-head attention.