Query, key, value

How does a word actually "look at" the others? Attention borrows a beautiful metaphor from search: every word issues a query ("what am I looking for?"), every word advertises a key ("here's what I offer"), and carries a value ("here's what I'd contribute"). Match the query against the keys, and blend the values of whatever matched best.

Scroll to run one word's query against the sentence.

Three roles from one word

Here's the clever part: each word generates all three vectors — query, key, value — from its own embedding, using three learned weight matrices. Same word, three different "views":

• Query (Q) — what this word wants to find.
• Key (K) — what this word can be found by.
• Value (V) — the information this word passes on if attended to.

The match between a query and a key is a dot product — big when the two vectors point the same way (from the embeddings course: aligned vectors = similar). Those matches become weights, and the answer is a weighted average of values.

Meaning that depends on context

Notice what came out: a new vector for "it" that has absorbed "cat." That's the magic — attention rewrites each word's representation to fold in the words it depends on. The word "bank" near "river" ends up with a different vector than "bank" near "money," purely through this weighting.

We've done it for one word. Now do it for every word, at the same time — and a structure appears. Next: self-attention.