The Transformer

Attention is the engine; the Transformer is the car built around it. Introduced in 2017 in a paper cheekily titled "Attention Is All You Need," it wraps multi-head self-attention with a few supporting parts and stacks the result deep. Every model you've heard of — GPT, BERT, Claude, Gemini — is a Transformer.

Scroll to assemble one from the bottom up.

The supporting cast

Attention alone isn't enough; three extras make it work:

• Positional encoding. Attention treats a sentence as a set — it has no inherent sense of order ("dog bites man" would look the same as "man bites dog"). So each token's embedding gets a position tag added, restoring word order.
• Feed-forward layers. After attention mixes information between words, a small network processes each word individually — this is where much of the model's raw knowledge lives.
• Residual connections & normalization ("add & norm"). These let gradients flow and keep the numbers stable, which is what makes it possible to stack many layers.

Why it took over

The Transformer won for two reasons we've already met. It's parallel — the whole sequence is processed at once (one big matrix multiply), so training scales across thousands of GPUs. And it handles long-range dependencies directly, no fading. Those two properties are exactly what let models grow to billions of parameters trained on the whole internet.

That's the architecture. To close, let's see just how far beyond chatbots it has spread. Next: where attention shows up.