Transformers Deep Dive

RNNs process tokens one at a time. Transformers process all tokens at once. That single architectural bet changed every scaling curve in deep learning after 2017.

Attention is a lookup table where every word asks "who matters to me?" - and learns the answer.

One attention head learns one relation at a time. Eight heads learn eight. Heads are free. Take more of them.

Attention is permutation-invariant. "The cat sat on the mat" and "mat the on sat cat the" produce the same output without positional signal. Three algorithms fix it — each with a different bet on what "position" means.

Attention is the star. Everything else — residuals, normalization, feed-forward, cross-attention — is the scaffolding that lets you stack it deep.

GPT predicts the next word. BERT predicts a missing word. One sentence of difference — and half a decade of everything embedding-shaped.

BERT sees both sides. GPT sees only the past. The triangle mask is the most consequential single line of code in modern AI.

Encoders understand. Decoders generate. Put them back together and you get a model built for input → output tasks: translate, summarize, rewrite, transcribe.

An image is a grid of patches. A sentence is a grid of tokens. The same transformer eats both.

Audio is an image of frequency over time. Whisper is a ViT that eats mel spectrograms and speaks back.

A dense 70B transformer activates every parameter for every token. A 671B MoE activates only 37B per token and beats it on every benchmark. Sparsity is the most important scaling idea of the decade.

Training is parallel and FLOP-bound. Inference is serial and memory-bound. Different bottleneck, different tricks.

The 2020 Kaplan paper said: bigger model, lower loss. The 2022 Hoffmann paper said: you were under-training. Compute goes into two buckets — parameters and tokens — and the split is not obvious.

Thirteen lessons. One model. No shortcuts.

Full attention is a circle. Every token sees every token, and memory pays the price. Four variants bend the shape of the circle and recover half the cost.

Autoregressive decoding is serial. Each token waits for the previous one. Speculative decoding breaks the chain: a cheap model drafts N tokens, the expensive model verifies all N in one forward pass. When the draft is right you paid one big forward for N generations.