Self-Attention Mechanics

Scaled dot-product attention:

[ \mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V ]

Why it works

Each token builds a context-dependent representation by softly selecting information from other tokens.

Multi-head attention

Multiple subspaces let the model attend to different relations in parallel.

Positional information

Without positions, attention is permutation-equivariant — add sinusoidal, learned, or RoPE encodings.