Multi-head attention

One head only sees one thing

The attention mechanism you learned about in the last lesson computes a single weighted average over values. “Single” is the problem. A real sentence has many relationships happening at once — syntax, semantics, coreference, positional nearness, dependency structure — and a single weighted average cannot express them all simultaneously without collapsing them.

The fix, almost trivially, is to run several attention heads in parallel, each with its own learned projections. Each head can specialise in a different pattern. Their outputs are then concatenated and linearly mixed into the final layer output.

The learned weight matrices $W_i^Q$, $W_i^K$, $W_i^V$are different for each head — that's how the heads end up attending to different things even though the input is identical.

Four heads, four archetypes

On the right is a sentence processed by four attention heads in parallel. Click through them to see the pattern each one has learned. These are hand-wired demonstrations of patterns that realattention heads learn — you can find almost all of them in Clark et al. 2019's analysis of BERT heads. Some heads are extremely boring (“always attend to the previous token”); others are sophisticated (“attend from verb to subject”). Both are useful, and a real transformer has heads of both kinds and many more besides.

The mixing happens at the end

After each head computes its own weighted sum, their outputs are concatenated and projected through a final matrix $W^O$. This is not decorative — the $W^O$ projection is what lets the downstream FFN receive a single unified vector that contains information from all heads. Without it, the layer would output a giant concatenated blob with no interaction between heads.

A subtle and important fact: the head dimension $d_h$ is usually $d_{\text{model}}/h$, so that after concatenation the shape is the same as the input. If $d_{\text{model}} = 3072$ and $h = 24$, each head operates in a 128-dimensional subspace. Notice that this is also how modern Phi-4-mini is configured — 24 query heads, each with $d_h = 128$. You've now seen where that number comes from.