What is multi-head attention and why is it better than a single attention head?

Multi-head attention runs several attention operations in parallel, each with its own learned query, key, and value projections. The model embedding is split into h lower-dimensional subspaces; each head computes scaled dot-product attention independently, and the outputs are concatenated and projected back. The benefit is representational diversity: different heads can specialize, one tracking syntactic dependencies, another coreference, another local n-gram patterns, so the model attends to information from multiple representation subspaces at different positions simultaneously. A single head must average all these relationships into one attention distribution, which is a bottleneck. Importantly, multiple heads cost roughly the same as one because the per-head dimension is d/h, keeping total compute constant. Variants like grouped-query and multi-query attention share keys/values across heads to shrink the KV cache.