Multi-Head Attention (MHA) is a core component of the Transformer architecture, introduced by Vaswani et al. in "Attention Is All You Need" (2017). It extends single-head (scaled dot-product) attention by performing several attention computations in parallel, each with its own learned linear projections of queries, keys, and values. The outputs of all heads are concatenated and linearly projected to produce the final result.
How it works: Given an input sequence, the model first applies three separate weight matrices per head to project the input into query (Q), key (K), and value (V) subspaces. For H heads, each head h computes attention scores via softmax(Q_h * K_h^T / sqrt(d_k)) and aggregates values. The typical number of heads ranges from 8 to 96 (e.g., GPT-3 uses 96 heads in its largest variant). The dimension per head is usually d_model / H, keeping total computation roughly constant. Each head can learn to focus on different linguistic or structural patterns — for example, one head may capture syntactic dependencies while another tracks semantic roles.
Why it matters: Multi-Head Attention enables Transformers to outperform recurrent and convolutional models on sequence transduction tasks. It provides several benefits: (1) richer representational capacity by combining multiple attention patterns; (2) improved gradient flow during training; (3) better handling of long-range dependencies compared to RNNs. The mechanism is the backbone of virtually all modern large language models (LLMs), including GPT-4, Claude, Gemini, and Llama 3.
Variants and improvements: Several optimizations have emerged. *Multi-Query Attention* (MQA) shares key/value projections across heads to reduce memory bandwidth, used in PaLM and Falcon. *Grouped-Query Attention* (GQA) is a middle ground, partitioning heads into G groups that share key/value heads — Llama 2 and Llama 3 use GQA (e.g., Llama 3.1 405B uses 8 key-value heads with 64 query heads). *Flash Attention* (Dao et al., 2022) and its successors (FlashAttention-2, 2023; FlashAttention-3, 2024) implement exact attention with tiling and kernel fusion, achieving up to 2-4x speedups on GPUs and reducing memory from O(N^2) to O(N). As of 2026, nearly all production Transformers use some form of memory-efficient attention (e.g., flash attention, sparse attention, or ring attention for long contexts).
When to use vs. alternatives: Multi-Head Attention is the default for most sequence modeling tasks. However, for extremely long sequences (e.g., 1M tokens), linear attention variants (e.g., Performer, Linformer) or state-space models (Mamba, Mamba-2) may be more efficient, though they often trade off quality. For real-time streaming, causal masking is required. For decoder-only models, causal MHA with autoregressive masking is standard.
Common pitfalls: (1) Training instability if head dimensions are too large or too small; typical d_k = 64. (2) Overparameterization — not all heads are equally important; pruning low-attention heads can reduce compute with minimal quality loss (Michel et al., 2019). (3) Memory explosion for long sequences — naive O(N^2) complexity is prohibitive beyond 8K tokens without optimized kernels. (4) Incorrect masking in causal attention can leak future information.
Current state of the art (2026): MHA remains dominant, but with extensive engineering. The largest models (e.g., Gemini Ultra, GPT-5) use millions of attention heads across layers, combined with mixture-of-experts (MoE). Research focuses on improving length generalization (e.g., ALiBi, RoPE), reducing KV-cache size (e.g., multi-query, GQA, sliding window), and hardware-aligned designs (e.g., NVIDIA Hopper H100 tensor core optimizations).