Attention Is All You Need
Introduced the Transformer, an architecture built entirely on self-attention, dropping RNNs and convolutions.
8authors, most of whom later left Google to found companies
Why it matters
It trains in parallel and handles long-range dependencies better. Almost every large model since, from GPT and BERT to Claude and Gemini, is built on it.
Costs and controversy
At the time it was just a machine-translation paper, and self-attention's cost grows with the square of sequence length, a problem that persists today.
Built on
Led to
Source: arxiv.org ↗