"Attention Is All You Need" is a 2017 paper by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin (Google Brain, Google Research, and the University of Toronto). It introduced the Transformer, a sequence-to-sequence architecture based entirely on attention mechanisms, dispensing with the recurrence and convolutions used in prior sequence models. The Transformer is the architecture underlying modern large language models, including GPT, Claude, and Gemini.
The paper was first submitted to arXiv on 12 June 2017 (arXiv:1706.03762) and presented at the 31st Conference on Neural Information Processing Systems (NeurIPS / NIPS 2017). It is 15 pages with 5 figures (Source: https://arxiv.org/abs/1706.03762). The paper states that all eight authors were equal contributors and that the listed author order was randomized (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need).
Summary
The paper proposed a model built around attention rather than recurrent or convolutional layers. Because the architecture processes all positions in a sequence in parallel rather than stepping through them sequentially, it enables greater parallelization during training and more direct modeling of long-range dependencies.
The abstract frames the contribution against the then-dominant approach: sequence transduction models based on recurrent or convolutional neural networks in an encoder-decoder configuration, with the best-performing variants connecting encoder and decoder through an attention mechanism. The paper proposes "a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely," reporting that the models are "superior in quality while being more parallelizable and requiring significantly less time to train" (Source: https://arxiv.org/abs/1706.03762).
Methods and architecture
The paper introduced several mechanisms as part of the Transformer architecture (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need):
- Scaled dot-product attention: attention is computed as a softmax over the scaled dot products of query (Q) and key (K) matrices, applied to a value (V) matrix, with the dot products scaled by the square root of the key dimension (set to 64 in the base configuration). Because Q, K, and V derive from the same source in self-attention, the mechanism removes the need for recurrence and makes the computation parallelizable (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need).
- Self-attention: the model attends to all positions in the input sequence simultaneously, capturing relationships between tokens without stepping through them in order (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need).
- Multi-head attention: multiple attention heads run in parallel, each learning different linear projections of Q, K, and V, so the model can capture different aspects of the relationships between tokens at once; the per-head outputs are concatenated and passed through a final linear transformation (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need).
- Positional encoding: because the architecture has no recurrence or convolution, position information is injected using sine and cosine functions of the token position. The authors wrote that they "chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training" (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need).
Experiments and results
The paper evaluated the Transformer primarily on machine translation. On the WMT 2014 English-to-German task, the model reported 28.4 BLEU, described as an improvement of more than 2 BLEU over the prior best results, including ensembles. On the WMT 2014 English-to-French task, it reported a single-model state-of-the-art BLEU score of 41.8 after 3.5 days of training on eight GPUs, which the authors characterized as a small fraction of the training cost of the best prior models (Source: https://arxiv.org/abs/1706.03762).
The English-to-German model was trained on the WMT 2014 English-German dataset of roughly 4.5 million sentence pairs; the English-to-French model used the larger WMT 2014 English-French dataset of about 36 million sentences. Both datasets were encoded with byte-pair encoding. Training used eight NVIDIA P100 GPUs; base models were trained for 100,000 steps (about 12 hours) and the larger models for 300,000 steps (about 3.5 days) (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need).
The paper also reported that the Transformer generalized to English constituency parsing, performing well with both large and limited training data without task-specific tuning, which the authors presented as evidence that the architecture was suited to a range of sequence-to-sequence tasks rather than translation alone (Source: https://arxiv.org/abs/1706.03762). According to a later account, early internal experiments included English-to-German translation, generating Wikipedia-style articles, and parsing, which the team took as indication that the Transformer was a general-purpose model rather than a translation-specific one (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need).
Naming and authorship
The paper's title references the Beatles song "All You Need Is Love." The name "Transformer" was chosen by Jakob Uszkoreit, who, according to a Wired account, liked the sound of the word; an early design document was titled "Transformers: Iterative Self-Attention and Processing for Various Tasks" and the working group was named Team Transformer (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need) (Source: https://www.wired.com/story/eight-google-employees-invented-modern-ai-transformers-paper/).
The Transformer built on earlier work, including the additive attention mechanism for sequence-to-sequence models introduced by Bahdanau et al. in 2014, and the encoder-decoder sequence-transduction framework developed in the early 2010s. Prior sequence models relied on recurrent neural networks such as the LSTM (formally published in 1997), which used sequential, one-token-at-a-time processing and could not operate in parallel across a sequence (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need).
Reception and influence
As of 2025 the paper had been cited more than 173,000 times, which Wikipedia, citing external reporting, places among the ten most-cited papers of the 21st century (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need). A University of Southern California Information Sciences Institute account reported the paper had received more than 60,000 Google Scholar citations as of its 2023 publication date (Source: https://www.isi.edu/news/54564/attention-is-all-you-need-usc-alumni-paved-path-for-chatgpt/).
After the paper was published, each of the eight authors left Google to join other companies or found startups (Source: https://en.wikipedia.org/wiki/Attention_Is_All_You_Need). Reported subsequent affiliations include Ashish Vaswani and Niki Parmar co-founding Essential AI; Noam Shazeer co-founding Character.AI before returning to Google to co-lead Gemini technical work; Llion Jones co-founding Sakana AI in Tokyo; Aidan Gomez co-founding Cohere; and Illia Polosukhin co-founding the NEAR Protocol blockchain project (Source: https://www.wired.com/story/eight-google-employees-invented-modern-ai-transformers-paper/) (Source: https://www.nec.com/en/press/202410/images/1501-01-02.pdf).
Relation to scaling and later work
The Transformer's scalability — a steeper scaling slope than LSTMs — is the architectural basis on which empirical scaling laws operate, and the shift from LSTMs to Transformers is described as one of the largest single sources of AI software progress (Scaling Laws, AI software progress). The architecture is also cited as supporting evidence for The Bitter Lesson, in which a general architecture that leverages computation outperformed specialized approaches. The Transformer was the starting point for the scaling era documented in Patel's oral history (The Scaling Era).
Provenance
PDF converted to markdown with images on 2026-04-13. Canonical source: arXiv:1706.03762, NeurIPS 2017 (Source: https://arxiv.org/abs/1706.03762).