This ICLR 2015 paper by Dzmitry Bahdanau (Jacobs University Bremen), KyungHyun Cho, and Yoshua Bengio (Université de Montréal) introduces the attention mechanism. It is the direct antecedent of "Attention Is All You Need" (2017), which removed the recurrence and kept the attention.
The bottleneck argument
The paper's target is the then-standard encoder-decoder design, which "encode[s] a source sentence into a fixed-length vector from which a decoder generates a translation."
Its central claim is stated as a conjecture rather than a proof: "we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder–decoder architecture." The information content of a sentence grows with its length; the vector's capacity does not.
The mechanism
The proposed extension is to allow "a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly."
The parenthetical "(soft-)" carries the technical weight. Alignment in statistical machine translation had been a discrete correspondence between source and target segments; here it becomes a differentiable weighting over all source positions, computed afresh for each target word. That is what makes it trainable end to end by gradient descent, and it is the operation that generalizes into self-attention three years later.
Results
The models are evaluated on English-to-French using WMT '14, with news-test-2012 and news-test-2013 as the development set and news-test-2014 (3,003 unseen sentences) as the test set, under a 30,000-word shortlist per language with out-of-vocabulary words mapped to an [UNK] token and no lowercasing or stemming.
Two model families are compared: RNNenc, the fixed-vector encoder-decoder baseline, and RNNsearch, the attention model, each trained on sentences up to 30 and up to 50 words.
The headline result is parity with the non-neural state of the art: "we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation."
The more diagnostic result concerns where the gain comes from. BLEU scores plotted against sentence length show the baseline degrading as sentences lengthen while the attention models hold up — which is what the bottleneck conjecture predicts, and is stronger evidence for the mechanism than the aggregate score is.
The paper adds a qualitative check: "the (soft-)alignments found by the model agree well with our intuition" — the learned attention weights recover recognizable word correspondences, including reordering between English and French, without ever being supervised on alignments.
Relationships
- depends-on: Attention Is All You Need — the transformer keeps this paper's attention mechanism and discards the recurrence around it
- related: Yoshua Bengio — co-author
- related: Scaling Laws, AI Alignment — unrelated senses of "alignment"; this paper's usage is the machine-translation one, source-to-target word correspondence