Neural Machine Translation by Jointly Learning to Align and Translate

The paper addresses a limitation in the standard neural machine translation encoder-decoder architecture, where an encoder compresses an entire source sentence into a single fixed-length vector from which a decoder must generate the full translation. The authors conjecture that this fixed-length vector acts as a bottleneck constraining translation quality. Their core method extends the basic encoder-decoder architecture by allowing the model to automatically and softly search for the parts of the source sentence relevant to predicting each target word, rather than requiring these relevant parts to be explicitly segmented as hard, fixed spans.

Using this soft-search (attention) mechanism, the model reaches translation performance comparable to the existing state-of-the-art phrase-based statistical machine translation system on English-to-French translation, and qualitative analysis shows that the soft alignments the model discovers between source and target words agree well with human intuition. This mattered because it removed a key architectural bottleneck in neural machine translation and introduced what became the attention mechanism, a foundational building block for later sequence models.