Sequence to Sequence Learning with Neural Networks

This paper presents a general end-to-end approach to sequence learning that makes minimal assumptions about sequence structure, addressing the limitation that deep neural networks could not map sequences to sequences. The method uses a multilayered Long Short-Term Memory (LSTM) network to map the input sequence to a vector of fixed dimensionality, and a second deep LSTM to decode the target sequence from that vector.

On the WMT-14 English-to-French translation task, the LSTM achieved a BLEU score of 34.8 on the full test set despite penalties on out-of-vocabulary words, surpassing a phrase-based SMT system's 33.3, and reached 36.5 — close to the prior state of the art — when used to rerank the SMT system's 1000 hypotheses. The LSTM handled long sentences without difficulty, learned representations sensitive to word order and relatively invariant to active versus passive voice, and benefited markedly from reversing the source word order, which introduced short-term dependencies that made optimization easier.