On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

This paper studies neural machine translation, a then-new approach to statistical machine translation based purely on neural networks in which an encoder extracts a fixed-length representation from a variable-length input sentence and a decoder produces the translation from that representation. The authors analyze the behaviour of two models: the RNN Encoder-Decoder and a newly proposed gated recursive convolutional neural network.

They show that neural machine translation performs relatively well on short sentences that contain no unknown words, but its performance degrades rapidly as sentence length and the number of unknown words increase. They also find that the proposed gated recursive convolutional network automatically learns a grammatical structure of a sentence, an early insight into the strengths and limitations of neural translation models.