On the difficulty of training recurrent neural networks

This paper investigates the two widely known obstacles to properly training recurrent neural networks: the vanishing gradient and exploding gradient problems, originally detailed by Bengio et al. in 1994. To improve understanding of the underlying causes, the authors examine these problems from three complementary angles, an analytical perspective, a geometric perspective, and a dynamical systems perspective. This analysis is then used to justify a simple yet effective solution.

Concretely, the authors propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint to address the vanishing gradient problem. They validate both their hypotheses about the causes and their proposed solutions empirically. Because exploding and vanishing gradients were central obstacles to learning long-range dependencies, these remedies offered a practical way to stabilize the training of recurrent networks.