LSTM: A Search Space Odyssey

Motivated by the many long short-term memory (LSTM) variants proposed since the architecture's introduction in 1995 and its rise to state-of-the-art status, the authors carry out the first large-scale empirical analysis of eight LSTM variants. They evaluate the variants on three representative tasks, speech recognition, handwriting recognition, and polyphonic music modeling, optimizing each variant's hyperparameters separately with random search and assessing hyperparameter importance using the functional ANalysis Of VAriance (fANOVA) framework. In total the study summarizes 5400 experimental runs, roughly 15 years of CPU time, making it the largest of its kind on LSTM networks.

The results show that none of the eight variants improves significantly upon the standard LSTM architecture, and they identify the forget gate and the output activation function as its most critical components. The authors further observe that the studied hyperparameters are virtually independent of one another and use this to derive practical guidelines for tuning them efficiently, providing empirically grounded guidance for practitioners working with LSTMs.