Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

This tutorial addresses a pedagogical gap in the RNN and LSTM literature, where inference formulas are often stated axiomatically, training formulas omitted, and the 'unrolling' of recurrent networks presented without justification. Drawing from concepts in signal processing, the author formally derives the canonical RNN formulation from differential equations and proposes and proves a precise statement that yields the RNN unrolling technique.

Through a series of logical arguments, the author addresses the difficulties of training standard RNNs by transforming the RNN into a 'Vanilla LSTM' network, providing all of the system's equations with detailed descriptions of its components. The analysis also identifies new opportunities to enrich the LSTM and incorporates these extensions into the Vanilla LSTM, producing what the author describes as the most general LSTM variant to date, aimed at practitioners and researchers implementing the model.