Long-term recurrent convolutional networks for visual recognition and description

The paper investigates whether models that are recurrent as well as deep, or 'temporally deep,' are effective for tasks involving sequences, visual and otherwise. It proposes long-term recurrent convolutional networks (LRCN), an end-to-end trainable architecture that connects convolutional networks with recurrent units so that convolutional perceptual representations and temporal dynamics are learned jointly. Because nonlinearities are incorporated into the network's state updates, the models can learn long-term dependencies and map variable-length inputs, such as video frames, to variable-length outputs, such as natural language text.

Unlike prior approaches that assume a fixed spatio-temporal receptive field or rely on simple temporal averaging, LRCN models are 'doubly deep,' being compositional across both spatial and temporal layers, which may be advantageous when target concepts are complex or training data are limited. The authors demonstrate the architecture on benchmark video recognition, image description and retrieval, and video narration tasks, reporting distinct advantages over state-of-the-art models that are separately defined or optimized for recognition or generation.