Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

While deep convolutional networks had achieved great success on still images, their advantage over traditional methods for video action recognition remained limited. This paper aims to identify principles for designing ConvNet architectures that work well for action recognition even with limited training data, and its central contribution is the temporal segment network (TSN), a framework built on long-range temporal structure modeling that combines a sparse temporal sampling strategy with video-level supervision to learn efficiently from entire action videos.

Alongside the architecture, the authors investigate a series of good practices for training ConvNets on video data. The resulting approach achieves state-of-the-art performance, reaching 69.4% on HMDB51 and 94.2% on UCF101, and visualizations of the learned models qualitatively illustrate why the temporal segment network and its accompanying practices are effective.