Recurrent Models of Visual Attention

The paper introduces a recurrent neural network model of visual attention that addresses the high computational cost of applying convolutional networks to large images, where computation scales linearly with the number of pixels. Instead of processing the whole image, the model adaptively selects a sequence of regions or locations and processes only those selected regions at high resolution. Like convolutional networks it has a degree of built-in translation invariance, but the amount of computation it performs can be controlled independently of the input image size.

Because the region-selection model is non-differentiable, it is trained using reinforcement learning to learn task-specific policies. On several image classification tasks it significantly outperforms a convolutional neural network baseline on cluttered images, and on a dynamic visual control problem it learns to track a simple object without any explicit training signal for tracking. This established a foundational framework for hard visual attention and glimpse-based processing.