Context Encoders: Feature Learning by Inpainting

The paper introduces Context Encoders, an unsupervised visual feature learning algorithm driven by context-based pixel prediction. By analogy with auto-encoders, a convolutional neural network is trained to generate the contents of an arbitrary image region conditioned on its surroundings, which requires the model to understand the content of the entire image and produce a plausible hypothesis for the missing parts. The authors experiment with both a standard pixel-wise reconstruction loss and a reconstruction plus adversarial loss.

The combined reconstruction-plus-adversarial loss produces much sharper results because it can better handle multiple modes in the output. The learned representation captures not just appearance but also the semantics of visual structures, and the authors quantitatively demonstrate its effectiveness for CNN pre-training on classification, detection, and segmentation tasks. Context encoders can also be used for semantic inpainting, either standalone or as an initialization for non-parametric methods.