Fully convolutional networks for semantic segmentation

The paper demonstrates that convolutional networks, powerful visual models that yield hierarchies of features, can themselves be trained end-to-end, pixels-to-pixels, to exceed the state of the art in semantic segmentation. Its key insight is building 'fully convolutional' networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning; the authors define this space of networks, explain their application to spatially dense prediction tasks, and adapt contemporary classification networks including AlexNet, VGG net, and GoogLeNet into fully convolutional form by fine-tuning their learned representations for segmentation.

The authors further define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce more accurate and detailed segmentations. The resulting fully convolutional network achieves state-of-the-art segmentation results on PASCAL VOC (a 20% relative improvement to 62.2% mean IU on the 2012 set), NYUDv2, and SIFT Flow, while inference takes less than one fifth of a second for a typical image.