Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

This paper brings together deep convolutional neural networks and probabilistic graphical models to tackle pixel-level classification, or semantic image segmentation. The authors show that the very invariance properties that make DCNNs excellent at high-level tasks like classification and detection also make their final-layer responses too coarsely localized for precise segmentation. To fix this, they combine the final DCNN layer's responses with a fully connected Conditional Random Field, producing the DeepLab system that recovers sharp object boundaries.

DeepLab localizes segment boundaries at an accuracy beyond previous methods and set a new state of the art on the PASCAL VOC-2012 segmentation benchmark, reaching 71.6% IOU on the test set. The authors also made inference efficient: careful re-purposing of the network together with a novel application of the 'hole' algorithm from the wavelet community allowed dense computation of network responses at 8 frames per second on a GPU. The work established the influential CNN-plus-CRF approach that shaped subsequent semantic segmentation research.