Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network

This paper tackles single-image and video super-resolution with deep networks. The authors observe that earlier deep methods first upscale the low-resolution input to high-resolution space using a single fixed filter, commonly bicubic interpolation, and then perform reconstruction in that high-resolution space. They argue this is sub-optimal and adds computational complexity. Instead, they propose a CNN architecture that extracts feature maps directly in the low-resolution space and introduce an efficient sub-pixel convolution layer that learns an array of upscaling filters to map the final low-resolution feature maps to the high-resolution output.

By learning upscaling filters tailored to each feature map rather than relying on a handcrafted bicubic filter, and by reducing the overall computational cost, the method becomes the first CNN capable of real-time super-resolution of 1080p video on a single K2 GPU. On public image and video datasets it improves reconstruction quality by +0.15dB on images and +0.39dB on videos while running an order of magnitude faster than previous CNN-based methods, making high-quality super-resolution practical for real-time use.