FlowNet: Learning Optical Flow with Convolutional Networks

FlowNet addresses optical flow estimation, a task where convolutional neural networks had not previously succeeded despite their strong performance on recognition problems. The authors construct CNNs that solve optical flow as a supervised learning task and propose two architectures: a generic network and an alternative that includes a correlation layer explicitly matching feature vectors at different image locations. Because existing ground-truth flow datasets were too small to train such networks, they generated a large synthetic dataset of flying chairs on random backgrounds.

Despite being trained on this unrealistic synthetic data, the networks generalized well to established benchmarks such as Sintel and KITTI, achieving competitive accuracy while running at frame rates of 5 to 10 frames per second. This demonstrated that optical flow could be learned directly by CNNs and that synthetic data could effectively train them, helping open the direction of learning-based optical flow estimation.