Understanding deep learning requires rethinking generalization

This work investigates why deep neural networks generalize well despite having far more parameters than training examples. Through extensive systematic experiments it tests conventional explanations that attribute small generalization error to the model family or to regularization techniques, including training convolutional image classifiers on data with randomized labels or images replaced by completely unstructured random noise.

The experiments establish that state-of-the-art networks can perfectly fit random labelings, a phenomenon qualitatively unaffected by explicit regularization, and a theoretical construction shows even depth-two networks have perfect finite-sample expressivity as soon as parameters exceed data points. These findings showed that traditional approaches cannot explain deep learning's generalization, reshaping how the community thinks about the problem.