Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

This paper identifies internal covariate shift, the phenomenon where the distribution of each layer's inputs keeps changing during training as previous layers' parameters are updated, as a key factor that slows down and complicates deep network training by forcing low learning rates and careful initialization. The authors address this by introducing Batch Normalization, which makes normalizing layer inputs part of the model architecture itself, performing the normalization using statistics computed over each training mini-batch.

Batch Normalization allows networks to use much higher learning rates and be less sensitive to initialization, and in some cases eliminates the need for Dropout regularization. Applied to a state-of-the-art image classification model, it reaches the same accuracy using 14 times fewer training steps and improves upon the original model by a significant margin, and an ensemble of batch-normalized networks achieves 4.82% top-5 error on ImageNet classification, surpassing the accuracy of human raters, establishing Batch Normalization as a standard component of deep network training.