Group Normalization

Batch Normalization is a milestone technique that enabled many networks to train, but normalizing along the batch dimension introduces problems: its error increases rapidly as the batch size becomes small because batch statistics are estimated inaccurately, which limits BN for training larger models and for transferring features to computer vision tasks such as detection, segmentation, and video that require small batches due to memory constraints. The paper presents Group Normalization as a simple alternative that divides the channels into groups and computes the mean and variance within each group for normalization.

Because Group Normalization's computation is independent of batch size, its accuracy stays stable across a wide range of batch sizes. On ResNet-50 trained on ImageNet, GN has 10.6% lower error than its BN counterpart at a batch size of 2, and it is comparably good with BN and outperforms other normalization variants at typical batch sizes. GN transfers naturally from pre-training to fine-tuning and outperforms BN-based counterparts for COCO object detection and segmentation and Kinetics video classification, and it can be implemented in a few lines of code.