Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Deep learning benefits from large networks and large datasets, but these also lead to long training times that slow research and development progress. This paper explores distributed synchronous SGD, which divides each minibatch over a pool of parallel workers; to be efficient, the per-worker workload must be large, which implies nontrivial growth in the overall minibatch size. The authors empirically show that on ImageNet very large minibatches initially cause optimization difficulties, but that these can be overcome, after which the trained networks still generalize well.

Their central techniques are a hyper-parameter-free linear scaling rule that adjusts the learning rate as a function of minibatch size, together with a new warmup scheme that handles the optimization challenges early in training. With these simple methods, they show no loss of accuracy at minibatch sizes up to 8192, and their Caffe2-based system trains ResNet-50 with a minibatch of 8192 on 256 GPUs in one hour while matching small-minibatch accuracy. The implementation achieves about 90% scaling efficiency when moving from 8 to 256 GPUs, enabling efficient training of visual recognition models on internet-scale data.