Improved Baselines with Momentum Contrastive Learning

This short note builds on recent progress in contrastive unsupervised learning, specifically Momentum Contrast (MoCo) and SimCLR. The authors take two of SimCLR's design improvements, an MLP projection head and more aggressive data augmentation, and implement them within the MoCo framework to test their effectiveness in a controlled way.

With these simple modifications, the authors establish stronger baselines that outperform SimCLR, and importantly they do so without requiring the large training batches that SimCLR depends on. This makes state-of-the-art unsupervised representation learning more accessible to researchers with limited computational resources, and the authors state that code will be made public.