Supervised Contrastive Learning

Cross-entropy is the most widely used loss function for supervised training of image classification models, but this paper proposes a training methodology that consistently outperforms it across different architectures and data augmentations. The approach modifies the batch contrastive loss—recently shown to be very effective for learning powerful representations in the self-supervised setting—so that it can leverage label information: clusters of points belonging to the same class are pulled together in embedding space while clusters from different classes are pushed apart. The method also relies on key ingredients such as large batch sizes and normalized embeddings.

On both ResNet-50 and ResNet-200, the supervised contrastive loss outperforms cross-entropy by over 1%, setting a new state-of-the-art of 78.8% among methods that use AutoAugment data augmentation. The loss additionally shows clear benefits for robustness to natural corruptions on standard benchmarks, improving both calibration and accuracy, and it is more stable than cross-entropy across hyperparameter choices such as optimizers and data augmentations, making it an attractive alternative for supervised representation learning.