SimCSE: Simple Contrastive Learning of Sentence Embeddings

SimCSE is a simple contrastive learning framework for producing sentence embeddings, presented in both unsupervised and supervised forms. The unsupervised approach takes an input sentence and predicts itself within a contrastive objective, relying only on standard dropout as the source of noise; the authors find that this dropout functions as minimal data augmentation and that removing it leads to representation collapse. The supervised approach extends the framework by incorporating annotated pairs from natural language inference datasets, treating entailment pairs as positives and contradiction pairs as hard negatives.

Evaluated on standard semantic textual similarity (STS) tasks, the unsupervised and supervised models built on BERT-base achieved average Spearman's correlations of 76.3% and 81.6% respectively, improvements of 4.2% and 2.2% over the previous best results, with the unsupervised model performing on par with earlier supervised methods. The authors also show, both theoretically and empirically, that the contrastive objective regularizes the anisotropic space of pre-trained embeddings to be more uniform and better aligns positive pairs when supervised signals are available, helping explain the method's effectiveness.