Exploring Simple Siamese Representation Learning

The paper investigates Siamese network architectures for unsupervised visual representation learning, where a model maximizes agreement between two augmented views of the same image. The authors strip away components commonly assumed necessary to avoid representational collapse, testing networks that use no negative sample pairs, no large batches, and no momentum encoders. They find that a stop-gradient operation is the key ingredient preventing collapse, and offer a hypothesis with proof-of-concept experiments about why it works.

Their SimSiam method achieves competitive accuracy on ImageNet and on downstream transfer tasks despite its simplicity. By showing that meaningful representations emerge without the usual machinery, the work reframes what Siamese architectures actually need and provides a minimal baseline that motivated rethinking of self-supervised learning design.