The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Neural network pruning can reduce a trained model's parameter count by more than 90% without losing accuracy, yet the sparse architectures it produces are notoriously difficult to train from scratch. Investigating this, the authors find that standard pruning naturally uncovers subnetworks whose original initial weights made them especially trainable, and they formalize this as the lottery ticket hypothesis: a dense, randomly-initialized feed-forward network contains smaller subnetworks (winning tickets) that can be trained in isolation to comparable accuracy in a similar number of iterations. They also present an algorithm for identifying these winning tickets.

Across fully-connected and convolutional architectures on MNIST and CIFAR10, the authors consistently identify winning tickets that are less than 10-20% of the original network's size, and above that size the tickets they find learn faster and reach higher test accuracy than the full network. The finding suggests that a network's trainability depends on fortuitous initializations of particular subnetworks, reframing how sparsity and initialization are understood in deep learning.