Learning both Weights and Connections for Efficient Neural Network

Neural networks are both computationally and memory intensive, which makes them difficult to deploy on embedded systems, and conventional networks fix their architecture before training so training cannot improve it. To address this, the authors propose reducing the storage and computation a network requires by learning only its important connections, using a three-step procedure: first train the network to learn which connections are important, then prune the unimportant connections, and finally retrain to fine-tune the weights of the connections that remain.

The method reduces network size by an order of magnitude without any loss of accuracy. On ImageNet it shrank AlexNet from 61 million to 6.7 million parameters (a 9x reduction) and VGG-16 from 138 million to 10.3 million parameters (a 13x reduction), demonstrating that much of a network's parameters are redundant and can be removed while preserving accuracy.