Learning Important Features Through Propagating Activation Differences

This paper introduces DeepLIFT (Deep Learning Important FeaTures), a method for interpreting neural network predictions, motivated by the fact that the purported black-box nature of neural networks is a barrier to adoption where interpretability is essential. DeepLIFT decomposes the output prediction of a network on a specific input by backpropagating the contributions of all neurons in the network to every feature of the input. It compares the activation of each neuron to a reference activation and assigns contribution scores according to the difference, and by optionally giving separate consideration to positive and negative contributions it can reveal dependencies missed by other approaches.

A practical benefit is efficiency: the contribution scores can be computed in a single backward pass. The authors apply DeepLIFT to models trained on MNIST and on simulated genomic data, showing significant advantages over gradient-based methods, both in the informativeness of the attributions and in exposing dependencies those methods overlook.