Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

The paper studies how to train models containing stochastic neurons or hard, non-smooth non-linearities, where the central difficulty is estimating the gradient of a loss function with respect to such units' inputs—that is, whether one can back-propagate through them. It examines the problem and compares four families of solutions applicable in different settings: a minimum-variance unbiased gradient estimator for stochastic binary neurons (a special case of REINFORCE); a new decomposition of a binary stochastic neuron into a stochastic binary part and a smooth differentiable part that approximates its expected effect to first order; the injection of additive or multiplicative noise into an otherwise differentiable computational graph; and a straight-through estimator that heuristically copies the gradient at the stochastic output straight back to the sigmoid argument.

To ground these estimators, the authors consider a small-scale form of conditional computation, in which sparse stochastic units act as gaters that can switch off, in combinatorially many ways, large chunks of the rest of the network's computation. Because it is important that these gaters actually output zero most of the time, the resulting sparsity can in principle be exploited to greatly reduce the computational cost of large deep networks. The paper's comparison—especially the straight-through estimator—provided practical tools for training with discrete, stochastic components.