Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

This paper introduces the Sparsely-Gated Mixture-of-Experts (MoE) layer to realize the long-standing promise of conditional computation, in which different parts of a network are active for different examples so that model capacity can grow without a proportional increase in computation. The layer consists of up to thousands of feed-forward sub-networks, or experts, and a trainable gating network that determines a sparse combination of experts to use for each example. The authors address the algorithmic and performance challenges that had previously prevented conditional computation from working in practice.

Applied convolutionally between stacked LSTM layers, the MoE reaches up to 137 billion parameters and achieves greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. On large language modeling and machine translation benchmarks, where capacity is critical for absorbing vast training knowledge, the MoE models significantly outperform the state of the art at lower computational cost, demonstrating that conditional computation can scale neural networks dramatically.