Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

In deep learning, models typically reuse the same parameters for all inputs, but Mixture of Experts (MoE) instead selects different parameters for each incoming example, producing a sparsely activated model with an enormous number of parameters yet a constant computational cost. Despite several notable successes, MoE adoption has been hindered by complexity, communication costs, and training instability. The Switch Transformer addresses these obstacles by simplifying the MoE routing algorithm and designing models with reduced communication and computational costs, together with training techniques that help control the resulting instabilities.

These improvements allow the authors to train large sparse models for the first time using lower-precision bfloat16 formats, and, building on T5-Base and T5-Large, they obtain up to 7x increases in pre-training speed at the same computational resources. The gains extend to multilingual settings, with improvements over mT5-Base measured across all 101 languages, and the approach advances the frontier of model scale by pre-training models up to a trillion parameters on the Colossal Clean Crawled Corpus while achieving a 4x speedup over the T5-XXL model, showing that sparsity can make extreme scale practical.