Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Foundation models are almost universally built on the Transformer and its attention module, and subquadratic alternatives such as linear attention, gated convolution, recurrent models, and structured state space models (SSMs) had not matched attention on modalities like language. The authors identify content-based reasoning as the key weakness and make the SSM parameters functions of the input, letting the model selectively propagate or forget information along the sequence depending on the current token. Because this breaks efficient convolutions, they design a hardware-aware parallel algorithm in recurrent mode and fold the selective SSM into a simplified architecture with no attention or MLP blocks, called Mamba.

Mamba delivers fast inference with 5x higher throughput than Transformers, scales linearly in sequence length, and keeps improving on real data up to million-length sequences. As a general sequence backbone it reaches state-of-the-art results across language, audio, and genomics, and the Mamba-3B model outperforms Transformers of its own size while matching Transformers twice as large in both pretraining and downstream evaluation.