Conformer: Convolution-augmented Transformer for Speech Recognition

This paper proposes the Conformer, a convolution-augmented Transformer architecture for automatic speech recognition. The authors observe that Transformer models are good at capturing content-based global interactions across a sequence, while convolutional neural networks effectively exploit local features. Conformer is designed to get the best of both worlds by combining convolution and self-attention to model both local and global dependencies of an audio sequence in a parameter-efficient way.

Conformer significantly outperforms the previous Transformer- and CNN-based models and achieves state-of-the-art accuracy. On the widely used LibriSpeech benchmark it reaches word error rates of 2.1%/4.3% on test/test-other without a language model and 1.9%/3.9% with an external language model. Even a small model with only 10 million parameters attains a competitive 2.7%/6.3%, reflecting the parameter efficiency of combining convolution and self-attention to capture both local and global audio dependencies.