HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

HuBERT (Hidden-Unit BERT) is a self-supervised approach to speech representation learning that confronts three difficulties unique to speech: each utterance contains multiple sound units, there is no lexicon of input sound units during pre-training, and sound units have variable lengths with no explicit segmentation. The method uses an offline clustering step to generate aligned target labels for a BERT-like prediction loss, and crucially applies that loss only over masked regions, forcing the model to learn a combined acoustic and language model over the continuous speech input.

A key insight is that HuBERT depends on the consistency of the unsupervised clustering rather than the intrinsic quality of the cluster labels, so even a simple k-means teacher with 100 clusters suffices when refined over two clustering iterations. HuBERT matches or improves upon the state-of-the-art wav2vec 2.0 on the LibriSpeech (960h) and Libri-light (60,000h) benchmarks across fine-tuning subsets from 10 minutes to 960 hours, and a 1B-parameter model yields up to 19% and 13% relative word error rate reductions on the challenging dev-other and test-other sets.