Neural Discrete Representation Learning

The paper tackles unsupervised representation learning by proposing the Vector Quantised-Variational AutoEncoder (VQ-VAE), a generative model that differs from standard VAEs in two ways: its encoder network outputs discrete rather than continuous codes, and its prior is learned rather than held static. To obtain a discrete latent representation, the model borrows ideas from vector quantisation, mapping encoder outputs to a learned codebook of embeddings.

This discretisation lets the model circumvent posterior collapse, the failure mode in which the latents are ignored when combined with a powerful autoregressive decoder. Pairing the learned discrete representations with an autoregressive prior, VQ-VAE generates high-quality images, videos, and speech, performs high-quality speaker conversion, and discovers phonemes through unsupervised learning, demonstrating the usefulness of the learned codes.