Pointer Sentinel Mixture Models

Neural sequence models with softmax classifiers achieve their best language modeling performance only when they use very large hidden states and vocabularies, and even then they struggle to predict rare or unseen words even when the surrounding context makes the correct word unambiguous. To address this, the paper introduces a pointer sentinel mixture architecture that gives a neural sequence model two ways to generate the next word: it can reproduce a word directly from the recent context using a pointer component, or it can produce a word from a standard softmax classifier, with a sentinel governing how the two are combined.

Instantiated as a pointer sentinel-LSTM, the model achieves state-of-the-art language modeling performance on the Penn Treebank, reaching 70.9 perplexity while using far fewer parameters than a standard softmax LSTM, which indicates the pointer mechanism handles rare words more efficiently. To better evaluate how well language models exploit longer contexts and cope with more realistic vocabularies and larger corpora, the authors also introduce and freely release the WikiText corpus, providing a new resource for the community.