GloVe: Global Vectors for Word Representation

The paper addresses the question of why vector-arithmetic regularities (like analogies) emerge in learned word representations, since earlier methods achieved this empirically without explaining the underlying mechanism. The authors analyze what model properties are required for such regularities to emerge and use this analysis to design GloVe, a new global log-bilinear regression model. GloVe merges two previously separate families of approaches: global matrix factorization methods, which use corpus-wide statistics, and local context window methods, which learn from small windows of text. Its key efficiency trick is training only on the nonzero entries of a word-word co-occurrence matrix, avoiding the cost of processing the entire sparse matrix or scanning every context window in a large corpus.

The resulting word vectors exhibit meaningful linear substructure, reaching 75% accuracy on a recent word analogy benchmark, and the model also outperforms related embedding methods on word similarity tasks and on named entity recognition. This mattered because it showed that combining global co-occurrence statistics with an efficient, targeted training scheme could match or beat both matrix-factorization and window-based embedding methods simultaneously, and it offered a principled explanation for the analogy-preserving structure that had previously been observed but not well understood.