GloVe: Global Vectors for Word Representation

Problem

Framing

Count models used global corpus statistics but missed the linear regularities seen in predictive embeddings. GloVe closes this gap with a weighted log-bilinear fit to co-occurrence counts, reaching 75.0% analogy accuracy with 300d vectors on 42B tokens.

Currently Used Methods

Foundational

• @mikolovWord2vec2013 — predictive CBOW and skip-gram embeddings with strong analogy structure.
  • Limitation in context: uses local windows, not explicit global co-occurrence statistics.
• Neural Probabilistic Language Model — neural embeddings from next-word prediction.
  • Limitation in context: expensive normalization, no direct count factorization.
• A Unified Architecture for Natural Language Processing — full-context neural word representations for NLP.
  • Limitation in context: task-tied training, not simple unsupervised corpus statistics.
• Improving Distributional Similarity with Lessons Learned from Word Embeddings — SVD-style count baselines for distributional semantics.
  • Limitation in context: weaker analogy structure than predictive embeddings.
• Finding Structure in Time — Hellinger PCA count-based embeddings.
  • Limitation in context: lower semantic and overall analogy accuracy.

Proposed Method

Architecture

GloVe learns word vectors $\mathbf{w}_i$, context vectors $\tilde{\mathbf{w}}_j$, and biases $b_i, \tilde{b}_j$. It regresses $\log X_{ij}$ from a bilinear score and uses $\mathbf{w}_i + \tilde{\mathbf{w}}_i$ as the final embedding.

Loss / Objective

The model minimizes a weighted least-squares fit over nonzero co-occurrences.

Algorithm

After optimization, the final representation sums the two learned embeddings.

Training Procedure

• Corpora: 1.5B, 6B, and 42B tokens.
• Dimensions: 50, 100, 300, 1000.
• Window size: 10 in key sweeps.
• Weighting: $x_{\max}=100$, $\alpha=3/4$.
• Optimizer: AdaGrad.

Evaluation

Datasets

• Word analogy.
• Word similarity.
• Named entity recognition.
• Training corpora: 1.5B, 6B, 42B tokens.

Metrics

• Analogy accuracy: semantic, syntactic, overall.
• Word similarity: Spearman correlation.
• NER: $F_1$.

Headline results

• Analogy, 100d, 1.6B: 67.5 semantic, 54.3 syntactic, 60.3 overall.
• Analogy, 300d, 1.6B: 80.8 semantic, 61.5 syntactic, 70.3 overall.
• Analogy, 300d, 6B: 77.4 semantic, 67.0 syntactic, 71.7 overall.
• Analogy, 300d, 42B: 81.9 semantic, 69.3 syntactic, 75.0 overall.
• NER, 50d: 85.0 $F_1$.

Table 1: Word analogy accuracy across models, dimensions, and corpus sizes

Ablations

• Vector size: accuracy rises with dimension, then saturates.
• Window size: larger windows help semantic analogies more.
• Context type: symmetric helps semantic; asymmetric helps syntactic.
• Training time: GloVe reaches target accuracy faster than CBOW and skip-gram.

Method Strengths and Weaknesses

Strengths

• Combines global counts with linear analogy structure.
• Avoids softmax normalization with sparse weighted regression.
• Best reported analogy score is 75.0 overall at 42B tokens.
• Summing word and context vectors improves results.

Weaknesses

• Requires explicit construction of the co-occurrence matrix.
• Weighting choices $x_{\max}$ and $\alpha$ are empirical.
• Learns static type embeddings, not token-specific meanings.
• Similarity gains are smaller than analogy gains.

Suggestions from the authors

• Compare count and prediction models under matched compute budgets.
• Explain why predictive models outperform many count baselines.
• Develop principled weighting and multi-pass training schemes.
• Study vector-space structure beyond scalar similarity.

Links

Prior Papers

• @mikolovWord2vec2013 — predictive embeddings are the direct baseline GloVe matches and often exceeds on analogies.

Further Papers

• @petersELMo2018 — contextual token representations address GloVe's static embedding limitation.
• @devlinBERT2018 — masked-language-model pretraining supersedes static word vectors on downstream NLP.
• @radfordGPT2018 — autoregressive contextual representations shift modeling from word types to token states.