The Curious Case of Neural Text Degeneration

Problem

Framing

Open-ended text generation lacked a decoding rule that avoids both beam-search repetition and unconstrained-sampling incoherence. The paper closes this gap with nucleus sampling, which truncates the unreliable tail adaptively and attains the best combined human-statistical score, HUSE $0.97$.

Currently Used Methods

Direct antecedents

• @vaswaniAttentionAllNeed2017 — Transformer architecture behind large neural language models.
  • Limitation in context: architecture strength does not determine a good open-ended decoding rule.
• @radfordGPT2_2019 — large-scale autoregressive language model with top-$k$ decoding.
  • Limitation in context: fixed $k$ truncation mismatches context-dependent distribution shape.
• "A Hierarchical Neural Autoencoder for Paragraphs and Documents" — stochastic beam search for diverse decoding.
  • Limitation in context: beam-style maximization still drifts toward bland or repetitive continuations.
• "Hierarchical Neural Story Generation" — top-$k$ sampling for long-form generation.
  • Limitation in context: constant-$k$ sampling either admits tail noise or over-truncates.
• "Language Modeling with Gated Convolutional Networks" — alternative neural LM backbone for next-token prediction.
  • Limitation in context: better LMs still expose an unreliable low-probability tail at decoding time.

Proposed Method

Architecture

The method is a decoding rule layered on an autoregressive LM, not a new network. At step $i$, it reads the model distribution $P(x_i \mid x_{1:i-1})$ and adaptively truncates to a context-specific candidate set before sampling.

Loss / Objective

The underlying LM still optimizes standard autoregressive likelihood:

Sampling Rule / Algorithm

Nucleus sampling defines the smallest top-probability set whose cumulative mass exceeds $p$, then renormalizes and samples from it.

Training Procedure

• LM family: GPT-2.
• Model scales discussed: Large and XL for examples.
• Evaluation corpora: WebText prompts, WritingPrompts, WikiText-103.
• Generation length for loop analysis: first $200$ tokens.
• HUSE classifier: KNN with $k=13$.

Evaluation

Datasets

• WebText continuations.
• WritingPrompts generations.
• WikiText-103 reference text.
• Human-written continuations as calibration target.

Metrics

• Perplexity.
• Self-BLEU4.
• Zipf coefficient.
• Repetition $\%$.
• HUSE.

Headline results

• Human reference: perplexity $12.38$, Self-BLEU4 $0.31$, Zipf $0.93$, repetition $0.28$.
• Greedy: perplexity $1.50$, repetition $73.66\%$.
• Beam, $b=16$: perplexity $1.48$, repetition $28.94\%$.
• Top-$k=640$: Self-BLEU4 $0.32$, repetition $0.28\%$, HUSE $0.94$.
• Nucleus, $p=0.95$: perplexity $13.13$, Self-BLEU4 $0.32$, repetition $0.36\%$, HUSE $0.97$.

Table 1: Main results for comparing decoding methods with selected parameters

Ablations

• Beam width: larger beams push perplexity unnaturally low and worsen repetition loops.
• Temperature: lower $t$ improves coherence but collapses diversity and still repeats.
• Top-$k$: small $k$ over-truncates; large $k$ restores diversity but admits tail noise.
• Top-$p$: values in $[0.9,1)$ best match human Self-BLEU and keep repetition low.

Method Strengths and Weaknesses

Strengths

• One-parameter rule. No retraining or architecture change.
• Best overall HUSE: nucleus $p=0.95$ reaches $0.97$.
• Matches human perplexity far better than beam or greedy decoding.
• Avoids both dominant failure modes in qualitative examples.

Weaknesses

• Still heuristic. The threshold $p$ remains hand-tuned.
• Gains are decoding-time only; LM tail unreliability remains unsolved.
• Nucleus examples still contain factual drift and topic confusion.
• Evaluation centers on GPT-2-era English open-ended generation.

Suggestions from the authors

• Study better decoding schemes for open-ended generation beyond likelihood maximization.
• Improve language models so low-confidence tail tokens are better calibrated.
• Test whether larger models reduce degeneration without truncation.
• Develop evaluations that jointly measure coherence and diversity.

Links

Prior Papers

• @radfordGPT2_2019 — supplies the autoregressive LM and top-$k$ decoding baseline that nucleus sampling critiques.
• @vaswaniAttentionAllNeed2017 — provides the Transformer backbone underlying the language models studied.

Further Papers

• @brownGPT3_2020 — scales autoregressive open-ended generation, where decoding pathologies and truncation choices remain central.
• @leviathanSpeculativeDecoding2023 — studies faster inference for autoregressive generation, another decoding-time intervention.
• @ouyangInstructGPT2022 — shifts generation quality via alignment, complementing this paper's decoding-side fix.
• @rafailovDPO2023 — optimizes preference-aligned text generation, reducing reliance on decoding heuristics alone.