Neural Machine Translation of Rare Words with Subword Units

Problem

Framing

Word-level NMT with fixed vocabularies collapses rare words into $\text{UNK}$ or brittle dictionary back-off, which hurts names, compounds, and transliterations. The paper closes this gap by encoding words as subword units learned with byte pair encoding, yielding open-vocabulary translation and better rare-word accuracy on WMT15 English$\rightarrow$German and English$\rightarrow$Russian.

Currently Used Methods

Foundational

• @bahdanauAttention2014 — attention-based encoder-decoder NMT with fixed target vocabulary.
  • Limitation in context: rare words still map to shortlist tokens or $\text{UNK}$.
• @choGRU2014 — GRU sequence models for conditional neural translation.
  • Limitation in context: recurrent architecture alone does not solve open vocabularies.
• @sutskeverSeq2Seq2014 — end-to-end sequence-to-sequence learning with word tokens.
  • Limitation in context: word-level tokenization explodes vocabulary and sparsity.
• "Addressing the Rare Word Problem in Neural Machine Translation" — back-off dictionary replacement for unknown targets.
  • Limitation in context: copies or substitutes words without modeling subword composition.
• "Linguistically Motivated Vocabulary Reduction for Neural Machine Translation from Turkish to English" — syntax-based segmentation for vocabulary reduction.
  • Limitation in context: language-specific preprocessing is less simple and less portable.

Proposed Method

Architecture

The translation model is the standard attention-based encoder-decoder with a bidirectional GRU encoder and a GRU decoder. The novelty is not a new network block; it replaces word tokens with variable-length subword units, learned either separately on source and target or jointly across both vocabularies.

Loss / Objective

The training objective is the standard conditional log-likelihood over the target sequence.

Algorithm

BPE starts from character symbols and repeatedly merges the most frequent adjacent pair.

Training Procedure

• Encoder: bidirectional GRU.
• Decoder: GRU with attention.
• Training toolkit: Nematus.
• Source/target shortlists compared at 30k, 50k, 60k, 90k, 300k, 500k.
• BPE learned either independently or jointly over source and target vocabularies.
• Evaluation uses single models and 8-model ensembles.

Evaluation

Datasets

• WMT15 English$\rightarrow$German newstest2015.
• WMT15 English$\rightarrow$Russian newstest2015.

Metrics

• BLEU.
• CHRF3.
• Unigram $F_1$ for all words.
• Unigram $F_1$ for rare words.
• Unigram $F_1$ for OOV words.

Headline results

• English$\rightarrow$German, syntax-based baseline: BLEU 24.4, CHRF3 55.3.
• English$\rightarrow$German, WDict: BLEU 23.2, CHRF3 52.7, OOV $F_1$ 26.5.
• English$\rightarrow$German, BPE-J90k ensemble: BLEU 24.7, CHRF3 55.6, OOV $F_1$ 38.6.
• English$\rightarrow$Russian, phrase-based baseline: BLEU 24.3, CHRF3 53.8.
• English$\rightarrow$Russian, BPE-J90k ensemble: BLEU 26.0, CHRF3 54.7, OOV $F_1$ 15.6.

Ablations

• Joint vs independent BPE: joint BPE improves consistency and rare-word accuracy.
• Vocabulary size: smaller subword vocabularies can outperform larger word vocabularies.
• Word frequency rank: subword models degrade more smoothly on rare words.
• Back-off dictionary vs subwords: subwords beat dictionary replacement on productive word formation.

Method Strengths and Weaknesses

Strengths

• Drops language-specific morphology rules; BPE is simple and data-driven.
• Improves OOV translation over dictionary back-off in both language pairs.
• Joint BPE reduces segmentation mismatch across source and target.
• Smaller vocabularies reduce sparsity while keeping competitive BLEU.

Weaknesses

• Gains over strong phrase-based or syntax-based baselines stay modest in BLEU.
• Transliteration errors remain when segmentations differ across languages.
• Performance variance across random seeds reaches about 1 BLEU.
• The method changes tokenization, not the underlying seq2seq architecture.

Suggestions from the authors

• Reduce training instability and variance across model runs.
• Improve transliteration and segmentation consistency across alphabets.
• Test better architectures, training algorithms, and stronger ensembles.
• Study vocabulary-size selection instead of fixing it heuristically.

Links

Prior Papers

• @bahdanauAttention2014 — supplies the attention-based NMT backbone that this paper keeps while changing tokenization.
• @choGRU2014 — provides the GRU recurrent unit used in the encoder-decoder architecture.
• @sutskeverSeq2Seq2014 — establishes end-to-end sequence transduction with neural encoder-decoders.

Further Papers

• @vaswaniAttentionAllNeed2017 — replaces recurrence with attention, but still inherits the subword-tokenization regime this paper popularized.
• @radfordGPT2018 — large language models adopt BPE-style subword vocabularies for open-vocabulary text modeling.
• @devlinBERT2018 — WordPiece continues the subword-tokenization line for bidirectional pretraining.
• @brownGPT3_2020 — scaling autoregressive language models retains byte-level or BPE-style subword segmentation.