Attention Is All You Need

Problem

Framing

RNN and CNN transducers bottleneck on sequential computation, so training scales poorly and long-range paths stay long. The paper replaces both with a pure self-attention encoder-decoder, reaching 28.4 BLEU on WMT14 En-De and 41.8 BLEU on En-Fr.

Currently Used Methods

Foundational

• @sutskeverSeq2Seq2014 — recurrent encoder-decoder sequence transduction.
  • Limitation in context: token-by-token state updates block full parallel training.
• @bahdanauAttention2014 — additive alignment attention for neural translation.
  • Limitation in context: attention still depends on a recurrent backbone.
• @choGRU2014 — gated recurrent units for sequence modeling.
  • Limitation in context: gating preserves sequential dependency across positions.
• @baLayerNormalization2016 — layer normalization for stable deep sequence models.
  • Limitation in context: optimization improves, but recurrence remains the bottleneck.

Proposed Method

Architecture

The model stacks 6 encoder and 6 decoder layers. Each encoder layer applies multi-head self-attention and a position-wise FFN; each decoder layer adds masked self-attention and encoder-decoder attention. Base settings are $d_{\mathrm{model}}=512$, $h=8$, $d_k=d_v=64$, and $d_{\mathrm{ff}}=2048$.

Loss / Objective

Training uses next-token cross-entropy with label smoothing $\epsilon_{ls}=0.1$.

Algorithm

The core computation is scaled dot-product attention, composed into multi-head attention.

Training Procedure

• Layers: $N=6$ encoder, $N=6$ decoder.
• Base: $d_{\mathrm{model}}=512$, $d_{\mathrm{ff}}=2048$, $h=8$, $d_k=d_v=64$.
• Big: $d_{\mathrm{model}}=1024$, $d_{\mathrm{ff}}=4096$, $h=16$.
• Optimizer: Adam, $\beta_1=0.9$, $\beta_2=0.98$, $\epsilon=10^{-9}$.
• Learning rate:

• Warmup steps: $4000$.
• Dropout: $P_{\mathrm{drop}}=0.1$ base, $0.3$ big.
• Label smoothing: $\epsilon_{ls}=0.1$.
• Training steps: $100\mathrm{K}$ base, $300\mathrm{K}$ big.
• Decoding: beam size $4$, length penalty $\alpha=0.6$.

Evaluation

Datasets

• WMT14 English-German: 4.5M sentence pairs.
• WMT14 English-French: 36M sentence pairs.
• WSJ Section 23 constituency parsing.

Metrics

• BLEU for translation.
• Development perplexity.
• WSJ23 F1 for parsing.
• Training cost in FLOPs and wall-clock time.

Headline results

• WMT14 En-De (Transformer big): BLEU 28.4.
• WMT14 En-De (base): BLEU 27.3.
• WMT14 En-Fr (single model): BLEU 41.8.
• WMT14 En-Fr (base): BLEU 38.1.
• WSJ23 parsing: F1 92.7.

Table 1: En-De development ablations over heads, width, dropout, label smoothing, and model size

Ablations

• Attention heads: 1 head is 0.9 BLEU below the best multi-head setting.
• Depth: reducing layers from 6 to 2 drops BLEU from 25.8 to 23.7.
• Width: $d_{\mathrm{model}}=1024$ raises BLEU to 26.0; $d_{\mathrm{ff}}=4096$ reaches 26.2.
• Regularization: dropout 0.2 helps BLEU; label smoothing 0.2 helps BLEU but hurts perplexity.

Method Strengths and Weaknesses

Strengths

• Pure self-attention beats prior translation systems on both WMT14 benchmarks.
• Base model trains in 12 hours on 8 P100 GPUs.
• Big model surpasses prior En-De results by over 2 BLEU.
• Same architecture transfers to parsing and reaches 92.7 F1.

Weaknesses

• Self-attention cost grows quadratically with sequence length.
• Big model still needs 300K steps and 3.5 days on 8 GPUs.
• Parsing trails the best reported generative parser, 92.7 versus 93.3 F1.
• Positional design shows little separation between sinusoidal and learned embeddings.

Suggestions from the authors

• Reduce attention cost with restricted neighborhoods for long sequences.
• Extend self-attention models to images, audio, and video.
• Test pure attention on more sequence transduction tasks.
• Study stronger positional representations for order-sensitive settings.

Links

Prior Papers

• @bahdanauAttention2014 — additive attention is the direct precursor Transformer internalizes without recurrence.
• @sutskeverSeq2Seq2014 — establishes the encoder-decoder translation template that Transformer parallelizes.
• @choGRU2014 — represents the recurrent family displaced by self-attention sequence modeling.
• @baLayerNormalization2016 — supplies the normalization primitive used in each residual sublayer.

Further Papers

• @devlinBERT2018 — builds large-scale bidirectional language pretraining on the Transformer encoder.
• @radfordGPT2018 — turns the Transformer decoder into a generative language model.
• @radfordGPT2_2019 — scales decoder-only Transformer language modeling to stronger zero-shot transfer.
• @brownGPT3_2020 — extends the same autoregressive Transformer recipe to massive scale.
• @raffelT5_2020 — recasts NLP tasks into a unified Transformer text-to-text framework.
• @dosovitskiyViT2020 — ports Transformer blocks from text sequences to image patches.
• @rameshDALLE2021 — uses Transformer sequence modeling for text-conditional image generation.
• @daoFlashAttention2022 — targets the quadratic attention bottleneck with a faster exact kernel.
• @huLoRA2021 — adapts large Transformer models with low-rank parameter updates.
• @kaplanScalingLaws2020 — analyzes performance scaling behavior in large Transformer language models.