RoFormer: Enhanced Transformer with Rotary Position Embedding

Problem

Framing

Standard Transformer position schemes either add absolute embeddings or alter attention with relative biases, but these forms do not transfer cleanly to linear attention. RoFormer closes this gap by encoding position as a rotation in each query-key subspace, so absolute indices induce explicit relative phase differences and distance decay.

Currently Used Methods

Foundational

• @vaswaniAttentionAllNeed2017 — sinusoidal absolute embeddings added to token representations.
  • Limitation in context: additive encoding does not express relative phase inside attention scores.
• @devlinBERT2018 — pre-trained bidirectional Transformer with learned absolute positions.
  • Limitation in context: position stays tied to embedding addition, not query-key interaction.
• "Self-Attention with Relative Position Representations" — relative offsets injected into attention.
  • Limitation in context: the formulation is awkward for linear-attention kernels.
• "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context" — relative encoding for longer-context recurrence.
  • Limitation in context: still relies on additive score terms instead of multiplicative rotations.
• "Rethinking Attention with Performers" — linearized attention for subquadratic sequence scaling.
  • Limitation in context: lacks a native relative-position mechanism compatible with linear attention.

Proposed Method

Architecture

RoFormer keeps the standard multi-head Transformer block and only replaces positional handling inside self-attention. Each even-dimensional query and key vector is partitioned into 2D pairs, then each pair is rotated by a position-dependent angle before the inner product.

Loss / Objective

The method changes position encoding, not the task loss; training uses the baseline objective for each task while computing attention with rotated queries and keys.

Sampling Rule / Algorithm

Self-attention uses the usual normalized similarity, but the similarity is computed from RoPE-transformed queries and keys.

Training Procedure

• Machine translation: single model averaged over last 10 checkpoints.
• MLM pre-training: batch size 64.
• MLM pre-training: max sequence length 512.
• MLM pre-training: 100k steps.
• MLM optimizer: AdamW.
• MLM learning rate: $5 \times 10^{-5}$.
• GLUE fine-tuning: 3 epochs.
• GLUE fine-tuning: max sequence length 512.
• GLUE fine-tuning: batch size 32.
• GLUE fine-tuning: learning rates $\{2,3,4,5\} \times 10^{-5}$.

Evaluation

Datasets

• WMT 2014 English-to-German translation.
• BookCorpus + Wikipedia for MLM pre-training.
• GLUE: MRPC, SST-2, QNLI, STS-B, QQP, MNLI.
• Enwik8 for linear-attention experiments.
• Chinese long-document task: CAIL2019-SCM.

Metrics

• BLEU for translation.
• MLM training loss for pre-training.
• F1 for MRPC and QQP.
• Spearman correlation for STS-B.
• Accuracy for SST-2, QNLI, MNLI, CAIL2019-SCM.

Headline results

• WMT14 En-De: BLEU 27.5 for RoFormer vs 27.3 for Transformer.
• GLUE: RoFormer beats BERT on 3 of 6 tasks.
• MLM pre-training: faster loss convergence than BERT.
• Performer on Enwik8: lower training loss with RoPE than without.
• CAIL2019-SCM long input: RoFormer-1024 reaches 69.79% test accuracy.

Ablations

• Position mechanism: replacing sinusoidal encoding with RoPE improves WMT14 BLEU.
• Pre-training dynamics: RoPE converges faster than BERT under matched MLM setup.
• Linear attention: adding RoPE to Performer lowers loss and speeds convergence.
• Context length: increasing maximum sequence length improves CAIL2019-SCM accuracy.

Method Strengths and Weaknesses

Strengths

• Changes only positional encoding; the Transformer backbone stays intact.
• Encodes relative position directly in query-key products.
• Preserves compatibility with linear attention, unlike additive relative-bias schemes.
• Improves BLEU, GLUE results, and long-context accuracy with one mechanism.

Weaknesses

• Translation gain is small: 27.5 BLEU vs 27.3 baseline.
• GLUE summary reports wins on only three of six tasks.
• Evidence centers on NLP; transfer to other modalities is untested here.
• Faster convergence is observed, but the paper says the cause remains unclear.

Suggestions from the authors

• Explain why rotary encoding converges faster than other position schemes.
• Study how RoPE's long-term decay affects different NLP tasks.
• Test RoPE beyond language tasks and beyond current Transformer variants.
• Extend RoPE-based pre-training to broader long-context benchmarks.

Links

Prior Papers

• @vaswaniAttentionAllNeed2017 — RoFormer modifies the Transformer's positional pathway while preserving self-attention structure.
• @devlinBERT2018 — the paper compares RoFormer against BERT pre-training and downstream fine-tuning.

Further Papers

• @touvronLLaMA2023 — LLaMA adopts rotary embeddings as a default positional mechanism in large language models.
• @daoFlashAttention2022 — efficient attention kernels pair naturally with RoPE because RoPE stays inside query-key algebra.
• @guMamba2023 — long-context sequence modeling work sharpens the same efficiency-versus-position tradeoff RoFormer targets.