Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela

2020 · NeurIPS

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Problem

Framing

Large parametric seq2seq models store facts, but they cannot update knowledge, expose provenance, or reliably ground generation on knowledge-intensive tasks. RAG closes this gap by treating retrieved Wikipedia passages as latent variables inside generation, then marginalizing over them end-to-end. It sets new open-domain QA results, including 44.5 EM on Natural Questions.

Currently Used Methods

Direct antecedents

"Retrieval-Augmented Language Model Pre-Training" — differentiable dense retrieval for open-domain extractive QA.
- Limitation in context: tied to extractive QA, not general seq2seq generation.
"Open Retrieval Question Answering" — retrieve-then-read pipeline for open-domain QA.
- Limitation in context: retrieval and reader are not jointly optimized.
@raffelT5_2020 — large parametric text-to-text model with strong closed-book QA.
- Limitation in context: facts stay in weights, so updates and provenance are hard.
"BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation" — strong pre-trained seq2seq generator.
- Limitation in context: no explicit external memory during inference.
"Dense Passage Retrieval for Open-Domain Question Answering" — bi-encoder neural retriever over Wikipedia passages.
- Limitation in context: retrieves evidence, but does not generate answers jointly.

Proposed Method

Architecture

RAG combines a DPR-style retriever $p_{\eta}(z\mid x)$ with a BART generator $p_{\theta}(y_i\mid x, z, y_{1:i-1})$ . The retriever scores 21M Wikipedia 100-word passages with a BERT-base query encoder against a fixed document index, then the generator conditions on the input and top- $K$ passages.

Overview diagram: query encoder retrieves top-K Wikipedia passages from a dense index, then a seq2seq generator marginalizes over retrieved documents for answer, label, or question generation.

Loss / Objective

The model treats retrieved documents as latent variables and optimizes negative log-likelihood after top- $K$ marginalization.

p_{\mathrm{RAG\mbox{-}Sequence}}(y\mid x) \approx \sum_{z \in \mathrm{top\mbox{-}}k(p(\cdot \mid x))} p_{\eta}(z\mid x) p_{\theta}(y\mid x, z) = \sum_{z \in \mathrm{top\mbox{-}}k(p(\cdot \mid x))} p_{\eta}(z\mid x) \prod_{i=1}^{N} p_{\theta}(y_i \mid x, z, y_{1:i-1})

p_{\mathrm{RAG\mbox{-}Token}}(y\mid x) \approx \prod_{i=1}^{N} \sum_{z \in \mathrm{top\mbox{-}}k(p(\cdot \mid x))} p_{\eta}(z\mid x) \, p_{\theta}(y_i \mid x, z, y_{1:i-1})

Sampling Rule / Algorithm

Decoding differs by marginalization level: sequence-level for RAG-Sequence, token-level for RAG-Token.

p'_{\theta}(y_i \mid x, y_{1:i-1}) = \sum_{z \in \mathrm{top\mbox{-}}k(p(\cdot \mid x))} p_{\eta}(z_i \mid x) p_{\theta}(y_i \mid x, z_i, y_{1:i-1})

Training Procedure

Non-parametric memory: December 2018 Wikipedia dump.
Passage granularity: disjoint 100-word chunks.
Index size: 21M documents.
Training retrieval depth: $k \in \{5, 10\}$ .
QA test retrieval depth: 15 for RAG-Token, 50 for RAG-Sequence.
Optimizer: Adam.
Hardware: 8 \times 32GB NVIDIA V100.
Precision: mixed precision.
Document encoder and index: fixed during training.

Evaluation

Datasets

Open-domain QA: Natural Questions, TriviaQA, WebQuestions, CuratedTrec.
Abstractive QA: MS-MARCO NLG.
Question generation: Jeopardy from SearchQA splits.
Fact verification: FEVER 3-way and 2-way.

Metrics

Open-domain QA: Exact Match.
MS-MARCO: Bleu-1, Rouge-L.
Jeopardy generation: Q-BLEU-1, human factuality, human specificity.
FEVER: label accuracy.
Diversity: distinct tri-gram ratio.

Headline results

Natural Questions, open-domain QA: 44.5 EM with RAG-Sequence.
TriviaQA unfiltered / Wiki: 56.1 / 72.7 EM with RAG-Sequence.
WebQuestions: 45.2 EM with RAG-Sequence.
CuratedTrec: 50.0 EM with RAG-Sequence.
MS-MARCO: 44.1 Bleu-1, 45.5 Rouge-L with RAG-Sequence.
Jeopardy question generation: 17.43 Q-BLEU-1 with RAG-Token.
FEVER 3-way / 2-way: 86.3 / 89.0 accuracy with RAG.

Ablations

Learned retrieval vs frozen retriever: learned retrieval improves every task.
Dense retriever vs BM25: dense retrieval wins on most tasks.
Train-time $k=5$ vs $k=10$ : little difference.
Test-time retrieved documents: more documents improve QA, with smaller gains for generation.

Results figure: heatmap of RAG-Token token-level document posterior over five retrieved passages during Jeopardy question generation, showing different tokens attending to different documents.

Method Strengths and Weaknesses

Strengths

Unifies retrieval and generation in one probabilistic model.
Sets SOTA on NQ, WQ, and CuratedTrec.
Improves MS-MARCO over BART by 2.6 Bleu-1 and 2.6 Rouge-L.
Retrieved passages provide inspectable provenance and hot-swappable knowledge.

Weaknesses

Document encoder stays fixed, limiting retrieval adaptation.
QA gains require more retrieved documents at test time.
FEVER trails specialized supervised pipelines by 4.3% on 3-way accuracy.
Wikipedia-only memory leaves some MS-MARCO questions unanswerable.

Suggestions from the authors

Scale non-parametric memory beyond Wikipedia and beyond text-only sources.
Improve retrieval and generation jointly with stronger end-to-end learning.
Study faster decoding and retrieval for lower-latency generation.
Explore richer memory updating as world knowledge changes.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Problem

Framing

Currently Used Methods

Direct antecedents

Proposed Method

Architecture

Loss / Objective

Sampling Rule / Algorithm

Training Procedure

Evaluation

Datasets

Metrics

Headline results

Ablations

Method Strengths and Weaknesses

Strengths

Weaknesses

Suggestions from the authors

Links

Prior Papers

Further Papers