RAG

Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 4 (April 2021) of the NeurIPS 2020 conference paper).

What it changed

It joined a text generator with a search over Wikipedia and trained them together, so a model could write answers from passages it looks up, and its knowledge could be updated by changing the documents instead of retraining.

The problem before it

Pretrained language models store a lot of facts in their weights. The authors list three problems with that. The models cannot easily add or correct what they know. They cannot show where an answer came from. And they sometimes produce “hallucinations”, text that sounds right but is false.

Models that also search a document collection can fix some of this, because documents can be read, checked and replaced. Two such models, REALM and ORQA, had shown good results. But they were built on masked language models and only picked a span of text out of a document. They could not write free text, so they were used only for extractive question answering.

The idea

The mechanism. RAG has two pretrained parts, which the authors call two kinds of memory:

• The retriever (the “non-parametric memory”) is dense retrieval. One BERT encoder turns each Wikipedia passage into a vector, and another turns the question into a vector. The passages whose vectors best match the question are found with a fast search library, FAISS. Wikipedia is split into 21 million passages of 100 words.
• The generator (the “parametric memory”) is BART, a pretrained encoder-decoder with 400 million parameters. It reads the input joined with one retrieved passage and writes the output.

The model retrieves the top 5 or 10 passages and lets the generator work with each one. Then it combines the results, weighted by how well each passage matched. Nobody tells the model which passage is correct. The passage is treated as a hidden choice, and training improves both the question encoder and the generator, from the input and output text alone. To save cost, the passage encoder and the index stay fixed.

There are two ways to combine passages:

• RAG-Sequence uses the same passage for the whole answer.
• RAG-Token can use a different passage for each output token, so one answer can combine facts from several passages.

This is retrieval-augmented generation, and the paper gives it its name.

The notation. The retriever gives p_η(z | x) ∝ exp(d(z)ᵀ q(x)), where d(z) is a passage’s vector and q(x) the question’s vector. RAG-Sequence computes p(y | x) ≈ Σ_z p_η(z | x) Π_i p_θ(y_i | x, z, y_(1:i-1)), summing over the top passages z. RAG-Token moves the sum inside the product: p(y | x) ≈ Π_i Σ_z p_η(z | x) p_θ(y_i | x, z, y_(1:i-1)).

What it showed

Reported in the paper. It was published at NeurIPS 2020, a peer-reviewed conference. Answers in question answering are scored by exact match: the share of answers that are exactly right.

• Open-domain question answering. RAG-Sequence scored 44.5 on Natural Questions, against 41.5 for DPR’s extract-a-span system and 36.6 for T5-11B, a much larger model with no retrieval. It also scored 45.2 on WebQuestions and 52.2 on CuratedTrec. On the standard TriviaQA test set it scored 56.8, below DPR’s 57.9. On a second TriviaQA test set, used by T5, it scored 68.0, against 60.5.
• The paper gives two counts of these wins. The abstract says RAG set a new best result on three open-domain question-answering tasks. Section 4.1 says it set a new best result on all four, counting TriviaQA only on the test set that T5 used. Both are given here.
• Answers not in any passage. On Natural Questions, RAG gave the right answer in 11.8% of cases where no retrieved passage contained it. A system that copies a span would score 0% there.
• Free-text answers (MS MARCO). RAG-Sequence beat a BART model without retrieval by 2.6 points on both BLEU-1 and ROUGE-L, which measure word overlap with a reference answer. It did not see the reference passages, which the best earlier systems used.
• Writing Jeopardy questions from an answer. In 452 pairs judged by people, RAG-Token was judged more factual than BART in 42.7% of cases, and BART more factual in 7.1%. RAG was also judged more specific: 37.4% against 16.8%.
• The paper gives two numbers for “both factual”. The text says both models were factual in a further 17% of cases. Table 4 gives 11.7% for “both good” and 17.7% for “both poor”. Both are given here.
• Fact checking (FEVER). 72.5% accuracy on the 3-way task, 4.3 points below the best system, which was trained with labels for the evidence sentences. RAG used no such labels.
• Updating knowledge. The authors asked “Who is {position}?” for 82 world leaders who changed between 2016 and 2018. With a 2016 Wikipedia index, RAG answered 70% of the 2016 leaders correctly. With a 2018 index, it answered 68% of the 2018 leaders. With the wrong year’s index, it got only 12% and 4%. Changing the documents changed the answers, with no retraining.
• Learned retrieval helps. Fine-tuning the question encoder improved results on every task, against a fixed retriever. A word-matching search (BM25) did better only on FEVER.
• Size. RAG has 626 million trainable parameters by the paper’s count, which includes the passage encoder that is not trained. T5-large, the closest T5 in size at 770 million, scored 28.9 on Natural Questions against RAG’s 44.5.

Limits, and what came later

The limits come from the paper:

• Wikipedia, or any other document source, is never entirely factual or free of bias, and RAG inherits this.
• On some tasks, such as story generation, the retriever “collapsed”: it returned the same passages for every input, and the generator learned to ignore them. The model then did no better than BART.
• The passage index needs about 100 GB of computer memory, or 36 GB when compressed.
• The passage encoder and index were kept fixed. The authors suggest that training both parts together from the start is future work.
• It does not extract FEVER’s evidence sentences, because FEVER uses a different Wikipedia version.

Later work built on it. RETRO treats RAG as an earlier, smaller-scale example of retrieval-augmented generation. Lost in the middle studied how well models use the passages they are given in retrieval settings.

What to know first

• Dense retrieval, and the DPR retriever.
• BART, and encoder-decoder models in general.
• Pretraining and fine-tuning.
• The idea of a hidden (“latent”) variable that is summed over, here the retrieved passage.

How to read it

The paper is 19 pages, including references and appendices.

1. Read section 1 with Figure 1.
2. Read section 2.1 for the two models. The two formulas are the core.
3. Read section 4.1 with Table 1, then section 4.3 with Figure 2, which shows RAG-Token using a different passage for each book title.
4. Read “Index hot-swapping” in section 4.5.
5. Read appendix H on retrieval collapse.

Expect about 1.5 hours for the parts above (an estimate).

Links

• The paper: arXiv 2005.11401
• Code: the paper links scripts in the Hugging Face Transformers library. That address no longer works. The RAG models are documented in the library at huggingface.co/docs/transformers/model_doc/rag.
• Video: none is linked here yet.