Papperoni

2020

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

P. Lewis, Ethan Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, Martha Lewis, W. T. Yih, Tim Rocktaschel, Sebastian Riedel, K. Douwe

Open PDF Google Scholar

citations

Cite Score

82

AI summary

This paper introduces Retrieval-Augmented Generation (RAG) models, which combine pre-trained seq2seq models with a dense vector index of Wikipedia accessed by a neural retriever, achieving state-of-the-art results on open domain QA tasks and generating more factual language.

Main Contributions

Introduces RAG models, which combine pre-trained parametric and non-parametric memory for language generation.
Compares two RAG formulations: RAG-Sequence and RAG-Token.
Achieves state-of-the-art results on three open domain QA tasks: Natural Questions, WebQuestions, and CuratedTrec.
Demonstrates that RAG models generate more specific, diverse, and factual language than a state-of-the-art parametric-only seq2seq baseline.
Shows that the non-parametric memory can be replaced to update the models' knowledge as the world changes.

Abstract

Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) - models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, and another which can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state of the art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.

Citation Graph

Loading graph...

References [65]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[3]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[4]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[5]Improving Language Understanding by Generative Pre-Training

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

23 papers in library cite

Very simple and very nice! Easy to understand and revolutionary maybe?

[6]BART: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension

Martha Lewis, Yibo Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer - 2019

6 papers in library cite

I expected SO much less from this paper, but it is simple, intuitive, AND achieves SotA!

[7]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[8]Huggingface's Transformers: State-of-the-Art Natural Language Processing

Thomas Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, Sam Shleifer, P. V. Platen, C. Ma, Yacine Jernite, J. Plu, Chenfeng Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, Alexander M. Rush - 2019

7 papers in library cite

It's a solid paper, but these framework papers are usually boring... Yet, it's nice to see that they started with reduced scope (and how large it got now).

[9]Natural Questions: A Benchmark for Question Answering Research

T. Kwiatkowski, J. Palomaki, O. Rhinehart, Michael Collins, A. P. Parikh, C. Alberti, D. Epstein, Illia Polosukhin, M. Kelcey, Jacob Devlin, K. Lee, K. N. Toutanova, Llion Jones, M. W. Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, Slav Petrov - 2019

9 papers in library cite

The dataset and methodology is very nice - it's amazing to see how Google does the summaries in search. However, the paper is too complex with the math stuff - unnecessary.

[10]Language Models as Knowledge Bases?

F. Petroni, Tim Rocktaschel, P. Lewis, A. Bakhtin, Yonghui Wu, A. H. Miller, Sebastian Riedel - 2019

4 papers in library cite

Very nice, but I expected a bit more. I thought it would be more of a philosophical discussion rather than a benchmark analysis. Still, probably the first ones to notice that LMs contain knowledge.

[11]TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer - 2017

18 papers in library cite

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

[12]End-to-End Memory Networks

S. Sukhbaatar, A. Szlam, Jason Weston, Rob Fergus - 2015

18 papers in library cite

This was so surprising! This is very similar to transformers and RAG. Who knew?!

[13]SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A. Wang, Y. Pruksachatkun, Nikita Nangia, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2019

15 papers in library cite

Nothing too surprising, just getting a bunch of stuff from different places and putting it all together. At least they do a good analysis of the benchmark vs. existing methodologies.

[14]Memory Networks

Jason Weston, S. Chopra, Antoine Bordes - 2015

18 papers in library cite

The first half of the paper (when they discuss the concept in a very abstract way) is amazing. However, the actual methodology was very convoluted - I did not like it. I thought that Neural Turing Machines were inspired in this, but actually they are contemporary... So anyway, the concept is nice, execution is not.

[15]Hierarchical Neural Story Generation

A. Fan, Martha Lewis, Yann Dauphin - 2018

4 papers in library cite

I had higher hopes for this paper. The methodology is nice but at this stage things start to get waaay to complicated

[16]Passage Re-Ranking With BERT

Rodrigo Nogueira, Kyunghyun Cho - 2019

1 paper in library cites

It's good but very very short. I think it's just good because it's SotA and straightforward, but lacks content.

[17]Generating Wikipedia by Summarizing Long Sequences

P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, Lukasz Kaiser, Noam Shazeer - 2018

7 papers in library cite

Very nice, the first ones that realized that they could use decoder only from Transformers. Also nice that they expanded the context window, and got nice results. Very inventive to use citations from wikipedia and the first paragraph as target.

[18]Wizard of Wikipedia: Knowledge-Powered Conversational Agents

E. Dinan, S. Roller, K. Shuster, A. Fan, Michael Auli, Jason Weston - 2019

4 papers in library cite

This could be the original RAG. It's soooo close to the original thing!

[19]How Much Knowledge Can You Pack Into the Parameters of a Language Model?

Noam Shazeer - 2020

2 papers in library cite

Very nice, probably the first to notice that LLMs can answer closed-book questions. However, they don't really answer the question in the title (and TBH there is not a ton of contribution)

[20]Simple and Effective Multi-Paragraph Reading Comprehension

C. Clark, Matt Gardner - 2017

7 papers in library cite

Very nice paper! I think it's a stretch to call it "simple", but the paper is very well written and easy to follow.

[21]Inferring Algorithmic Patterns With Stack-Augmented Recurrent Nets

Armand Joulin, Tomas Mikolov - 2015

9 papers in library cite

Very underwhelming TBH. I expected more after reading the Neural Turing Machine paper. This reads like "yeah, we lost the race, here's what we were doing before they did something better"

[22]Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Wentao Li, P. J. Liu - 2019

17 papers in library cite

44 pages; T5 paper - "unifying framework where all text-based NLP problems are cast as text-to-text tasks"

[23]Billion-Scale Similarity Search With Gpus

J. Johnson, M. Douze, Hervé Jégou - 2017

4 papers in library cite

Efficient KNN with GPUs

[24]fairseq: A Fast, Extensible Toolkit for Sequence Modeling

M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, Michael Auli - 2019

4 papers in library cite

[25]Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs

Y. A. Malkov, D. A. Yashunin - 2016

1 paper in library cites

[26]Reading Wikipedia to Answer Open-Domain Questions

Deli Chen, Adam Fisch, Jason Weston, Antoine Bordes - 2017

10 papers in library cite

Open Domain QA with wikipedia

[27]REALM: Retrieval-Augmented Language Model Pre-Training

K. Guu, K. Lee, Z. Tung, P. Panupong, M. W. Chang - 2020

5 papers in library cite

I think it's famous

[28]MS MARCO: A Human Generated Machine Reading Comprehension Dataset

T. N. Nguyen, M. Rosenberg, X. Song, Jianfeng Gao, S. Tiwary, R. Majumder, L. Deng - 2016

8 papers in library cite

I am not sure if this is very relevant, but it has a few citations.

[29]ELI5: Long Form Question Answering

A. Fan, Yacine Jernite, Ethan Perez, D. Grangier, Jason Weston, Michael Auli - 2019

4 papers in library cite

[30]A Knowledge-Grounded Neural Conversation Model

M. Ghazvininejad, Chris Brockett, M. W. Chang, B. Dolan, Jianfeng Gao, W. T. Yih, M. Galley - 2017

3 papers in library cite

[31]SearchQA: A New Q&A Dataset Augmented With Context From a Search Engine

M. Dunn, L. Sagun, M. Higgins, V. U. Guney, V. Cirik, Kyunghyun Cho - 2017

5 papers in library cite

[32]How Context Affects Language Models’ Factual Predictions

F. Petroni, P. Lewis, A. Piktus, Tim Rocktaschel, Yonghui Wu, A. H. Miller, Sebastian Riedel - 2020

1 paper in library cites

[33]Semantic Parsing on Freebase From Question-Answer Pairs

Jonathan Berant, A. Chou, R. Frostig, Percy Liang - 2013

8 papers in library cite

[34]A Diversity-Promoting Objective Function for Neural Conversation Models

Jeffrey Li, M. Galley, Chris Brockett, Jianfeng Gao, B. Dolan - 2016

4 papers in library cite

[35]Finding Generalizable Evidence by Learning to Convince q&a Models

Ethan Perez, S. Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, Kyunghyun Cho - 2019

4 papers in library cite

[36]Dense Passage Retrieval for Open-Domain Question Answering

V. Karpukhin, B. Ouguz, S. Min, L. Y. Wu, S. Edunov, Deli Chen, W. T. Yih - 2020

3 papers in library cite

[37]Acute-eval: Improved Dialogue Evaluation With Optimized Questions and Multi-Turn Comparisons

M. Li, Jason Weston, S. Roller - 2019

2 papers in library cite

[38]Entities as Experts: Sparse Memory Access With Entity Supervision

T. Fevry, L. B. Soares, N. Fitzgerald, E. Choi, T. Kwiatkowski - 2020

2 papers in library cite

[39]FEVER: A Large-Scale Dataset for Fact Extraction and VERification

J. Thorne, A. Vlachos, C. Christodoulopoulos, A. Mittal - 2018

2 papers in library cite

[40]Generalization Through Memorization: Nearest neighbor Language Models

U. Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, Martha Lewis - 2020

2 papers in library cite

[41]Latent Retrieval for Weakly Supervised Open Domain Question Answering

K. Lee, M. W. Chang, Kristina Toutanova - 2019

2 papers in library cite

[42]Mixed Precision Training

P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, H. Wu - 2018

2 papers in library cite

[43]R3: Reinforced Ranker-Reader for Open-Domain Question Answering

Shijie Wang, M. Yu, X. Guo, Zhengtao Wang, T. Klinger, Wenxuan Zhang, S. Chang, Gerald Tesauro, B. Zhou, J. J. Jiang - 2018

2 papers in library cite

[44]The Next Decade in Ai: Four Steps Towards Robust Artificial Intelligence

G. Marcus - 2020

2 papers in library cite

[45]Towards Exploiting Background Knowledge for Building Conversation Systems

N. Moghe, S. Arora, S. Banerjee, M. M. Khapra - 2018

2 papers in library cite

[46]A Retrieve-and-Edit Framework for Predicting Structured Outputs

T. B. Hashimoto, K. Guu, Y. Oren, P. S. Liang - 2018

1 paper in library cites

[47]Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

S. Zhang, Mohit Bansal - 2019

1 paper in library cites

[48]Augmenting Transformers With KNN-based Composite Memory

A. Fan, C. Gardent, C. Braud, Antoine Bordes - 2020

1 paper in library cites

[49]Avoiding Catastrophic Forgetting in Mitigating Model Biases in Sentence-Pair Classification With Elastic Weight Consolidation

J. H. Thorne, A. Vlachos - 2020

1 paper in library cites

[50]Coarse-to-Fine Question Answering for Long Documents

E. Choi, D. Hewlett, Jakob Uszkoreit, Illia Polosukhin, A. Lacoste, Jonathan Berant - 2017

1 paper in library cites

[51]Diverse Beam Search for Improved Description of Complex Scenes

A. Vijayakumar, M. Cogswell, R. Selvaraju, Q. Sun, S. Lee, D. Crandall, D. Batra - 2018

1 paper in library cites

[52]Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering

Shijie Wang, M. Yu, J. J. Jiang, Wenxuan Zhang, X. Guo, S. Chang, Zhengtao Wang, T. Klinger, Gerald Tesauro, M. Campbell - 2018

1 paper in library cites

[53]Generating Sentences by Editing Prototypes

K. Guu, T. B. Hashimoto, Y. Oren, Percy Liang - 2018

1 paper in library cites

[54]How Decoding Strategies Affect the Verifiability of Generated Text

L. Massarelli, F. Petroni, A. Piktus, M. Ott, Tim Rocktaschel, V. Plachouras, F. Silvestri, Sebastian Riedel - 2019

1 paper in library cites

[55]Large Memory Layers With Product Keys

G. Lample, A. Sablayrolles, M. A. Ranzato, L. Denoyer, Hervé Jégou - 2019

1 paper in library cites

[56]Modeling of the Question Answering Task in the Yodaqa System

P. Baudis, J. Sedivy - 2015

1 paper in library cites

[57]Palm: Pre-Training an Autoencoding&autoregressive Language Model for Context-Conditioned Generation

B. Bi, Chun-Liang Li, Chiyu Wu, Minghao Yan, Wenyi Wang - 2020

1 paper in library cites

[58]Reasoning Over Semantic-Level Graph for Fact Checking

W. Zhong, Jiacheng Xu, D. Tang, Zhiwei Xu, N. Duan, M. Zhou, J. Wang, J. Yin - 2019

1 paper in library cites

[59]Retrieve and Refine: Improved Sequence Generation Models for Dialogue

Jason Weston, E. Dinan, A. Miller - 2018

1 paper in library cites

[60]Robust Neural Machine Translation With Joint Textual and Phonetic Embedding

Haozhe Liu, M. Ma, L. Huang, H. Xiong, Z. He - 2019

1 paper in library cites

[61]Search Engine Guided Neural Machine Translation

J. Gu, Yuzhi Wang, Kyunghyun Cho, V. O. K. Li - 2018

1 paper in library cites

[62]Simple and Effective Retrieve-Edit-Rerank Text Generation

N. Hossain, M. Ghazvininejad, Luke Zettlemoyer - 2020

1 paper in library cites

[63]The Probabilistic Relevance Framework: Bm25 and Beyond

S. Robertson, H. Zaragoza - 2009

1 paper in library cites

[64]Towards a Better Metric for Evaluating Question Generation Systems

P. Nema, M. M. Khapra - 2018

1 paper in library cites

[65]When Will AI Exceed Human Performance? Evidence From AI Experts

K. Grace, J. Salvatier, A. Dafoe, B. Zhang, Owain Evans - 2017

1 paper in library cites

Cited by

5

papers in your library

Cites

32

papers in your library

Read

on May 25, 2025

It coined the term "RAG" but really it was a very different concept.

Tags

Paper Aliases

No aliases