Papperoni

2020

How Much Knowledge Can You Pack Into the Parameters of a Language Model?

Noam Shazeer

Open PDF Google Scholar

citations

Cite Score

39

AI summary

The paper explores fine-tuning pre-trained T5 language models to answer questions without external knowledge, demonstrating competitive performance with open-domain systems. It leverages T5 models and introduces a closed-book question answering approach. The main result shows performance scales with model size.

Main Contributions

Demonstrates that fine-tuning pre-trained language models can effectively answer questions without external knowledge.
Shows that this approach scales with model size, with larger models achieving better performance.
Achieves competitive results compared to open-domain question answering systems that explicitly retrieve information from external sources.
Releases code and trained models to facilitate reproducibility and future research.
Introduces a closed-book question answering paradigm where models must rely solely on internalized knowledge.

Abstract

It has recently been observed that neural language models trained on unstructured text can implicitly store and retrieve knowledge using natural language queries. In this short paper, we measure the practical utility of this approach by fine-tuning pre-trained models to answer questions without access to any external context or knowledge. We show that this approach scales with model size and performs competitively with open-domain systems that explicitly retrieve answers from an external knowledge source when answering questions. To facilitate reproducibility and future work, we release our code and trained models.

Citation Graph

Loading graph...

References [36]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[3]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[4]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[5]Improving Language Understanding by Generative Pre-Training

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

23 papers in library cite

Very simple and very nice! Easy to understand and revolutionary maybe?

[6]XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Z. Dai, Yining Yang, J. Carbonell, Ruslan Salakhutdinov, Quoc V. Le - 2019

11 papers in library cite

The method is nice and the results are very good but this paper is just soooo hard to follow...

[7]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[8]ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations

Z. Lan, Mark Chen, S. Goodman, Kevin Gimpel, P. Sharma, Radu Soricut - 2019

8 papers in library cite

I like how they can achieve very close results with very few params! Very nice tricks to do that as well.

[9]Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder - 2018

14 papers in library cite

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

[10]Natural Questions: A Benchmark for Question Answering Research

T. Kwiatkowski, J. Palomaki, O. Rhinehart, Michael Collins, A. P. Parikh, C. Alberti, D. Epstein, Illia Polosukhin, M. Kelcey, Jacob Devlin, K. Lee, K. N. Toutanova, Llion Jones, M. W. Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, Slav Petrov - 2019

9 papers in library cite

The dataset and methodology is very nice - it's amazing to see how Google does the summaries in search. However, the paper is too complex with the math stuff - unnecessary.

[11]Language Models as Knowledge Bases?

F. Petroni, Tim Rocktaschel, P. Lewis, A. Bakhtin, Yonghui Wu, A. H. Miller, Sebastian Riedel - 2019

4 papers in library cite

Very nice, but I expected a bit more. I thought it would be more of a philosophical discussion rather than a benchmark analysis. Still, probably the first ones to notice that LMs contain knowledge.

[12]TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer - 2017

18 papers in library cite

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

[13]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[14]Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Wentao Li, P. J. Liu - 2019

17 papers in library cite

44 pages; T5 paper - "unifying framework where all text-based NLP problems are cast as text-to-text tasks"

[15]Reading Wikipedia to Answer Open-Domain Questions

Deli Chen, Adam Fisch, Jason Weston, Antoine Bordes - 2017

10 papers in library cite

Open Domain QA with wikipedia

[16]REALM: Retrieval-Augmented Language Model Pre-Training

K. Guu, K. Lee, Z. Tung, P. Panupong, M. W. Chang - 2020

5 papers in library cite

I think it's famous

[17]BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

C. Clark, K. Lee, M. W. Chang, T. Kwiatkowski, Michael Collins, Kristina Toutanova - 2019

4 papers in library cite

Part of SuperGLUE

[18]Adafactor: Adaptive Learning Rates With Sublinear Memory Cost

Noam Shazeer, M. Stern - 2018

3 papers in library cite

Used for GPT3 training

[19]DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

D. Dua, Yuzhi Wang, P. Dasigi, G. Stanovsky, Shivalika Singh, Matt Gardner - 2019

4 papers in library cite

[20]How Can We Know What Language Models Know?

Zhejun Jiang, F. F. Xu, J. Araki, Graham Neubig - 2019

2 papers in library cite

[21]Semantic Parsing on Freebase From Question-Answer Pairs

Jonathan Berant, A. Chou, R. Frostig, Percy Liang - 2013

8 papers in library cite

[22]Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

T. Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal - 2018

6 papers in library cite

[23]Looking Beyond the Surface: A Challenge Set for Reading Comprehension Over Multiple Sentences

Daniel Khashabi, S. Chaturvedi, M. Roth, Shyam Upadhyay, Dan Roth - 2018

4 papers in library cite

[24]ReCoRD: Bridging the Gap Between Human and Machine Commonsense Reading Comprehension

S. Zhang, Xiaodong Liu, Joseph Liu, Jianfeng Gao, K. Duh, B. V. Durme - 2018

4 papers in library cite

[25]Dense Passage Retrieval for Open-Domain Question Answering

V. Karpukhin, B. Ouguz, S. Min, L. Y. Wu, S. Edunov, Deli Chen, W. T. Yih - 2020

3 papers in library cite

[26]Entities as Experts: Sparse Memory Access With Entity Supervision

T. Fevry, L. B. Soares, N. Fitzgerald, E. Choi, T. Kwiatkowski - 2020

2 papers in library cite

[27]Latent Retrieval for Weakly Supervised Open Domain Question Answering

K. Lee, M. W. Chang, Kristina Toutanova - 2019

2 papers in library cite

[28]olmpics-on What Language Model Pre-Training Captures

A. Talmor, Y. Elazar, Y. Goldberg, Jonathan Berant - 2019

2 papers in library cite

[29]A Discrete Hard EM Approach for Weakly Supervised Question Answering

S. Min, Deli Chen, Hananneh Hajishirzi, Luke Zettlemoyer - 2019

1 paper in library cites

[30]Ccnet: Extracting High Quality Monolingual Datasets From Web Crawl Data

G. Wenzek, M. A. Lachaux, Alexis Conneau, V. Chaudhary, F. Guzman, Armand Joulin, E. Grave - 2019

1 paper in library cites

[31]Freebase: A Collaboratively Created Graph Database for Structuring Human Knowledge

K. Bollacker, C. Evans, P. Paritosh, T. Sturge, J. Taylor - 2008

1 paper in library cites

[32]Frustratingly Easy Natural Question Answering

L. Pan, R. Chakravarti, A. Ferritto, M. Glass, A. Gliozzo, S. Roukos, R. Florian, A. Sil - 2019

1 paper in library cites

[33]Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering

S. Min, Deli Chen, Luke Zettlemoyer, Hananneh Hajishirzi - 2019

1 paper in library cites

[34]Learning Cross-Context Entity Representations From Text

J. Ling, N. Fitzgerald, Z. Shan, L. B. Soares, T. Fevry, D. Weiss, T. Kwiatkowski - 2020

1 paper in library cites

[35]Learning to Retrieve Reasoning Paths Over Wikipedia Graph for Question Answering

A. Asai, K. Hashimoto, Hananneh Hajishirzi, Richard Socher, Caiming Xiong - 2019

1 paper in library cites

[36]Open-Domain Question-Answering

J. Prager - 2006

1 paper in library cites

Cited by

2

papers in your library

Cites

20

papers in your library

Read

on December 29, 2025

Very nice, probably the first to notice that LLMs can answer closed-book questions. However, they don't really answer the question in the title (and TBH there is not a ton of contribution)

Tags

Paper Aliases

No aliases