Papperoni

2019

Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever

Open PDF Google Scholar

citations

Cite Score

91

AI summary

This paper introduces GPT-2, a 1.5B parameter Transformer language model trained on WebText, a new dataset of millions of webpages, that achieves state-of-the-art zero-shot results on various NLP tasks like question answering (55 F1 on CoQA), translation, reading comprehension and summarization.

Main Contributions

Introduces GPT-2, a 1.5B parameter Transformer language model.
Introduces WebText, a new dataset of millions of webpages.
Demonstrates language models can perform down-stream tasks in a zero-shot setting – without any parameter or architecture modification.
Achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting.
Shows strong zero-shot results on question answering (55 F1 on CoQA).

Abstract

Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on task-specific datasets. We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText. When conditioned on a document plus questions, the answers generated by the language model reach 55 F1 on the CoQA dataset - matching or exceeding the performance of 3 out of 4 baseline systems without using the 127,000+ training examples. The capacity of the language model is essential to the success of zero-shot task transfer and increasing it improves performance in a log-linear fashion across tasks. Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting but still underfits WebText. Samples from the model reflect these improvements and contain coherent paragraphs of text. These findings suggest a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.

Citation Graph

Loading graph...

References [78]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[3]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[4]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[5]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[6]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[7]Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Chelsea Finn, P. Abbeel, Sergey Levine - 2017

4 papers in library cite

Very nice and very well explained. At first it seems that it will be tough to follow, but once they explain it it gets very intuitive.

[8]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[9]Layer Normalization

Jimmy Lei Ba, R. Kiros, Geoffrey E. Hinton - 2016

14 papers in library cite

Very nice! At first I had a little bit of prejudice because it seemed way too mathy, but actually the math is easy to follow and the results are very nice.

[10]Improving Language Understanding by Generative Pre-Training

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

23 papers in library cite

Very simple and very nice! Easy to understand and revolutionary maybe?

[11]Identity Mappings in Deep Residual Networks

K. He, X. Zhang, S. Ren, Jian Sun - 2016

4 papers in library cite

This paper is amazing - simple, easy to follow, intuitive, while also being impactful! ResNets are already a big leap forward, and they can improve on that

[12]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[13]Multitask Learning

Rich Caruana - 1997

13 papers in library cite

I expected waaaaaay more from this paper. The idea is sooooo simple and the results are underwhelming. Also, 30 pages for something that could be said in 10. The writing style is a bit boring. TBH it seems like it's just a re-writing of Caruana's PhD thesis.

[14]Overcoming Catastrophic Forgetting in Neural Networks

J. Kirkpatrick, Razvan Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. G. Barwinska, Demis Hassabis, C. Clopath, D. Kumaran, Raia Hadsell - 2017

5 papers in library cite

Very nice, intuitive, and impressive results.

[15]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[16]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[17]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[18]Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder - 2018

14 papers in library cite

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

[19]Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Z. Dai, Zhilin Yang, Yining Yang, W. Cohen, J. Carbonell, Quoc Le, Ruslan Salakhutdinov - 2019

9 papers in library cite

It's so cool to see context expansion without the need to actually expand context! Such a simple context and so effective!

[20]Pointer Networks

Oriol Vinyals, M. Fortunato, Navdeep Jaitly - 2015

10 papers in library cite

Cool concept. Nice that it works and can find good solutions for TSP.

[21]Natural Questions: A Benchmark for Question Answering Research

T. Kwiatkowski, J. Palomaki, O. Rhinehart, Michael Collins, A. P. Parikh, C. Alberti, D. Epstein, Illia Polosukhin, M. Kelcey, Jacob Devlin, K. Lee, K. N. Toutanova, Llion Jones, M. W. Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, Slav Petrov - 2019

9 papers in library cite

The dataset and methodology is very nice - it's amazing to see how Google does the summaries in search. However, the paper is too complex with the math stuff - unnecessary.

[22]Get to the Point: Summarization With Pointer-Generator Networks

A. See, P. J. Liu, Christopher D. Manning - 2017

8 papers in library cite

It's a bit of the same thing of the other ones. I am not sure if this was the first or not, but I am getting a bit bored of this

[23]Pointer Sentinel Mixture Models

S. Merity, Caiming Xiong, J. Bradbury, Richard Socher - 2017

12 papers in library cite

I really liked the methodology, but I had to read it a few times to understand it intuitively - I think they should have done a better job at explaining it.

[24]Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond

R. Nallapati, B. Zhou, C. N. D. Santos, C. G. Gulcehre, Bing Xiang - 2016

10 papers in library cite

Cool that they use pointer switching, and they introduce the CNN/daily mail dataset for summarization, which is a nice insight.

[25]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[26]Supervised Learning of Universal Sentence Representations From Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, L. Barrault, Antoine Bordes - 2017

11 papers in library cite

It's nice. Maybe the first to do NLI right (after the SNLI paper tried but failed miserably). It is simple and effective. After that people started "performance maxxxing"

[27]A Neural Conversational Model

Oriol Vinyals, Quoc V. Le - 2015

7 papers in library cite

No new methodology, no measurements, no results. It should be like 4 pages long if they didn't fill in with the conversation logs.

[28]Hierarchical Neural Story Generation

A. Fan, Martha Lewis, Yann Dauphin - 2018

4 papers in library cite

I had higher hopes for this paper. The methodology is nice but at this stage things start to get waaay to complicated

[29]Adversarial Examples for Evaluating Reading Comprehension Systems

R. Jia, Percy Liang - 2017

11 papers in library cite

I liked it a lot! It's good to see people testing things rather than just trying to beat SotA!

[30]The Winograd Schema Challenge

Hector J. Levesque, E. Davis, Leora Morgenstern - 2011

13 papers in library cite

It's amazing to see a paper that is very easy to read, opinionated, but also introducing an important contribution to the field. I love that they introduced this challenge as an alternative to the Imitation Game.

[31]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[32]Visualizing and Understanding Recurrent Networks

Li Fei Fei - 2015

3 papers in library cite

Very nice! The results were a bit underwhelming though. Nice visualizations but few explanations. Good nonetheless.

[33]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[34]CoQA: A Conversational Question Answering Challenge

Siva Reddy, Deli Chen, Christopher D. Manning - 2018

6 papers in library cite

It's a fine paper and a solid addition to QA data + NLU.

[35]Word Translation Without Parallel Data

Alexis Conneau, G. Lample, Marc'aurelio Ranzato, L. Denoyer, Hervé Jégou - 2018

3 papers in library cite

I really really liked the methodology, but it is a bit hard to read given the math jargon they use. Also, the task (word translation) is very boring and I think lacks practical usages

[36]One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

C. Chelba, Tomas Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, Tony Robinson - 2013

13 papers in library cite

It's somewhat shallow, but I can see the importance of this paper.

[37]Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher - 2017

14 papers in library cite

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

[38]Generating Wikipedia by Summarizing Long Sequences

P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, Lukasz Kaiser, Noam Shazeer - 2018

7 papers in library cite

Very nice, the first ones that realized that they could use decoder only from Transformers. Also nice that they expanded the context window, and got nice results. Very inventive to use citations from wikipedia and the first paragraph as target.

[39]Wizard of Wikipedia: Knowledge-Powered Conversational Agents

E. Dinan, S. Roller, K. Shuster, A. Fan, Michael Auli, Jason Weston - 2019

4 papers in library cite

This could be the original RAG. It's soooo close to the original thing!

[40]Unsupervised Neural Machine Translation

M. Artetxe, G. Labaka, E. Agirre, Kyunghyun Cho - 2017

4 papers in library cite

Very nice methodology! I am impressed that this is even possible. However, results are somewhat underwhelming...

[41]The LAMBADA dataset: Word Prediction Requiring a Broad Discourse Context

D. Paperno, German Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, Raquel Fernandez - 2016

12 papers in library cite

Very nice paper - very interesting methodology to building it and very good when they bring a dataset that is meant to make machines to fail

[42]The Natural Language Decathlon: Multitask Learning as Question Answering

Richard Socher - 2018

9 papers in library cite

Very nice results, but TBH the architecture is waaaay to complicated

[43]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[44]The Goldilocks Principle: Reading Children's Books With Explicit Memory Representations

F. Hill, Antoine Bordes, S. Chopra, Jason Weston - 2015

14 papers in library cite

Cool use of memory networks.

[45]Learning to Generate Reviews and Discovering Sentiment

Alec Radford, R. Jozefowicz, Ilya Sutskever - 2017

8 papers in library cite

Very impressive results. We can see that they were onto something (later became GPT2)

[46]Do CIFAR-10 Classifiers Generalize to CIFAR-10?

Vaishaal Shankar - 2018

2 papers in library cite

It's a very nice analysis and it's interesting it took so long for people to see these problems.

[47]Character-Level Language Modeling With Deeper Self-Attention

R. A. Rfou, D. Choe, Noah Constant, M. Guo, Llion Jones - 2018

6 papers in library cite

It's good, and almost a 4, but I think it is a bit boring. Plus character LMs are not meta.

[48]One Model to Learn Them All

Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, Jakob Uszkoreit - 2017

2 papers in library cite

The idea is amazing, but it gets waaaay too complex, and in the end it feels like they tested it, it didn't give good performance, and they said "just publish it as it is"

[49]Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects

M. A. Alcorn, Q. Li, Z. Gong, Caitlin Wang, L. Mai, W. S. Ku, Anh Nguyen - 2018

1 paper in library cites

Very cool the way that they propagate the parameters back to the renderer and it's nice that this is way more applicable than just plain adversarial examples

[50]Unsupervised Pretraining for Sequence to Sequence Learning

P. Ramachandran, P. J. Liu, Quoc V. Le - 2017

9 papers in library cite

It's alright, but it's the same Seq2Seq thing with pretraining

[51]Improving Neural Language Models With a Continuous Cache

E. Grave, Armand Joulin, Nicolas Usunier - 2016

7 papers in library cite

This was a surprise to me - I expected this to suck. However, they can provide a simple and intuitive way of improving LMs - nice

[52]Multilingual Language Processing From Bytes

Amarnag Subramanya - 2016

2 papers in library cite

Nothing impressive... Their main innovation was the byte stuff. Plus the old-school NLP stuff makes it boring.

[53]Dialog-Based Language Learning

J. E. Weston - 2016

1 paper in library cites

It's a nice concept but very abstract and results are very very underwhelming

[54]A BERT Baseline for the Natural Questions

C. Alberti, K. Lee, Michael Collins - 2019

2 papers in library cite

It's very simple and short, but it's nice that it set a baseline.

[55]No Training Required: Exploring Random Encoders for Sentence Classification

J. Wieting, Douwe Kiela - 2019

1 paper in library cites

It's very amazing what they can do with random projections. Plus, it's a good push for better baselines.

[56]Do We Train on Test Data? Purging CIFAR of Near-Duplicates

Joachim Denzler - 2020

1 paper in library cites

It's a bit derivative from the other paper regarding CIFAR. It's good though, solid addition.

[57]Learning and Evaluating General Linguistic Intelligence

D. Yogatama, C. D. M. D'autume, J. Connor, T. Kocisky, M. Chrzanowski, L. Kong, A. Lazaridou, W. Ling, Longhui Yu, C. Dyer - 2019

2 papers in library cite

It's a nice contribution and good analysis on generalization, but TBH I expected a bit more given the title.

[58]Embracing Data Abundance: Booktest Dataset for Reading Comprehension

O. Bajgar, R. Kadlec, Jan Kleindienst - 2016

2 papers in library cite

A bit poorly written, throws some shade in other papers, and very marginal contributions (they just perform the same data extraction as others on more data)

[59]Deep Speech 2: End-to-End Speech Recognition in English and mandarin

Dario Amodei, S. Ananthanarayanan, R. Anubhai, Jinze Bai, E. Battenberg, C. Case, J. Casper, Bryan Catanzaro, Q. Cheng, Guanduo Chen - 2016

3 papers in library cite

Speech recog. improved

[60]Building Machines That Learn and Think Like People

B. M. Lake, T. D. Ullman, Joshua B. Tenenbaum, S. J. Gershman - 2017

1 paper in library cites

Sounds really releant

[61]Neural Word Embeddings as Implicit Matrix Factorization

Omer Levy, Y. Goldberg - 2014

4 papers in library cite

Improve understanding of embeddings

[62]Deep Learning Scaling Is Predictable, Empirically

J. Hestness, S. Narang, N. Ardalani, G. Diamos, Heewoo Jun, H. Kianinejad, M. Patwary, M. Ali, Yining Yang, Y. Zhou - 2017

5 papers in library cite

[63]Unsupervised Machine Translation Using monolingual corpora Only

G. Lample, L. Denoyer, Marc'aurelio Ranzato - 2017

4 papers in library cite

How?

[64]Transfertransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents

Thomas Wolf, V. Sanh, J. Chaumond, C. Delangue - 2019

1 paper in library cites

Generative dialog

[65]A Simple Method for Commonsense Reasoning

T. H. Trinh, Quoc V. Le - 2018

4 papers in library cite

[66]Learning General Purpose Distributed Sentence Representations via Large Scale Multi-Task Learning

S. Subramanian, A. Trischler, Yoshua Bengio, C. Pal - 2018

4 papers in library cite

Bengio

[67]On the Evaluation of Common-Sense Reasoning in Natural Language Understanding

P. Trichelair, A. Emami, J. C. K. Cheung, A. Trischler, K. Suleman, F. Diaz - 2018

1 paper in library cites

Changed name, seems a bit flopped but :shrug:

[68]Universal Transformers

Mostafa Dehghani, S. Gouws, Oriol Vinyals, Jakob Uszkoreit, Lukasz Kaiser - 2018

6 papers in library cite

[69]Interpolated Estimation of Markov Source Parameters From Sparse Data

Frederick Jelinek, R. L. Mercer - 1980

8 papers in library cite

[70]Bottom-Up Abstractive Summarization

Sebastian Gehrmann, Y. Deng, Alexander M. Rush - 2018

2 papers in library cite

[71]An Effective Approach to Unsupervised Machine Translation

M. Artetxe, G. Labaka, E. Agirre - 2019

1 paper in library cites

[72]Content Extraction Using Diverse Feature Sets

M. E. Peters, D. Lecocq - 2013

1 paper in library cites

[73]Entity Tracking Improves Cloze-Style Reading Comprehension

L. Hoang, Sam Wiseman, Alexander M. Rush - 2018

1 paper in library cites

[74]Frage: Frequency-Agnostic Word Representation

C. Gong, D. He, X. Tan, T. Qin, Lisa Wang, T. Y. Liu - 2018

1 paper in library cites

[75]Looking for Elmo's Friends: Sentence-Level Pretraining Beyond Language Modeling

Samuel R. Bowman, Ellie Pavlick, E. Grave, B. V. Durme, A. Wang, J. Hula, P. Xia, R. Pappagari, R. T. Mccoy, R. Patel - 2018

1 paper in library cites

[76]Story Cloze Task: Uw NLP System

Richard Schwartz, Maarten Sap, I. Konstas, L. Zilles, Yejin Choi, Noah A. Smith - 2017

1 paper in library cites

[77]The 14 Billion Word Iweb Corpus

M. Davies - 2018

1 paper in library cites

[78]Towards Principled Unsupervised Learning

Ilya Sutskever, R. Jozefowicz, K. Gregor, D. Rezende, T. Lillicrap, Oriol Vinyals - 2015

1 paper in library cites

Cited by

27

papers in your library

Cites

68

papers in your library

Read

on October 28, 2025

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

Tags

Paper Aliases

No aliases