Papperoni

2018

BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova

Open PDF Google Scholar

citations

Cite Score

99

AI summary

This paper introduces BERT, a new language representation model leveraging bidirectional transformers, achieving state-of-the-art results on GLUE, MultiNLI, and SQUAD benchmarks. It pre-trains deep bidirectional representations from unlabeled text and can be fine-tuned for various tasks without significant task-specific modifications.

Main Contributions

Demonstrates the importance of bidirectional pre-training for language representations using masked language models.
Shows that pre-trained representations reduce the need for heavily-engineered task-specific architectures.
Achieves state-of-the-art performance on a large suite of sentence-level and token-level tasks.
Advances the state of the art for eleven NLP tasks.
Introduces a next sentence prediction task that jointly pre-trains text-pair representations.

Abstract

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQUAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQUAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Citation Graph

Loading graph...

References [56]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]ImageNet: A Large-Scale Hierarchical Image Database

J. Deng, W. Dong, Richard Socher, L. J. Li, K. Li, Li Fei Fei - 2009

28 papers in library cite

Very nice idea and huge impact!

[3]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[4]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[5]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[6]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[7]How Transferable Are Features in Deep Neural Networks?

Hod Lipson - 2014

2 papers in library cite

Such a good paper to analyze pretraining behavior: simple, intuitive, nice to read

[8]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[9]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[10]Extracting and Composing Robust Features With Denoising Autoencoders

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre Antoine Manzagol - 2008

25 papers in library cite

I am *so* glad we found an alternative to DBNs. Also, introduced the idea of denoising which is nice.

[11]Google's Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation

Yonghui Wu, M. Schuster, Ziru Chen, Quoc V. Le, M. Norouzi, W. Macherey, M. Krikun, Yue Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. J. Johnson, Xiaodong Liu, Lukasz Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, Wenyi Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, Oriol Vinyals, G. S. Corrado, M. Hughes, Jeffrey Dean - 2016

15 papers in library cite

It's a very good paper but TBH doesn't bring anything new other than joining a bunch of existing stuff. I think it ended up being foundational because it's Google and several people used it as a base for future research. Good contribution then :)

[12]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[13]Bridging Nonlinearities and Stochastic Regularizers With Gaussian Error Linear Units

Dan Hendrycks, Kevin Gimpel - 2016

9 papers in library cite

Very understandable, and very nice! I don't think the justification is good, but hey, it works!

[14]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[15]Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder - 2018

14 papers in library cite

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

[16]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[17]A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman - 2018

19 papers in library cite

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

[18]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[19]TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer - 2017

18 papers in library cite

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

[20]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[21]Word Representations: A Simple and General Method for Semi-Supervised Learning

J. Turian, L. Ratinov, Yoshua Bengio - 2010

17 papers in library cite

This basically introduced the concept of embeddings. Very nice.

[22]Supervised Learning of Universal Sentence Representations From Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, L. Barrault, Antoine Bordes - 2017

11 papers in library cite

It's nice. Maybe the first to do NLI right (after the SNLI paper tried but failed miserably). It is simple and effective. After that people started "performance maxxxing"

[23]Bidirectional Attention Flow for Machine Comprehension

M. Seo, A. Kembhavi, Ali Farhadi, Hananneh Hajishirzi - 2017

13 papers in library cite

It's alright but the method seems absurdly complex. Maybe I am a bit biased because it's like the 20th paper that I read with attention + LSTMs...

[24]Automatically Constructing a Corpus of Sentential Paraphrases

W. Dolan, Chris Brockett - 2005

9 papers in library cite

Small dataset, questionable methodology, not useful for training models

[25]The Winograd Schema Challenge

Hector J. Levesque, E. Davis, Leora Morgenstern - 2011

13 papers in library cite

It's amazing to see a paper that is very easy to read, opinionated, but also introducing an important contribution to the field. I love that they introduced this challenge as an alternative to the Imitation Game.

[26]A Framework for Learning Predictive Structures From Multiple Tasks and Unlabeled Data

Rie Kubota Ando, Tong Zhang - 2005

10 papers in library cite

Very nice and clever way of solving the problem of semi-supervised learning, and makes a lot of sense. I give them more credit for formalizing the concept. The methodology is a bit boring.

[27]A Decomposable Attention Model for Natural Language Inference

A. P. Parikh, O. Tackstrom, Dipanjan Das, Jakob Uszkoreit - 2016

11 papers in library cite

Very nice alternative to the common LSTM encoder-decoder architecture! Seems similar o the Transformers arch in the sense that they don't use RNNs. Nice that they analyze computational complexity as well.

[28]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[29]One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

C. Chelba, Tomas Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, Tony Robinson - 2013

13 papers in library cite

It's somewhat shallow, but I can see the importance of this paper.

[30]Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher - 2017

14 papers in library cite

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

[31]A Scalable Hierarchical Distributed Language Model

A. Mnih, Geoffrey E. Hinton - 2009

16 papers in library cite

Good paper that introduces hierarchical trees as an alternative to the expensive softmax output. I think this is not really relevant anymore, but good read.

[32]Semi-Supervised Sequence Tagging With Bidirectional Language Models

M. E. Peters, W. Ammar, C. Bhagavatula, Russell Power - 2017

5 papers in library cite

Nothing too different from other LSTM + Attention papers

[33]SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Yejin Choi - 2018

5 papers in library cite

Very nice methodology with the adversarial stuff! I think not too relevant though.

[34]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[35]context2vec: Learning Generic Context Embedding With Bidirectional LSTM

O. Melamud, J. Goldberger, Ido Dagan - 2016

5 papers in library cite

I liked the methodology a lot but the paper reads like an MSc course project.

[36]Simple and Effective Multi-Paragraph Reading Comprehension

C. Clark, Matt Gardner - 2017

7 papers in library cite

Very nice paper! I think it's a stretch to call it "simple", but the paper is very well written and easy to follow.

[37]Character-Level Language Modeling With Deeper Self-Attention

R. A. Rfou, D. Choe, Noah Constant, M. Guo, Llion Jones - 2018

6 papers in library cite

It's good, and almost a 4, but I think it is a bit boring. Plus character LMs are not meta.

[38]Neural Network Acceptability Judgments

Alex Warstadt, A. Singh, S. Bowman - 2018

8 papers in library cite

CoLA dataset

[39]Maskgan: Better Text Generation via Filling in The

William Fedus, I. Goodfellow, A. M. Dai - 2018

2 papers in library cite

Masked task for filling words before BERT, goodfellow

[40]Class-Based N-Gram Models of Natural Language

P. F. Brown, P. V. Desouza, R. L. Mercer, Vincent J. Della Pietra, J. C. Lai - 1992

12 papers in library cite

[41]The Fifth PASCAL Recognizing Textual Entailment Challenge

L. Bentivogli, Peter Clark, Ido Dagan, D. Giampiccolo - 2009

7 papers in library cite

[42]SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

D. Cer, M. Diab, E. Agirre, I. L. Gazpio, L. Specia - 2017

6 papers in library cite

[43]Cloze Procedure: A New Tool for Measuring Readability

W. L. Taylor - 1953

4 papers in library cite

[44]Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

Yacine Jernite, S. Bowman, D. Sontag - 2017

4 papers in library cite

[45]Dissecting Contextual Word Embeddings: Architecture and Representation

M. E. Peters, M. Neumann, Luke Zettlemoyer, W. T. Yih - 2018

4 papers in library cite

[46]Domain Adaptation With Structural Correspondence Learning

John Blitzer, R. Mcdonald, Fernando Pereira - 2006

4 papers in library cite

[47]Improving Language Understanding With Unsupervised Learning

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

4 papers in library cite

[48]Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

E. F. T. K. Sang, F. D. Meulder - 2003

4 papers in library cite

[49]An Efficient Framework for Learning Sentence Representations

L. Logeswaran, Honglak Lee - 2018

3 papers in library cite

[50]Qanet: Combining Local Convolution With Global Self-Attention for Reading Compre- Hension

A. W. Yu, David Dohan, M. T. Luong, R. Zhao, K. Chen, M. Norouzi, Quoc V. Le - 2018

3 papers in library cite

[51]Reinforced Mnemonic Reader for Machine Reading Comprehension

M. Hu, Y. Peng, Zhongqiang Huang, X. Qiu, F. Wei, M. Zhou - 2018

3 papers in library cite

[52]Quora Question Pairs

Ziru Chen, Haowei Zhang, X. Zhang, L. Zhao - 2018

2 papers in library cite

[53]Contextual String Embeddings for Sequence Labeling

A. Akbik, D. Blythe, R. Vollgraf - 2018

1 paper in library cites

[54]Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering

Wenyi Wang, Minghao Yan, Chiyu Wu - 2018

1 paper in library cites

[55]Semi-Supervised Sequence Modeling With Cross-View Training

K. Clark, M. T. Luong, Christopher D. Manning, Quoc Le - 2018

1 paper in library cites

[56]U-Net: Machine Reading Comprehension With Unanswerable Questions

F. Sun, Lei Li, X. Qiu, Yibo Liu - 2018

1 paper in library cites

Cited by

39

papers in your library

Cites

39

papers in your library

Read

on October 17, 2025

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

Tags

Paper Aliases

No aliases