Papperoni

2019

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov

Open PDF Google Scholar

citations

Cite Score

92

AI summary

This paper introduces RoBERTa, a robustly optimized BERT pretraining approach, using dynamic masking, FULL-SENTENCES without NSP loss, large mini-batches, and a larger byte-level BPE. It achieves state-of-the-art results on GLUE, RACE and SQUAD without multi-task finetuning.

Main Contributions

Introduces a set of important BERT design choices and training strategies.
Introduces a novel dataset, CC-NEWS.
Confirms that using more data for pre-training further improves performance on downstream tasks.
Shows that masked language model pretraining, under the right design choices, is competitive with all other recently published methods.
Releases the model, pretraining and fine-tuning code implemented in PyTorch.

Abstract

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQUAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

Citation Graph

Loading graph...

References [50]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[3]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[4]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[5]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[6]Automatic Differentiation in PyTorch

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Devito, Zongyu Lin, A. Desmaison, L. Antiga, Adam Lerer - 2017

3 papers in library cite

Very boring and poorly explained. It looks like they said "we need to cash in some citations! quick, publish anything you have!!!"

[7]XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Z. Dai, Yining Yang, J. Carbonell, Ruslan Salakhutdinov, Quoc V. Le - 2019

11 papers in library cite

The method is nice and the results are very good but this paper is just soooo hard to follow...

[8]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[9]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[10]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[11]Gaussian Error Linear Units (Gelus)

Dan Hendrycks, Kevin Gimpel - 2016

5 papers in library cite

Very understandable, and very nice! I don't think the justification is good, but hey, it works!

[12]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[13]Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder - 2018

14 papers in library cite

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

[14]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[15]A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman - 2018

19 papers in library cite

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

[16]Know What You Don't Know: Un-Answerable Questions for SQuAD

P. Rajpurkar, R. Jia, Percy Liang - 2018

14 papers in library cite

It's alright... It's an extension to the other paper/dataset. I feel that it didn't need to be a full paper (maybe a 6-pager).

[17]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[18]SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A. Wang, Y. Pruksachatkun, Nikita Nangia, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2019

15 papers in library cite

Nothing too surprising, just getting a bunch of stuff from different places and putting it all together. At least they do a good analysis of the benchmark vs. existing methodologies.

[19]The PASCAL Recognising Textual Entailment Challenge

Ido Dagan, O. Glickman, Bernardo Magnini - 2005

19 papers in library cite

It's very nice how they had the foresight to create a challenge that became relevant like 10 years later.

[20]SpanBERT: Improving Pre-Training by Representing and Predicting Spans

M. Joshi, Deli Chen, Yibo Liu, D. Weld, Luke Zettlemoyer, Omer Levy - 2019

5 papers in library cite

Yet another BERT, and it's not surprising since I've seen this in other papers. But is good nonetheless.

[21]Automatically Constructing a Corpus of Sentential Paraphrases

W. Dolan, Chris Brockett - 2005

9 papers in library cite

Small dataset, questionable methodology, not useful for training models

[22]The Winograd Schema Challenge

Hector J. Levesque, E. Davis, Leora Morgenstern - 2011

13 papers in library cite

It's amazing to see a paper that is very easy to read, opinionated, but also introducing an important contribution to the field. I love that they introduced this challenge as an alternative to the Imitation Game.

[23]Cross-Lingual Language Model Pretraining

G. Lample, Alexis Conneau - 2019

5 papers in library cite

I like how they learn cross-lingual stuff via vocabulary sharing. it's probably the start of the multi-language LMs.

[24]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[25]RACE: Large-Scale Reading Comprehension Dataset From Examinations

Guokun Lai, Q. Xie, Haozhe Liu, Yining Yang, Eduard Hovy - 2017

11 papers in library cite

I really like the idea of using human tests for testing AI. Also, very nice insige for using chinese tests!

[26]Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher - 2017

14 papers in library cite

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

[27]Reducing BERT Pre-Training Time From 3 Days to 76 Minutes

Y. You, Jeffrey Li, J. Hseu, X. Song, J. Demmel, Cho Jui Hsieh - 2019

2 papers in library cite

The changes are very simple and very derivative, but they make it so complicated! Also, they don't discuss drawbacks, which I am sure there are many!

[28]fairseq: A Fast, Extensible Toolkit for Sequence Modeling

M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, Michael Auli - 2019

4 papers in library cite

[29]Unified Language Model Pre-Training for Natural Language Understanding and Generation

L. Dong, N. Yang, Wenyi Wang, F. Wei, Xiaodong Liu, Yuzhi Wang, Jianfeng Gao, M. Zhou, H. W. Hon - 2019

4 papers in library cite

[30]Neural Network Acceptability Judgments

Alex Warstadt, A. Singh, S. Bowman - 2018

8 papers in library cite

CoLA dataset

[31]Multi-Task Deep Neural Networks for Natural Language Understanding

Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao - 2019

6 papers in library cite

[32]Defending Against Neural Fake News

Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, F. Roesner, Yejin Choi - 2019

5 papers in library cite

[33]Mass: Masked Sequence to Sequence Pre-Training for Language Generation

K. Song, X. Tan, T. Qin, J. Lu, T. Y. Liu - 2019

5 papers in library cite

[34]A Simple Method for Commonsense Reasoning

T. H. Trinh, Quoc V. Le - 2018

4 papers in library cite

[35]The Fifth PASCAL Recognizing Textual Entailment Challenge

L. Bentivogli, Peter Clark, Ido Dagan, D. Giampiccolo - 2009

7 papers in library cite

[36]The Third PASCAL Recognizing Textual Entailment Challenge

D. Giampiccolo, Bernardo Magnini, Ido Dagan, B. Dolan - 2007

7 papers in library cite

[37]The Second PASCAL Recognising Textual Entailment Challenge

R. B. Haim, Ido Dagan, B. Dolan, L. Ferro, D. Giampiccolo, Bernardo Magnini, I. Szpektor - 2006

6 papers in library cite

[38]A Surprisingly Robust Trick for Winograd Schema Challenge

V. Kocijan, A. M. Cretu, O. M. Camburu, Y. Yordanov, T. Lukasiewicz - 2019

4 papers in library cite

[39]Improving Language Understanding With Unsupervised Learning

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

4 papers in library cite

[40]Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding

Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao - 2019

3 papers in library cite

[41]Scaling Neural Machine Translation

M. Ott, S. Edunov, D. Grangier, Michael Auli - 2018

3 papers in library cite

[42]spaCy 2: Natural Language Understanding With Bloom Embeddings, Convolutional Neural Networks and Incremental Parsing

M. Honnibal, I. Montani - 2017

3 papers in library cite

[43]ERNIE: Enhanced Representation Through Knowledge Integration

Y. S. Sun, Shijie Wang, Yiwei Li, S. Feng, X. Chen, Haowei Zhang, X. Tian, D. Zhu, H. Tian, H. Wu - 2019

2 papers in library cite

[44]First quora dataset release: Question Pairs

S. Iyer, N. Dandekar, K. Csernai - 2017

2 papers in library cite

[45]KERMIT: Generative Insertion-Based Modeling for Sequences

W. Chan, N. Kitaev, K. Guu, M. Stern, Jakob Uszkoreit - 2019

2 papers in library cite

[46]Openwebtext corpus

A. Gokaslan, V. Cohen - 2019

2 papers in library cite

[47]Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval 2007)

E. Agirre, L. M'arquez, R. Wicentowski - 2007

2 papers in library cite

S. Nagel - 2016

1 paper in library cites

[49]Cloze-Driven Pretraining of Self-Attention Networks

A. Baevski, S. Edunov, Yibo Liu, Luke Zettlemoyer, Michael Auli - 2019

1 paper in library cites

[50]News-Please: A Generic News Crawler and Extractor

F. Hamborg, N. Meuschke, C. Breitinger, B. Gipp - 2017

1 paper in library cites

Cited by

17

papers in your library

Cites

34

papers in your library

Read

on November 18, 2025

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

Tags

Paper Aliases

No aliases