Papperoni

2019

SpanBERT: Improving Pre-Training by Representing and Predicting Spans

M. Joshi, Deli Chen, Yibo Liu, D. Weld, Luke Zettlemoyer, Omer Levy

Open PDF Google Scholar

citations

Cite Score

60

AI summary

This paper introduces SpanBERT, a new pre-training method that is designed to represent and predict spans of text by masking contiguous random spans and training the span boundary representations to predict the entire content of the masked span, achieving state-of-the-art results on SQuAD1.1, SQuAD 2.0, and OntoNotes benchmarks.

Main Contributions

Introduces a new pre-training method called SpanBERT that is designed to better represent and predict spans of text.
Extends BERT by masking contiguous random spans, rather than random tokens.
Trains the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it.
Achieves state-of-the-art results on the OntoNotes coreference resolution task (79.6% F1).
Achieves strong performance on the TACRED relation extraction benchmark, and even gains on GLUE.

Abstract

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERTlarge, our single model obtains 94.6% and 88.7% F1 on SQUAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE.

Citation Graph

Loading graph...

References [53]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[3]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[4]Decoupled Weight Decay Regularization

I. Loshchilov, Frank Hutter - 2017

7 papers in library cite

Very nice! Very simple to understand despite the complex topic. Very good results also

[5]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[6]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[7]Layer Normalization

Jimmy Lei Ba, R. Kiros, Geoffrey E. Hinton - 2016

14 papers in library cite

Very nice! At first I had a little bit of prejudice because it seemed way too mathy, but actually the math is easy to follow and the results are very nice.

[8]XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Z. Dai, Yining Yang, J. Carbonell, Ruslan Salakhutdinov, Quoc V. Le - 2019

11 papers in library cite

The method is nice and the results are very good but this paper is just soooo hard to follow...

[9]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[10]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[11]Gaussian Error Linear Units (Gelus)

Dan Hendrycks, Kevin Gimpel - 2016

5 papers in library cite

Very understandable, and very nice! I don't think the justification is good, but hey, it works!

[12]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[13]Huggingface's Transformers: State-of-the-Art Natural Language Processing

Thomas Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, Sam Shleifer, P. V. Platen, C. Ma, Yacine Jernite, J. Plu, Chenfeng Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, Alexander M. Rush - 2019

7 papers in library cite

It's a solid paper, but these framework papers are usually boring... Yet, it's nice to see that they started with reduced scope (and how large it got now).

[14]Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder - 2018

14 papers in library cite

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

[15]A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman - 2018

19 papers in library cite

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

[16]Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Z. Dai, Zhilin Yang, Yining Yang, W. Cohen, J. Carbonell, Quoc Le, Ruslan Salakhutdinov - 2019

9 papers in library cite

It's so cool to see context expansion without the need to actually expand context! Such a simple context and so effective!

[17]Natural Questions: A Benchmark for Question Answering Research

T. Kwiatkowski, J. Palomaki, O. Rhinehart, Michael Collins, A. P. Parikh, C. Alberti, D. Epstein, Illia Polosukhin, M. Kelcey, Jacob Devlin, K. Lee, K. N. Toutanova, Llion Jones, M. W. Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, Slav Petrov - 2019

9 papers in library cite

The dataset and methodology is very nice - it's amazing to see how Google does the summaries in search. However, the paper is too complex with the math stuff - unnecessary.

[18]Know What You Don't Know: Un-Answerable Questions for SQuAD

P. Rajpurkar, R. Jia, Percy Liang - 2018

14 papers in library cite

It's alright... It's an extension to the other paper/dataset. I feel that it didn't need to be a full paper (maybe a 6-pager).

[19]TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer - 2017

18 papers in library cite

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

[20]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[21]The PASCAL Recognising Textual Entailment Challenge

Ido Dagan, O. Glickman, Bernardo Magnini - 2005

19 papers in library cite

It's very nice how they had the foresight to create a challenge that became relevant like 10 years later.

[22]Automatically Constructing a Corpus of Sentential Paraphrases

W. Dolan, Chris Brockett - 2005

9 papers in library cite

Small dataset, questionable methodology, not useful for training models

[23]The Winograd Schema Challenge

Hector J. Levesque, E. Davis, Leora Morgenstern - 2011

13 papers in library cite

It's amazing to see a paper that is very easy to read, opinionated, but also introducing an important contribution to the field. I love that they introduced this challenge as an alternative to the Imitation Game.

[24]Cross-Lingual Language Model Pretraining

G. Lample, Alexis Conneau - 2019

5 papers in library cite

I like how they learn cross-lingual stuff via vocabulary sharing. it's probably the start of the multi-language LMs.

[25]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[26]Using the Output Embedding to Improve Language Models

O. Press, Lior Wolf - 2017

7 papers in library cite

I did not like this paper at all - The paper is not bad, it's just that I expected *way* more. Good results but uninteresting

[27]context2vec: Learning Generic Context Embedding With Bidirectional LSTM

O. Melamud, J. Goldberger, Ido Dagan - 2016

5 papers in library cite

I liked the methodology a lot but the paper reads like an MSc course project.

[28]Hotpotqa: A Dataset for Diverse, Explainable Multi-Hop Question Answering

Zhilin Yang, P. Qi, S. Zhang, Yoshua Bengio, W. Cohen, Ruslan Salakhutdinov, Christopher D. Manning - 2018

4 papers in library cite

[29]fairseq: A Fast, Extensible Toolkit for Sequence Modeling

M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, Michael Auli - 2019

4 papers in library cite

[30]Unified Language Model Pre-Training for Natural Language Understanding and Generation

L. Dong, N. Yang, Wenyi Wang, F. Wei, Xiaodong Liu, Yuzhi Wang, Jianfeng Gao, M. Zhou, H. W. Hon - 2019

4 papers in library cite

[31]Neural Network Acceptability Judgments

Alex Warstadt, A. Singh, S. Bowman - 2018

8 papers in library cite

CoLA dataset

[32]Multi-Task Deep Neural Networks for Natural Language Understanding

Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao - 2019

6 papers in library cite

[33]Mass: Masked Sequence to Sequence Pre-Training for Language Generation

K. Song, X. Tan, T. Qin, J. Lu, T. Y. Liu - 2019

5 papers in library cite

[34]SearchQA: A New Q&A Dataset Augmented With Context From a Search Engine

M. Dunn, L. Sagun, M. Higgins, V. U. Guney, V. Cirik, Kyunghyun Cho - 2017

5 papers in library cite

[35]The Third PASCAL Recognizing Textual Entailment Challenge

D. Giampiccolo, Bernardo Magnini, Ido Dagan, B. Dolan - 2007

7 papers in library cite

[36]SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

D. Cer, M. Diab, E. Agirre, I. L. Gazpio, L. Specia - 2017

6 papers in library cite

[37]The Second PASCAL Recognising Textual Entailment Challenge

R. B. Haim, Ido Dagan, B. Dolan, L. Ferro, D. Giampiccolo, Bernardo Magnini, I. Szpektor - 2006

6 papers in library cite

[38]Improving Language Understanding With Unsupervised Learning

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

4 papers in library cite

[39]An Efficient Framework for Learning Sentence Representations

L. Logeswaran, Honglak Lee - 2018

3 papers in library cite

[40]End-to-End Neural Coreference Resolution

K. Lee, Luheng He, Martha Lewis, L. S. Zettlemoyer - 2017

3 papers in library cite

[41]Learning Recurrent Span Representations for Extractive Question Answering

K. Lee, S. Salant, T. Kwiatkowski, A. P. Parikh, Dipanjan Das, Jonathan Berant - 2017

3 papers in library cite

[42]Position-Aware Attention and Supervised Data Improve Slot Filling

Y. Z. Zhang, Victor Zhong, Deli Chen, G. Angeli, Christopher D. Manning - 2017

3 papers in library cite

[43]Conll-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in Ontonotes

S. Pradhan, A. Moschitti, N. Xue, O. Uryupina, Y. Z. Zhang - 2012

2 papers in library cite

[44]ERNIE: Enhanced Representation Through Knowledge Integration

Y. S. Sun, Shijie Wang, Yiwei Li, S. Feng, X. Chen, Haowei Zhang, X. Tian, D. Zhu, H. Tian, H. Wu - 2019

2 papers in library cite

[45]KERMIT: Generative Insertion-Based Modeling for Sequences

W. Chan, N. Kitaev, K. Guu, M. Stern, Jakob Uszkoreit - 2019

2 papers in library cite

[46]BERT for Coreference Resolution: Baselines and Analysis

M. Joshi, Omer Levy, D. S. Weld, Luke Zettlemoyer, Omer Levy - 2019

1 paper in library cites

[47]Blockwise Parallel Decoding for Deep Autoregressive Models

M. Stern, Noam Shazeer, Jakob Uszkoreit - 2018

1 paper in library cites

[48]ERNIE: Enhanced Language Representation With Informative Entities

Zhengyou Zhang, X. Han, Ze Liu, Xu Jiang, Maosong Sun, Qian Liu - 2019

1 paper in library cites

[49]Higher-Order Coreference Resolution With Coarse-to-Fine Inference

K. Lee, Luheng He, Luke Zettlemoyer - 2018

1 paper in library cites

[50]Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling

Luheng He, K. Lee, Omer Levy, Luke Zettlemoyer - 2018

1 paper in library cites

[51]Matching the Blanks: Distributional Similarity for Relation Learning

L. B. Soares, N. A. Fitzgerald, J. Ling, T. Kwiatkowski - 2019

1 paper in library cites

[52]MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension

Adam Fisch, A. Talmor, R. Jia, M. Seo, E. Choi, Deli Chen - 2019

1 paper in library cites

[53]Pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference

M. Joshi, E. Choi, Omer Levy, D. Weld, Luke Zettlemoyer - 2019

1 paper in library cites

Cited by

5

papers in your library

Cites

34

papers in your library

Read

on December 29, 2025

Yet another BERT, and it's not surprising since I've seen this in other papers. But is good nonetheless.

Tags

Paper Aliases

No aliases