Papperoni

2019

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Z. Dai, Yining Yang, J. Carbonell, Ruslan Salakhutdinov, Quoc V. Le

Open PDF Google Scholar

citations

Cite Score

87

AI summary

The paper introduces XLNet, a generalized autoregressive pretraining method, which enables learning bidirectional contexts, overcomes BERT limitations via autoregressive formulation, and integrates Transformer-XL ideas, achieving superior performance on 20 tasks.

Main Contributions

Introduces a generalized autoregressive pretraining method called XLNet.
Enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order.
Overcomes the limitations of BERT thanks to its autoregressive formulation.
Integrates ideas from Transformer-XL into pretraining.
Achieves state-of-the-art results on 20 tasks, outperforming BERT on question answering, natural language inference, sentiment analysis, and document ranking.

Abstract

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

Citation Graph

Loading graph...

References [40]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[3]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[4]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[5]Improving Language Understanding by Generative Pre-Training

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

23 papers in library cite

Very simple and very nice! Easy to understand and revolutionary maybe?

[6]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[7]ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations

Z. Lan, Mark Chen, S. Goodman, Kevin Gimpel, P. Sharma, Radu Soricut - 2019

8 papers in library cite

I like how they can achieve very close results with very few params! Very nice tricks to do that as well.

[8]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[9]Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder - 2018

14 papers in library cite

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

[10]Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Z. Dai, Zhilin Yang, Yining Yang, W. Cohen, J. Carbonell, Quoc Le, Ruslan Salakhutdinov - 2019

9 papers in library cite

It's so cool to see context expansion without the need to actually expand context! Such a simple context and so effective!

[11]SentencePiece: A Simple and Language Independent subword Tokenizer and Detokenizer for Neural Text Processing

John Richardson - 2018

3 papers in library cite

I like that this is just a simple library that wraps functionality. It's also minimal, so can be included in a bunch of places.

[12]Know What You Don't Know: Un-Answerable Questions for SQuAD

P. Rajpurkar, R. Jia, Percy Liang - 2018

14 papers in library cite

It's alright... It's an extension to the other paper/dataset. I feel that it didn't need to be a full paper (maybe a 6-pager).

[13]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[14]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[15]RACE: Large-Scale Reading Comprehension Dataset From Examinations

Guokun Lai, Q. Xie, Haozhe Liu, Yining Yang, Eduard Hovy - 2017

11 papers in library cite

I really like the idea of using human tests for testing AI. Also, very nice insige for using chinese tests!

[16]Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher - 2017

14 papers in library cite

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

[17]Character-Level Language Modeling With Deeper Self-Attention

R. A. Rfou, D. Choe, Noah Constant, M. Guo, Llion Jones - 2018

6 papers in library cite

It's good, and almost a 4, but I think it is a bit boring. Plus character LMs are not meta.

[18]Adaptive Input Representations for Neural Language Modeling

A. Baevski, Michael Auli - 2018

3 papers in library cite

I like the idea and it's a nice paper. Maybe it is a bit outdated now that everything is wordpiece, but I like that they reduce the number of parameters.

[19]Character-Level Convolutional Networks for Text Classification

X. Zhang, J. Zhao, Yann Lecun - 2015

7 papers in library cite

LeCun + Character CNNs

[20]Multi-Task Deep Neural Networks for Natural Language Understanding

Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao - 2019

6 papers in library cite

[21]Deep Pyramid Convolutional Neural Networks for Text Categorization

R. Johnson, Tong Zhang - 2017

2 papers in library cite

SotA language categorization

[22]Maskgan: Better Text Generation via Filling in The

William Fedus, I. Goodfellow, A. M. Dai - 2018

2 papers in library cite

Masked task for filling words before BERT, goodfellow

[23]Pixel Recurrent Neural Networks

A. V. D. Oord, N. Kalchbrenner, Koray Kavukcuoglu - 2016

3 papers in library cite

Seems like they introduce input masking

[24]A Surprisingly Robust Trick for Winograd Schema Challenge

V. Kocijan, A. M. Cretu, O. M. Camburu, Y. Yordanov, T. Lukasiewicz - 2019

4 papers in library cite

[25]Adversarial Training Methods for Semi-Supervised Text Classification

T. Miyato, A. M. Dai, I. Goodfellow - 2016

4 papers in library cite

[26]Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

Zhilin Yang, Z. Dai, Ruslan Salakhutdinov, W. W. Cohen - 2017

4 papers in library cite

[27]BAM! Born-Again Multi-Task Networks for Natural Language Understanding

K. Clark, M. Luong, U. Khandelwal, C. Manning, Quoc Le - 2019

3 papers in library cite

[28]Modeling High-Dimensional Discrete Data With Multi-Layer Neural Networks

Yoshua Bengio, Samy Bengio - 2000

3 papers in library cite

[29]A Deep Relevance Matching Model for Ad-Hoc Retrieval

J. Guo, Yu Fan, Q. Ai, W. B. Croft - 2016

2 papers in library cite

[30]Clueweb09 Data Set

J. Callan, M. Hoy, C. Yoo, L. Zhao - 2009

2 papers in library cite

[31]Common Crawl

C. Crawl - 2019

2 papers in library cite

[32]Convolutional Neural Networks for Soft-Matching N-Grams in Ad-Hoc Search

Z. Dai, Caiming Xiong, J. Callan, Ze Liu - 2018

2 papers in library cite

[33]DCMN+: Dual Co-Matching Network for Multi-Choice Reading Comprehension

S. Zhang, H. Zhao, Yonghui Wu, Zhengyou Zhang, Xinyu Zhou, Xinyu Zhou - 2019

2 papers in library cite

[34]End-to-End Neural Ad-Hoc Ranking With Kernel Pooling

Caiming Xiong, Z. Dai, J. Callan, Ze Liu, Russell Power - 2017

2 papers in library cite

[35]English Gigaword fifth Edition, Linguistic Data Consortium

R. Parker, D. Graff, J. Kong, K. Chen, K. Maeda - 2011

1 paper in library cites

[36]Improving Question Answering With External Knowledge

Xuehai Pan, Ke Sun, D. Yu, H. Ji, D. Yu - 2019

1 paper in library cites

[37]Made: Masked Autoencoder for Distribution Estimation

M. Germain, K. Gregor, I. Murray, Hugo Larochelle - 2015

1 paper in library cites

[38]Neural Autoregressive Distribution Estimation

B. Uria, M. A. Cote, K. Gregor, I. Murray, Hugo Larochelle - 2016

1 paper in library cites

[39]Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function

D. S. Sachan, M. Zaheer, Ruslan Salakhutdinov - 2018

1 paper in library cites

[40]Unsupervised Data Augmentation

Q. Xie, Z. Dai, Eduard Hovy, M. T. Luong, Quoc V. Le - 2019

1 paper in library cites

Cited by

11

papers in your library

Cites

23

papers in your library

Read

on November 17, 2025

The method is nice and the results are very good but this paper is just soooo hard to follow...

Tags

Paper Aliases

No aliases