Papperoni

2019

ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations

Z. Lan, Mark Chen, S. Goodman, Kevin Gimpel, P. Sharma, Radu Soricut

Open PDF Google Scholar

citations

Cite Score

84

AI summary

This paper introduces ALBERT, a lite BERT architecture, using factorized embedding parameterization and cross-layer parameter sharing to reduce memory consumption and improve training speed, achieving state-of-the-art results on GLUE, RACE, and SQUAD benchmarks with fewer parameters than BERT-large.

Main Contributions

Introduces factorized embedding parameterization to separate the hidden layer size from vocabulary embedding size.
Presents cross-layer parameter sharing to prevent parameter growth with network depth.
Achieves significant parameter reduction (18x fewer parameters than BERT-large).
Introduces a self-supervised loss for sentence-order prediction (SOP) that focuses on inter-sentence coherence.
Establishes new state-of-the-art results on GLUE, SQUAD, and RACE benchmarks.

Abstract

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT (Devlin et al., 2019). Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and SQUAD benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at https://github.com/google-research/ALBERT.

Citation Graph

Loading graph...

References [62]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[3]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[4]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[5]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[6]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[7]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[8]Improving Language Understanding by Generative Pre-Training

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

23 papers in library cite

Very simple and very nice! Easy to understand and revolutionary maybe?

[9]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[10]XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Z. Dai, Yining Yang, J. Carbonell, Ruslan Salakhutdinov, Quoc V. Le - 2019

11 papers in library cite

The method is nice and the results are very good but this paper is just soooo hard to follow...

[11]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[12]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[13]Gaussian Error Linear Units (Gelus)

Dan Hendrycks, Kevin Gimpel - 2016

5 papers in library cite

Very understandable, and very nice! I don't think the justification is good, but hey, it works!

[14]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[15]Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder - 2018

14 papers in library cite

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

[16]A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman - 2018

19 papers in library cite

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

[17]Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Z. Dai, Zhilin Yang, Yining Yang, W. Cohen, J. Carbonell, Quoc Le, Ruslan Salakhutdinov - 2019

9 papers in library cite

It's so cool to see context expansion without the need to actually expand context! Such a simple context and so effective!

[18]SentencePiece: A Simple and Language Independent subword Tokenizer and Detokenizer for Neural Text Processing

John Richardson - 2018

3 papers in library cite

I like that this is just a simple library that wraps functionality. It's also minimal, so can be included in a bunch of places.

[19]Know What You Don't Know: Un-Answerable Questions for SQuAD

P. Rajpurkar, R. Jia, Percy Liang - 2018

14 papers in library cite

It's alright... It's an extension to the other paper/dataset. I feel that it didn't need to be a full paper (maybe a 6-pager).

[20]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[21]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[22]The PASCAL Recognising Textual Entailment Challenge

Ido Dagan, O. Glickman, Bernardo Magnini - 2005

19 papers in library cite

It's very nice how they had the foresight to create a challenge that became relevant like 10 years later.

[23]Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

M. Shoeybi, M. Patwary, Raul Puri, P. Legresley, J. Casper, Bryan Catanzaro - 2019

3 papers in library cite

Simply amazing! very well explained and easy to follow + nice results, and shows that scaling is good for transformers

[24]SpanBERT: Improving Pre-Training by Representing and Predicting Spans

M. Joshi, Deli Chen, Yibo Liu, D. Weld, Luke Zettlemoyer, Omer Levy - 2019

5 papers in library cite

Yet another BERT, and it's not surprising since I've seen this in other papers. But is good nonetheless.

[25]Automatically Constructing a Corpus of Sentential Paraphrases

W. Dolan, Chris Brockett - 2005

9 papers in library cite

Small dataset, questionable methodology, not useful for training models

[26]The Winograd Schema Challenge

Hector J. Levesque, E. Davis, Leora Morgenstern - 2011

13 papers in library cite

It's amazing to see a paper that is very easy to read, opinionated, but also introducing an important contribution to the field. I love that they introduced this challenge as an alternative to the Imitation Game.

[27]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[28]RACE: Large-Scale Reading Comprehension Dataset From Examinations

Guokun Lai, Q. Xie, Haozhe Liu, Yining Yang, Eduard Hovy - 2017

11 papers in library cite

I really like the idea of using human tests for testing AI. Also, very nice insige for using chinese tests!

[29]Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher - 2017

14 papers in library cite

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

[30]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[31]Adaptive Input Representations for Neural Language Modeling

A. Baevski, Michael Auli - 2018

3 papers in library cite

I like the idea and it's a nice paper. Maybe it is a bit outdated now that everything is wordpiece, but I like that they reduce the number of parameters.

[32]Efficient Softmax Approximation for GPUs

E. Grave, Armand Joulin, M. Cisse, D. Grangier, Hervé Jégou - 2017

4 papers in library cite

I loved the methodology and the idea behind it. It's good to see some practical improvements (rather than methodological). I just think it's a bit tough to read. I had to ask AI to help explain some stuff.

[33]Reducing BERT Pre-Training Time From 3 Days to 76 Minutes

Y. You, Jeffrey Li, J. Hseu, X. Song, J. Demmel, Cho Jui Hsieh - 2019

2 papers in library cite

The changes are very simple and very derivative, but they make it so complicated! Also, they don't discuss drawbacks, which I am sure there are many!

[34]Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Wentao Li, P. J. Liu - 2019

17 papers in library cite

44 pages; T5 paper - "unifying framework where all text-based NLP problems are cast as text-to-text tasks"

[35]Inception-V4, Inception-Resnet and the Impact of Residual Connections on Learning

Christian Szegedy, S. Ioffe, Vincent Vanhoucke, A. A. Alemi - 2017

3 papers in library cite

[36]Generating Long Sequences With Sparse Transformers

Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever - 2019

5 papers in library cite

[37]Neural Network Acceptability Judgments

Alex Warstadt, A. Singh, S. Bowman - 2018

8 papers in library cite

CoLA dataset

[38]Training Deep Nets With Sublinear Memory Cost

T. Chen, B. Xu, Chiyuan Zhang, C. Guestrin - 2016

2 papers in library cite

[39]Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation

I. Turc, M. Chang, K. Lee, Kristina Toutanova - 2019

2 papers in library cite

[40]Mesh-Tensorflow: Deep Learning for Supercomputers

Noam Shazeer, Y. Cheng, Niki Parmar, D. Tran, Ashish Vaswani, P. Koanantakool, P. Hawkins, Honglak Lee, M. Hong, C. Young - 2018

4 papers in library cite

[41]Understanding the Disharmony Between Dropout and Batch Normalization by Variance Shift

Xiang Lisa Li, S. Chen, X. Hu, Jihan Yang - 2019

1 paper in library cites

Explain why they dont work well together

[42]Universal Transformers

Mostafa Dehghani, S. Gouws, Oriol Vinyals, Jakob Uszkoreit, Lukasz Kaiser - 2018

6 papers in library cite

[43]The Fifth PASCAL Recognizing Textual Entailment Challenge

L. Bentivogli, Peter Clark, Ido Dagan, D. Giampiccolo - 2009

7 papers in library cite

[44]The Third PASCAL Recognizing Textual Entailment Challenge

D. Giampiccolo, Bernardo Magnini, Ido Dagan, B. Dolan - 2007

7 papers in library cite

[45]SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

D. Cer, M. Diab, E. Agirre, I. L. Gazpio, L. Specia - 2017

6 papers in library cite

[46]The Second PASCAL Recognising Textual Entailment Challenge

R. B. Haim, Ido Dagan, B. Dolan, L. Ferro, D. Giampiccolo, Bernardo Magnini, I. Szpektor - 2006

6 papers in library cite

[47]Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

Yacine Jernite, S. Bowman, D. Sontag - 2017

4 papers in library cite

[48]DisSent: Learning Sentence Representations From Explicit Discourse Relations

Allen Nie, E. Bennett, N. Goodman - 2017

4 papers in library cite

[49]BAM! Born-Again Multi-Task Networks for Natural Language Understanding

K. Clark, M. Luong, U. Khandelwal, C. Manning, Quoc Le - 2019

3 papers in library cite

[50]DCMN+: Dual Co-Matching Network for Multi-Choice Reading Comprehension

S. Zhang, H. Zhao, Yonghui Wu, Zhengyou Zhang, Xinyu Zhou, Xinyu Zhou - 2019

2 papers in library cite

[51]Deep Equilibrium Models

Shuai Bai, J. Zico Kolter, V. Koltun - 2019

2 papers in library cite

[52]First quora dataset release: Question Pairs

S. Iyer, N. Dandekar, K. Csernai - 2017

2 papers in library cite

[53]Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling

T. Shen, T. Zhou, G. Long, J. J. Jiang, Chiyuan Zhang - 2018

1 paper in library cites

[54]Centering: A Framework for Modeling the Local Coherence of Discourse

B. Grosz, A. Joshi, S. Weinstein - 1995

1 paper in library cites

[55]Coherence and Coreference

J. Hobbs - 1979

1 paper in library cites

[56]Cohesion in English

M. Halliday, R. Hasan - 1976

1 paper in library cites

[57]Efficient Training of BERT by Progressively Stacking

L. Gong, D. He, Zhiyuan Li, T. Qin, Lisa Wang, T. Liu - 2019

1 paper in library cites

[58]Learning Generic Sentence Representations Using Convolutional Neural Networks

Z. Gan, Y. Pu, R. Henao, Chun-Liang Li, X. He, L. Carin - 2017

1 paper in library cites

[59]Modeling Recurrence for Transformer

J. Hao, Xinpeng Wang, B. Yang, Lisa Wang, J. Zhang, Zhuowen Tu - 2019

1 paper in library cites

[60]Patient Knowledge Distillation for BERT model Compression

S. Sun, Y. Cheng, Z. Gan, Joseph Liu - 2019

1 paper in library cites

[61]StructBERT: Incorporating Language Structures Into Pre-Training for Deep Language Understanding

Wenyi Wang, B. Bi, Minghao Yan, Chiyu Wu, Z. Bao, L. Peng, L. Si - 2019

1 paper in library cites

[62]The Reversible Residual Network: Backpropagation Without Storing Activations

A. Gomez, M. Ren, R. Urtasun, R. Grosse - 2017

1 paper in library cites

Cited by

8

papers in your library

Cites

42

papers in your library

Read

on November 19, 2025

I like how they can achieve very close results with very few params! Very nice tricks to do that as well.

Tags

Paper Aliases

No aliases