Papperoni

2019

DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter

Thomas Wolf

Open PDF Google Scholar

citations

Cite Score

85

AI summary

This paper introduces DistilBERT, a smaller, faster, and lighter version of BERT, pre-trained using knowledge distillation. It reduces the size of BERT by 40% while retaining 97% of its language understanding capabilities and being 60% faster, achieving comparable performance on GLUE and downstream tasks.

Main Contributions

Introduces DistilBERT, a distilled version of BERT that is 40% smaller and 60% faster.
Leverages knowledge distillation during the pre-training phase to reduce model size while retaining performance.
Introduces a triple loss combining language modeling, distillation, and cosine-distance losses.
Demonstrates DistilBERT's capabilities for on-device computations with a proof-of-concept experiment.
Achieves 97% of BERT's performance on the GLUE benchmark with 40% fewer parameters.

Abstract

As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.

Citation Graph

Loading graph...

References [22]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[3]Distilling the Knowledge in a Neural Network

Jeffrey Dean - 2015

6 papers in library cite

Very nice, very relevant.

[4]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[5]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[6]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[7]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[8]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[9]Huggingface's Transformers: State-of-the-Art Natural Language Processing

Thomas Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, Sam Shleifer, P. V. Platen, C. Ma, Yacine Jernite, J. Plu, Chenfeng Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, Alexander M. Rush - 2019

7 papers in library cite

It's a solid paper, but these framework papers are usually boring... Yet, it's nice to see that they started with reduced scope (and how large it got now).

[10]Learning Word Vectors for Sentiment Analysis

A. L. Maas, R. E. Daly, P. T. Pham, Dong Huang, Andrew Y. Ng, Christopher Potts - 2011

12 papers in library cite

Very convoluted method. I did not like it and I had to go back twice to remember if I had read it or not. Main contribution is the IMDB movie review dataset. (It is cited by Mikolov as a comparison)

[11]Energy and Policy Considerations for Deep Learning in NLP

E. Strubell, A. Ganesh, Andrew Mccallum - 2019

3 papers in library cite

Very nice push and maybe the first one to actually raise a flag - also, very contemporary and relevant discussion

[12]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[13]Model Compression

C. Bucilua, Rich Caruana, Alexandru Niculescu Mizil - 2006

2 papers in library cite

Very cool idea, and very early as well!

Oren Etzioni - 2019

4 papers in library cite

I like the push a lot! Specially when it comes to make research more democratic, which I personally benefit from. Also, AllenAI seems like they do good, honest work!

[15]Are Sixteen Heads Really Better Than One?

P. Michel, Omer Levy, Graham Neubig - 2019

1 paper in library cites

I expected more. They don't answer the main question of the paper. I thought they would explain what the heads are for, but they only compare results with/without. It's more of an empirical analysis.

[16]Deep Learning With Limited Numerical Precision

S. Gupta, A. Agrawal, Karthik Gopalakrishnan, P. Narayanan - 2015

3 papers in library cite

[17]Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation

I. Turc, M. Chang, K. Lee, Kristina Toutanova - 2019

2 papers in library cite

[18]Jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models

Y. Pruksachatkun, P. Yeres, Haozhe Liu, Jason Phang, Phu Mon Htut, A. Wang, I. Tenney, Samuel R. Bowman - 2020

3 papers in library cite

[19]Distilling Task-Specific Knowledge From BERT into Simple Neural Networks

R. Tang, Y. Lu, L. Liu, L. Mou, O. Vechtomova, Junyang Lin - 2019

1 paper in library cites

[20]Making Neural Machine Reading Comprehension Faster

D. Chatterjee - 2019

1 paper in library cites

[21]Model Compression With Multi-Task Knowledge Distillation for Web-Scale Question Answering System

Zhilin Yang, L. Shou, M. Gong, Wuwei Lin, D. Jiang - 2019

1 paper in library cites

[22]Small and Practical BERT Models for Sequence Labeling

H. Tsai, J. Riesa, M. J. Johnson, N. Arivazhagan, Xiang Lisa Li, A. Archer - 2019

1 paper in library cites

Cited by

6

papers in your library

Cites

17

papers in your library

Read

on November 21, 2025

Surprisingly simple and amazing results!

Tags

Paper Aliases

No aliases