Papperoni

2018

Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder

Open PDF Google Scholar

citations

Cite Score

76

AI summary

This paper introduces ULMFiT, a transfer learning method that leverages discriminative fine-tuning, slanted triangular learning rates, and gradual unfreezing, achieving state-of-the-art results on six text classification tasks and demonstrating sample efficiency with limited labeled data, using a 3-layer LSTM architecture.

Main Contributions

Introduces Universal Language Model Fine-tuning (ULMFiT) for NLP tasks, enabling CV-like transfer learning.
Proposes discriminative fine-tuning, slanted triangular learning rates, and gradual unfreezing to retain knowledge and avoid catastrophic forgetting.
Achieves state-of-the-art results on six text classification datasets, reducing error by 18-24% on most datasets.
Demonstrates sample-efficient transfer learning with extensive ablation analysis.
Releases pretrained models and code for wider adoption.

Abstract

Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100× more data. We open-source our pretrained models and code¹.

Citation Graph

Loading graph...

References [52]

Sort:

Filter:

[1]Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

S. Ioffe, Christian Szegedy - 2015

18 papers in library cite

Very good paper! Similar feel as ResNets: simple idea, elegant. Not too mathy

[2]Fully Convolutional Networks for Semantic Segmentation

J. Long, E. Shelhamer, Trevor Darrell - 2015

7 papers in library cite

I didn't really like the way the paper is written and the results seem a bit underwhelming. However, it's nice that they do it in a fully convolutional way and that they increase the speed.

[3]Densely Connected Convolutional Networks

G. Huang, Ze Liu, K. Weinberger, Laurens Van Der Maaten - 2017

5 papers in library cite

I liked this paper so much! The way that it's written makes it very easy to follow. Results are nice, explanations are intuitive. Very nice!

[4]A Survey on Transfer Learning

Sinno Jialin Pan, Qiang Yang - 2010

1 paper in library cites

I love that it formalized what is transfer learning and the main concepts. I think the citations are because of that as well. The part where they survey the methods is a bit boring and TBH most methods are not relevant. Also, transfer learning occurs quite naturally in neural networks, not so much in traditional methods.

[5]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[6]How Transferable Are Features in Deep Neural Networks?

Hod Lipson - 2014

2 papers in library cite

Such a good paper to analyze pretraining behavior: simple, intuitive, nice to read

[7]SGDR: Stochastic Gradient Descent With Warm Restarts

Frank Hutter - 2017

4 papers in library cite

Very simple, intuitive, and effective. I just hoped they provided a better motivation for the restarts rather than "it exists in other literature"

[8]Learning Word Vectors for Sentiment Analysis

A. L. Maas, R. E. Daly, P. T. Pham, Dong Huang, Andrew Y. Ng, Christopher Potts - 2011

12 papers in library cite

Very convoluted method. I did not like it and I had to go back twice to remember if I had read it or not. Main contribution is the IMDB movie review dataset. (It is cited by Mikolov as a comparison)

[9]DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

J. Donahue, Y. Jia, Oriol Vinyals, J. Hoffman, N. Zhang, E. Tzeng, Trevor Darrell - 2014

15 papers in library cite

Very nice paper. First I've seen (and based on the text, first ever) about feature extraction for images. It's very nice to see embeddings doing SotA

[10]Why Does Unsupervised Pre-Training Help Deep Learning?

Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre Antoine Manzagol, Pascal Vincent, Samy Bengio - 2010

12 papers in library cite

Good paper, easy to follow, and brings some light to the pre-training stuff (layer-by-layer). I just wish it wasn't so long. It's a chore.

[11]Pointer Sentinel Mixture Models

S. Merity, Caiming Xiong, J. Bradbury, Richard Socher - 2017

12 papers in library cite

I really liked the methodology, but I had to read it a few times to understand it intuitively - I think they should have done a better job at explaining it.

[12]Biographies, Bollywood, Boom-Boxes and Blenders: Domain Adaptation for Sentiment Classification

John Blitzer, Mark Dredze, Fernando Pereira - 2007

4 papers in library cite

It's alright. I think they focus a lot on methodology of prediction but most citations are actually because of the dataset.

[13]Supervised Learning of Universal Sentence Representations From Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, L. Barrault, Antoine Bordes - 2017

11 papers in library cite

It's nice. Maybe the first to do NLI right (after the SNLI paper tried but failed miserably). It is simple and effective. After that people started "performance maxxxing"

[14]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[15]Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher - 2017

14 papers in library cite

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

[16]Semi-Supervised Sequence Tagging With Bidirectional Language Models

M. E. Peters, W. Ammar, C. Bhagavatula, Russell Power - 2017

5 papers in library cite

Nothing too different from other LSTM + Attention papers

[17]Learning to Generate Reviews and Discovering Sentiment

Alec Radford, R. Jozefowicz, Ilya Sutskever - 2017

8 papers in library cite

Very impressive results. We can see that they were onto something (later became GPT2)

[18]Character-Level Convolutional Networks for Text Classification

X. Zhang, J. Zhao, Yann Lecun - 2015

7 papers in library cite

LeCun + Character CNNs

[19]Learning Transferable Features With Deep Adaptation Networks

M. Long, Yue Cao, J. Wang, Michael I. Jordan - 2015

1 paper in library cites

Transfer learning

[20]CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

A. Razavian, H. Azizpour, J. Sullivan, S. Carlsson - 2014

6 papers in library cite

CNN feature extraction

[21]Cyclical Learning Rates for Training Neural Networks

L. S. Smith - 2016

3 papers in library cite

Triangular learning rates

[22]Improving Neural Machine Translation Models With Monolingual Data

R. Sennrich, B. Haddow, Alexandra Birch - 2016

4 papers in library cite

The most cited for multilingual/monolingual NMT (out of other references)

[23]Deep Boltzmann Machines

Ruslan Salakhutdinov, Geoffrey E. Hinton - 2009

9 papers in library cite

Not sure what's the difference vs. DBNs

[24]Exploring the Limits of Weakly Supervised Pretraining

D. Mahajan, Ross Girshick, V. Ramanathan, K. He, M. Paluri, Yiwei Li, A. Bharambe, Laurens Van Der Maaten - 2018

2 papers in library cite

CNN + transfer learning

[25]Regularizing and Optimizing 1stm Language Models

S. Merity, Nitish Shirish Keskar, Richard Socher - 2017

6 papers in library cite

SotA LM in 2017

[26]Deep Pyramid Convolutional Neural Networks for Text Categorization

R. Johnson, Tong Zhang - 2017

2 papers in library cite

SotA language categorization

[27]Using Millions of Emoji Occurrences to Learn Any-Domain Representations for Detecting Sentiment, Emotion and Sarcasm

B. Felbo, A. Mislove, A. Sogaard, I. Rahwan, S. Lehmann - 2017

1 paper in library cites

"Chain thaw", technique for fine tuning

[28]What Makes ImageNet Good for Transfer Learning?

M. Huh, P. Agrawal, A. A. Efros - 2016

1 paper in library cites

Nice provocation

[29]How Transferable Are Neural Networks in NLP Applications?

L. Mou, Z. Meng, R. Yan, G. Li, Yiheng Xu, Li Zhang, Z. Jin - 2016

3 papers in library cite

Seems like this said that pre-training didnt work

[30]Assessing the Ability of lstms to Learn Syntax-Sensitive Dependencies

Tal Linzen, E. Dupoux, Y. Goldberg - 2016

5 papers in library cite

[31]The trec-8 Question Answering Track Evaluation

E. M. Voorhees, D. M. Tice - 1999

5 papers in library cite

[32]Adversarial Training Methods for Semi-Supervised Text Classification

T. Miyato, A. M. Dai, I. Goodfellow - 2016

4 papers in library cite

[33]Question Answering Through Transfer Learning From Large Fine-Grained Supervision Data

S. Min, M. J. Seo, Hananneh Hajishirzi - 2017

4 papers in library cite

[34]Colorless Green Recurrent Networks Dream Hierarchically

K. Gulordava, Piotr Bojanowski, E. Grave, Tal Linzen, M. Baroni - 2018

3 papers in library cite

[35]Discriminative Neural Sentence Modeling by Tree-Based Convolution

L. Mou, H. Peng, G. Li, Yiheng Xu, Li Zhang, Z. Jin - 2015

3 papers in library cite

[36]Multitask Learning: A Knowledge-Based Source of Inductive Bias

Rich Caruana - 1993

3 papers in library cite

[37]Semi-Supervised Multitask Learning for Sequence Labeling

M. Rei - 2017

3 papers in library cite

[38]Text Classification Improved by Integrating Bidirectional LSTM With Two-Dimensional Max Pooling

P. Zhou, Z. Qi, S. Zheng, Jiacheng Xu, H. Bao, B. Xu - 2016

3 papers in library cite

[39]A Model of Inductive Bias Learning

J. Baxter - 2000

2 papers in library cite

[40]Deep Biaffine Attention for Neural Dependency Parsing

T. Dozat, Christopher D. Manning - 2017

2 papers in library cite

[41]Estimation of Dependences Based on Empirical Data

V. N. Vapnik - 1983

2 papers in library cite

[42]Hypercolumns for Object Segmentation and Fine-Grained Localization

B. Hariharan, P. Arbelaez, Ross Girshick, Jitendra Malik - 2015

2 papers in library cite

[43]Supervised and Semi-Supervised Text Categorization Using LSTM for Region Embeddings

R. Johnson, Tong Zhang - 2016

2 papers in library cite

[44]An Overview of Gradient Descent Optimization Algorithms

Sebastian Ruder - 2016

1 paper in library cites

[45]Classifying Text Messages for the haiti Earthquake

C. Caragea, N. Mcneese, A. Jaiswal, G. Traylor, H. W. Kim, P. Mitra, D. Wu, A. H. Tapia, L. Giles, B. J. Jansen - 2011

1 paper in library cites

[46]Detecting Automation of Twitter Accounts: Are You a Human, Bot, or cyborg?

Z. Chu, S. Gianvecchio, Haiming Wang, S. Jajodia - 2012

1 paper in library cites

[47]Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review

H. L. Roitblat, A. Kershaw, P. Oot - 2010

1 paper in library cites

[48]Empower sequence Labeling With Task-Aware Neural Language Model

L. Liu, J. Shang, Frank Xu, Xiang Ren, H. Gui, J. Peng, J. Han - 2018

1 paper in library cites

[49]GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

Ziru Chen, V. Badrinarayanan, Chen Yu Lee, Andrew Rabinovich - 2017

1 paper in library cites

[50]Review Spam Detection

N. Jindal, Bing Liu - 2007

1 paper in library cites

[51]Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings

J. Wieting, Kevin Gimpel - 2017

1 paper in library cites

[52]UNITN: Training Deep Convolutional Neural Network for Twitter Sentiment Classification

A. Severyn, A. Moschitti - 2015

1 paper in library cites

Cited by

14

papers in your library

Cites

29

papers in your library

Read

on October 17, 2025

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

Tags

Paper Aliases

No aliases