Papperoni

2020

Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei

Open PDF Google Scholar

citations

Cite Score

81

AI summary

This paper empirically studies scaling laws for language model performance, revealing power-law relationships with model size, dataset size, and compute, and shows that larger models are more sample-efficient, leading to optimal training with very large models on modest data.

Main Contributions

Identified empirical power-law scaling for language model performance with model size, dataset size, and compute, spanning over seven orders of magnitude.
Showed that architectural details (network width/depth) have minimal impact on performance within a wide range.
Derived simple equations governing overfitting dependence on model/dataset size and training speed dependence on model size.
Determined that optimal compute allocation involves training very large models on modest data and stopping significantly before convergence due to larger models being more sample-efficient.
Proposed a predictive framework for language model performance based on these scaling laws.

Abstract

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

Citation Graph

Loading graph...

References [49]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[3]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[4]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[5]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[6]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[7]Improving Language Understanding by Generative Pre-Training

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

23 papers in library cite

Very simple and very nice! Easy to understand and revolutionary maybe?

[8]XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Z. Dai, Yining Yang, J. Carbonell, Ruslan Salakhutdinov, Quoc V. Le - 2019

11 papers in library cite

The method is nice and the results are very good but this paper is just soooo hard to follow...

[9]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[10]ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations

Z. Lan, Mark Chen, S. Goodman, Kevin Gimpel, P. Sharma, Radu Soricut - 2019

8 papers in library cite

I like how they can achieve very close results with very few params! Very nice tricks to do that as well.

[11]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[12]SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A. Wang, Y. Pruksachatkun, Nikita Nangia, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2019

15 papers in library cite

Nothing too surprising, just getting a bunch of stuff from different places and putting it all together. At least they do a good analysis of the benchmark vs. existing methodologies.

[13]Generating Wikipedia by Summarizing Long Sequences

P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, Lukasz Kaiser, Noam Shazeer - 2018

7 papers in library cite

Very nice, the first ones that realized that they could use decoder only from Transformers. Also nice that they expanded the context window, and got nice results. Very inventive to use citations from wikipedia and the first paragraph as target.

[14]Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Wentao Li, P. J. Liu - 2019

17 papers in library cite

44 pages; T5 paper - "unifying framework where all text-based NLP problems are cast as text-to-text tasks"

[15]Wide Residual Networks

S. Zagoruyko, N. Komodakis - 2016

5 papers in library cite

Turn networks wide vs. deep

[16]Reconciling Modern Machine-Learning Practice and the Classical Bias-Variance Trade-Off

M. Belkin, D. Hsu, S. Ma, S. Mandal - 2019

2 papers in library cite

[17]Generating Long Sequences With Sparse Transformers

Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever - 2019

5 papers in library cite

[18]Residual Networks Behave Like Ensembles of Relatively Shallow Networks

A. Veit, M. J. Wilber, S. Belongie - 2016

4 papers in library cite

Resnets are ensembles

[19]Deep Learning Scaling Is Predictable, Empirically

J. Hestness, S. Narang, N. Ardalani, G. Diamos, Heewoo Jun, H. Kianinejad, M. Patwary, M. Ali, Yining Yang, Y. Zhou - 2017

5 papers in library cite

[20]Adafactor: Adaptive Learning Rates With Sublinear Memory Cost

Noam Shazeer, M. Stern - 2018

3 papers in library cite

Used for GPT3 training

[21]Mesh-Tensorflow: Deep Learning for Supercomputers

Noam Shazeer, Y. Cheng, Niki Parmar, D. Tran, Ashish Vaswani, P. Koanantakool, P. Hawkins, Honglak Lee, M. Hong, C. Young - 2018

4 papers in library cite

[22]An Empirical Model of Large-Batch Training

Sam McCandlish, Jared Kaplan, Dario Amodei, O. D. Team - 2018

3 papers in library cite

Analysis of large batch impacts

[23]Universal Transformers

Mostafa Dehghani, S. Gouws, Oriol Vinyals, Jakob Uszkoreit, Lukasz Kaiser - 2018

6 papers in library cite

[24]A Bit of Progress in Language Modeling

J. Goodman - 2001

15 papers in library cite

Focuses on n-grams.

[25]A Constructive Prediction of the Generalization Error Across Scales

J. S. Rosenfeld, A. Rosenfeld, Yonatan Belinkov, N. Shavit - 2019

5 papers in library cite

[26]Beyond Human-Level Accuracy: Computational Challenges in Deep Learning

J. Hestness, N. Ardalani, G. Diamos - 2019

4 papers in library cite

[27]Scaling to Very Very Large Corpora for Natural Language Disambiguation

M. Banko, E. Brill - 2001

3 papers in library cite

[28]Common Crawl

C. Crawl - 2019

2 papers in library cite

[29]Gpipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism

Y. Huang, Y. Cheng, Deli Chen, Honglak Lee, J. Ngiam, Quoc V. Le, Ziru Chen - 2018

2 papers in library cite

[30]GPU kernels for Block-Sparse Weights

Scott Gray, Alec Radford, D. P. Kingma - 2017

2 papers in library cite

[31]Measuring the Effects of Data Parallelism on Neural Network Training

C. J. Shallue, Jaehoon Lee, J. Antognini, Jascha Sohl Dickstein, R. Frostig, George E. Dahl - 2018

2 papers in library cite

[32]All of Nonparametric Statistics

L. Wasserman - 2006

1 paper in library cites

[33]An Investigation Into Neural Net Optimization via Hessian eigenvalue Density

B. Ghorbani, S. Krishnan, Y. Xiao - 2019

1 paper in library cites

[34]Analysis of a Random Forests Model

G. Biau - 2012

1 paper in library cites

[35]Autogrow: Automatic Layer Growing in Deep Convolutional Networks

W. Wen, F. Yan, H. Li - 2019

1 paper in library cites

[36]Criticality in Formal Languages and Statistical Physics

H. W. Lin, M. Tegmark - 2016

1 paper in library cites

[37]Efficientnet: Rethinking Model Scaling for Convolutional Neural Networks

M. Tan, Quoc V. Le - 2019

1 paper in library cites

[38]Entropy and Long-Range Correlations in Literary English

W. Ebeling, T. Poschel - 1994

1 paper in library cites

[39]Gradient Descent Happens in a Tiny Subspace

Guy Gur Ari, D. A. Roberts, Ethan Dyer - 2018

1 paper in library cites

[40]Growing a Brain: Fine-Tuning by Increasing Model Capacity

Y. X. Wang, D. Ramanan, M. Hebert - 2017

1 paper in library cites

[41]High-Dimensional Dynamics of Generalization Error in Neural Networks

M. S. Advani, Andrew M. Saxe - 2017

1 paper in library cites

[42]Introduction to the Theory of Complex Systems

S. Thurner, R. Hanel, P. Klimek - 2018

1 paper in library cites

[43]Neural tangent kernel: Convergence and Generalization in Neural Networks

A. Jacot, F. Gabriel, C. Hongler - 2018

1 paper in library cites

[44]On the Origin of Long-Range Correlations in Texts

E. G. Altmann, G. Cristadoro, M. D. Esposti - 2012

1 paper in library cites

[45]One Epoch Is All You Need

A. Komatsuzaki - 2019

1 paper in library cites

[46]Scaling Description of Generalization With Number of Parameters in Deep Learning

M. Geiger, A. Jacot, S. Spigler, F. Gabriel, L. Sagun, S. D'ascoli, G. Biroli, C. Hongler, M. Wyart - 2019

1 paper in library cites

[47]The Full Spectrum of Deep Net Hessians at Scale: Dynamics With Sample Size

V. Papyan - 2018

1 paper in library cites

[48]Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model

G. Zhang, Lei Li, Z. Nado, James Martens, S. Sachdeva, George E. Dahl, C. J. Shallue, R. B. Grosse - 2019

1 paper in library cites

[49]Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

Jaehoon Lee, L. Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl Dickstein, Jeffrey Pennington - 2019

1 paper in library cites

Cited by

12

papers in your library

Cites

23

papers in your library

Read

on May 26, 2026

Very nice! An amazing contribution. Problem is, the paper is just like 3 pages of actual interesting content, and 10 pages of detailed results. Boring to read but very good otherwise.

Tags

Paper Aliases

No aliases