Papperoni

2025

Pre-Training Under Infinite compute

Konwoo Kim, Suhas Kotha, Percy Liang, Tatsunori Hashimoto

citations

Cite Score

AI summary

This paper investigates pre-training language models under data constraints and infinite compute, proposing regularization and ensembling techniques to achieve significantly lower loss asymptotes and improved data efficiency, with distillation enabling smaller models to retain most benefits and achieve better performance on downstream benchmarks.

Main Contributions

Demonstrates that existing data-constrained approaches (increasing epoch count and parameter count) eventually overfit.
Introduces proper regularization, finding an optimal weight decay 30x larger than standard practice, leading to monotonically decreasing loss following a power law.
Shows that ensembling independently trained models achieves a significantly lower loss asymptote than regularized recipes.
Develops an intervention combining epoching, regularization, parameter scaling, and ensemble scaling, achieving a 5.17x data efficiency improvement at 200M tokens.
Achieves a 9% improvement for pre-training evaluations and a 17.5x data efficiency improvement on continued pre-training for math tasks.

Abstract

Since compute grows much faster than web text available for language model pre-training, we ask how one should approach pre-training under fixed data and no compute constraints. We first show that existing data-constrained approaches of increasing epoch count and parameter count eventually overfit, and we significantly improve upon such recipes by properly tuning regularization, finding that the optimal weight decay is 30x larger than standard practice. Since our regularized recipe monotonically decreases loss following a simple power law in parameter count, we estimate its best possible performance via the asymptote of its scaling law rather than the performance at a fixed compute budget. We then identify that ensembling independently trained models achieves a significantly lower loss asymptote than the regularized recipe. Our best intervention combining epoching, regularization, parameter scaling, and ensemble scaling achieves an asymptote at 200M tokens using 5.17x less data than our baseline, and our data scaling laws predict that this improvement persists at higher token budgets. We find that our data efficiency gains can be realized at much smaller parameter counts as we can distill an ensemble into a student model that is 8x smaller and retains 83% of the ensembling benefit. Finally, our interventions designed for validation loss generalize to downstream benchmarks, achieving a 9% improvement for pre-training evals and a 17.5x data efficiency improvement over continued pre-training on math mid-training data. Our results show that simple algorithmic improvements can enable significantly more data-efficient pre-training in a compute-rich future.

Citation Graph

Loading graph...

References [107]

Sort:

Filter:

[1]ImageNet: A Large-Scale Hierarchical Image Database

J. Deng, W. Dong, Richard Socher, L. J. Li, K. Li, Li Fei Fei - 2009

28 papers in library cite