Papperoni

2017

SGDR: Stochastic Gradient Descent With Warm Restarts

Frank Hutter

Open PDF Google Scholar

citations

Cite Score

86

AI summary

This paper introduces a simple warm restart technique, SGDR, for stochastic gradient descent to improve its anytime performance when training deep neural networks, achieving state-of-the-art results on the CIFAR-10 and CIFAR-100 datasets.

Main Contributions

Introduces a simple warm restart technique for stochastic gradient descent (SGDR).
Empirically demonstrates SGDR's performance on CIFAR-10 and CIFAR-100 datasets.
Achieves new state-of-the-art results on CIFAR-10 (3.14%) and CIFAR-100 (16.21%) using SGDR.
Demonstrates the advantages of SGDR on a dataset of EEG recordings and a downsampled version of the ImageNet dataset.
Shows that SGDR can achieve good test error faster than with the default schedule.

Abstract

Restart techniques are common in gradient-free optimization to deal with multi-modal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR

Citation Graph

Loading graph...

References [38]

Sort:

Filter:

[1]Deep Residual Learning for Image Recognition

K. He, X. Zhang, S. Ren, Jian Sun - 2016

20 papers in library cite

This is simply amazing. Very very simple idea, totally revolutionary. No maths, just "it works!". Amazing.

[2]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[3]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[4]Densely Connected Convolutional Networks

G. Huang, Ze Liu, K. Weinberger, Laurens Van Der Maaten - 2017

5 papers in library cite

I liked this paper so much! The way that it's written makes it very easy to follow. Results are nice, explanations are intuitive. Very nice!

[5]Learning Multiple Layers of Features From Tiny Images

Alex Krizhevsky - 2009

27 papers in library cite

It's alright. It mainly focuses on RBMs and their features and the actual part that describes the dataset is like 1 page. However, it's maybe the best intuitive description of an RBM I have seen. Other than that, it reads very much like an undergraduate thesis.

[6]Identity Mappings in Deep Residual Networks

K. He, X. Zhang, S. Ren, Jian Sun - 2016

4 papers in library cite

This paper is amazing - simple, easy to follow, intuitive, while also being impactful! ResNets are already a big leap forward, and they can improve on that

[7]SGDR: Stochastic Gradient Descent With Warm Restarts

Frank Hutter - 2017

4 papers in library cite

Very simple, intuitive, and effective. I just hoped they provided a better motivation for the restarts rather than "it exists in other literature"

[8]Adadelta: An Adaptive Learning Rate Method

Matthew D. Zeiler - 2012

13 papers in library cite

Cool contribution. Simple and straightforward, and not too mathy. Based on ADAGRAD.

[9]DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

J. Donahue, Y. Jia, Oriol Vinyals, J. Hoffman, N. Zhang, E. Tzeng, Trevor Darrell - 2014

15 papers in library cite

Very nice paper. First I've seen (and based on the text, first ever) about feature extraction for images. It's very nice to see embeddings doing SotA

[10]Deep Compression: Compressing Deep Neural Network With Pruning, Trained Quantization and Huffman Coding

S. Han, H. Mao, W. J. Dally - 2015

3 papers in library cite

Very well cited

[11]Wide Residual Networks

S. Zagoruyko, N. Komodakis - 2016

5 papers in library cite

Turn networks wide vs. deep

[12]Cyclical Learning Rates for Training Neural Networks

L. S. Smith - 2016

3 papers in library cite

Triangular learning rates

[13]Deep Networks With Stochastic Depth

G. Huang, Y. S. Sun, Ze Liu, D. Sedra, K. Q. Weinberger - 2016

3 papers in library cite

'Stochastic Depth' - randomly dropping some layers

[14]Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization

Yann N. Dauphin, Razvan Pascanu, C. G. Gulcehre, Kyunghyun Cho, Surya Ganguli, Yoshua Bengio - 2014

4 papers in library cite

[15]The Loss Surfaces of Multilayer Networks

A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, Yann Lecun - 2015

4 papers in library cite

[16]A Method of Solving a Convex Programming Problem With Convergence Rate O (1/K2)

Y. Nesterov - 1983

3 papers in library cite

[17]Deep Pyramidal Residual Networks

D. Han, Jeremy Kim, Jeremy Kim - 2017

3 papers in library cite

[18]Snapshot Ensembles: Train 1, Get M for Free

G. Huang, Yiwei Li, G. Pleiss, Ze Liu, J. Hopcroft, K. Weinberger - 2016

3 papers in library cite

[19]Introductory Lectures on Convex Optimization: A Basic Course

Y. Nesterov - 2013

2 papers in library cite

[20]On the Limited Memory BFGS Method for Large Scale Optimization

D. C. Liu, J. Nocedal - 1989

2 papers in library cite

[21]Rmsprop and Equilibrated Adaptive Learning Rates for Non-Convex Optimization

Yann N. Dauphin, H. D. Vries, J. Chung, Yoshua Bengio - 2015

2 papers in library cite

[22]SGD-Qn: Careful Quasi-Newton Stochastic Gradient Descent

Antoine Bordes, Leon Bottou, P. Gallinari - 2009

2 papers in library cite

[23]Adaptive Restart for Accelerated Gradient Schemes

B. O'donoghue, Emmanuel Candès - 2012

1 paper in library cites

[24]Alternative Restart Strategies for CMA-ES

I. Loshchilov, M. Schoenauer, M. Sebag - 2012

1 paper in library cites

[25]Benchmarking a BI-population CMA-ES on the BBOB-2009 Function Testbed

N. Hansen - 2009

1 paper in library cites

[26]Benchmarking the bfgs algorithm on the Bbob-2009 Function Testbed

R. Ros - 2009

1 paper in library cites

[27]Deep Learning With Convolutional Neural Networks for Brain Mapping and Decoding of Movement-Related Information From the Human Eeg

R. Schirrmeister, J. Springenberg, L. Fiederer, M. Glasstetter, K. Eggensperger, M. Tangermann, Frank Hutter, W. Burgard, T. Ball - 2017

1 paper in library cites

[28]Evaluating the Cma Evolution Strategy on Multimodal Test Functions

N. Hansen, S. Kern - 2004

1 paper in library cites

[29]Function Minimization by Conjugate Gradients

R. Fletcher, C. Reeves - 1964

1 paper in library cites

[30]Local minima and Plateaus in Hierarchical Structures of Multilayer Perceptrons

K. Fukumizu, S. Amari - 2000

1 paper in library cites

[31]New Types of Deep Neural Network Learning for Speech Recognition and Related Applications: An Overview

L. Deng, Geoffrey Hinton, Brian Kingsbury - 2013

1 paper in library cites

[32]Niching Methods and Multimodal Optimization Performance

M. Preuss - 2015

1 paper in library cites

[33]Niching the CMA-ES via Nearest-Better Clustering

M. Preuss - 2010

1 paper in library cites

[34]No More Pesky Learning Rate Guessing Games

L. S. Smith - 2015

1 paper in library cites

[35]Residual Networks of Residual Networks: Multilevel Residual Networks

K. Zhang, Maosong Sun, T. Han, X. Yuan, L. Guo, T. Liu - 2016

1 paper in library cites

[36]Restart Procedures for the Conjugate Gradient Method

M. Powell - 1977

1 paper in library cites

[37]Stochastic Subgradient Methods With Linear Convergence for Polyhedral Convex Optimization

T. Yang, Q. Lin - 2015

1 paper in library cites

[38]Tiny imagenet Visual Recognition Challenge

H. Pouransari, S. Ghili - 2015

1 paper in library cites

Cited by

4

papers in your library

Cites

13

papers in your library

Read

on November 28, 2025

Very simple, intuitive, and effective. I just hoped they provided a better motivation for the restarts rather than "it exists in other literature"

Tags

Paper Aliases

No aliases