AI summary

This paper introduces a Hessian-free optimization method for training deep auto-encoders. It achieves results superior to existing methods without pre-training on the same tasks, scaling well to large datasets. The method addresses pathological curvature, offering a practical and effective approach to deep learning optimization.

Abstract

We develop a 2nd-order optimization method based on the "Hessian-free" approach, and apply it to training deep auto-encoders. Without using pre-training, we obtain results superior to those reported by Hinton & Salakhutdinov (2006) on the same tasks they considered. Our method is practical, easy to use, scales nicely to very large datasets, and isn't limited in applicability to autoencoders, or any specific model class. We also discuss the issue of "pathological curvature" as a possible explanation for the difficulty of deep-learning and how 2nd-order optimization, and our method in particular, effectively deals with it.