Papperoni

2013

Maxout Networks

Yoshua Bengio

Open PDF Google Scholar

citations

Cite Score

64

AI summary

The paper introduces maxout networks, a novel model that leverages dropout for optimization and model averaging. Using maxout with dropout, the authors achieved state-of-the-art results on MNIST, CIFAR-10, CIFAR-100, and SVHN datasets, demonstrating the model's effectiveness in classification tasks.

Main Contributions

Introduced the maxout unit, a new type of activation function suitable for dropout.
Proved that maxout networks are universal approximators.
Demonstrated that dropout attains good approximation to model averaging in deep models, especially with maxout units.
Showed that maxout improves the bagging-style training phase of dropout.
Achieved state-of-the-art results on MNIST, CIFAR-10, CIFAR-100 and SVHN datasets using maxout and dropout.

Abstract

We consider the problem of designing models to leverage a recently introduced approximate model averaging technique called dropout. We define a simple new model called maxout (so named because its output is the max of a set of inputs, and because it is a natural companion to dropout) designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique. We empirically verify that the model successfully accomplishes both of these tasks. We use maxout and dropout to demonstrate state of the art classification performance on four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN.

Citation Graph

Loading graph...

References [24]

Sort:

Filter:

[1]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[2]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[3]Learning Multiple Layers of Features From Tiny Images

Alex Krizhevsky - 2009

27 papers in library cite

It's alright. It mainly focuses on RBMs and their features and the actual part that describes the dataset is like 1 page. However, it's maybe the best intuitive description of an RBM I have seen. Other than that, it reads very much like an undergraduate thesis.

[4]Deep Sparse Rectifier neural Networks

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

17 papers in library cite

How can you not love Bengio? This paper is everything that Hinton's is not. Really well explained, surprising results, good comparisons... Amazing impact!

[5]Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors

Geoffrey E. Hinton, N. Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2012

25 papers in library cite

Dropout, super impactful. The idea that you are training many estimators at once is also very nice.

[6]Reading Digits in Natural Images With Unsupervised Feature Learning

Y. Netzer, Tianle Wang, A. Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng - 2011

8 papers in library cite

It's a bit meh. It's also 2011 so maybe it was impressive, but I think the main contribution is the dataset.

[7]What Is the Best Multi-Stage Architecture for Object Recognition?

K. Jarrett, Koray Kavukcuoglu, Marc'aurelio Ranzato, Yann Lecun - 2009

20 papers in library cite

So boring and convoluted... They try to make things more abstract by saying "filter banks" and "2 stages", but in the end it's CNNs. It seems like they are trying to tie NNs to the rest of ML, but it doesn't work.

[8]Theano: New Features and Speed Improvements

F. Bastien, P. Lamblin, Razvan Pascanu, James Bergstra, I. Goodfellow, A. Bergeron, A. Bouchard, N. Nicolas, Yoshua Bengio - 2012

13 papers in library cite

The paper itself is ok, but it seems like they just wanted to make a paper to say "hey, ours is so much better now after these improvements, we crushed Torch7". A bit tryhard tbh.

[9]Convolutional neural Networks Applied to House Numbers Digit Classification

P. Sermanet, S. Chintala, Yann Lecun - 2012

6 papers in library cite

This reads like an undergrad's research paper or a recent master's paper. Only applying stuff to a new dataset.

[10]Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition

Dan C. Ciresan, Ueli Meier, Luca M. Gambardella, Jürgen Schmidhuber - 2010

10 papers in library cite

It's short, simple and straight to the point (as the network in the paper). It's refreshing to read something less "academic" and more "in your face"

[11]Theano: A CPU and GPU Math Expression Compiler

James Bergstra, O. Breuleux, F. Bastien, P. Lamblin, Razvan Pascanu, G. Desjardins, J. Turian, D. W. Farley, Yoshua Bengio - 2010

22 papers in library cite

Very nice framework. Symbolic programming is very nice. However, I think that this had very little impact and was mostly used by Bengio's lab.

[12]Bagging Predictors

L. Breiman - 1994

4 papers in library cite

20 pages; Bagging, 40k citations

[13]Deep Boltzmann Machines

Ruslan Salakhutdinov, Geoffrey E. Hinton - 2009

9 papers in library cite

Not sure what's the difference vs. DBNs

[14]Practical Bayesian Optimization of Machine Learning Algorithms

J. Snoek, Hugo Larochelle, R. P. Adams - 2012

9 papers in library cite

[15]Improving Neural Networks With Dropout

N. Srivastava - 2013

6 papers in library cite

[16]Stochastic Pooling for Regularization of Deep Convolutional Neural Networks

Matthew D. Zeiler, Rob Fergus - 2013

5 papers in library cite

[17]Rank, Trace-Norm and Max-Norm

N. Srebro, A. Shraibman - 2005

3 papers in library cite

[18]A Model of Multiplicative Neural Responses in Parietal Cortex

E. Salinas, L. F. Abbott - 1996

2 papers in library cite

[19]Deep Convex Network: A Scalable Architecture for Speech Pattern Classification

L. Deng, D. Yu - 2011

2 papers in library cite

[20]Learnable Pooling Regions for Image Classification

M. Malinowski, M. Fritz - 2013

2 papers in library cite

[21]On the Piecewise Analysis of Networks of Linear Threshold Neurons

R. H. R. Hahnloser - 1998

2 papers in library cite

[22]The Manifold Tangent classifier

S. Rifai, Yann Dauphin, Pascal Vincent, Yoshua Bengio, X. Muller - 2011

2 papers in library cite

[23]General Constructive Representations for Continuous Piecewise-Linear Functions

Shijie Wang - 2004

1 paper in library cites

[24]Joint Training of Deep Boltzmann Machines for Classification

Ian J. Goodfellow, Aaron Courville, Yoshua Bengio - 2013

1 paper in library cites

Cited by

17

papers in your library

Cites

13

papers in your library

Read

on April 29, 2025

A bit hard to understand, but very nice idea.

Tags

Paper Aliases

No aliases