Papperoni

2016

Quantized Neural Networks: Training Neural Networks With Low Precision Weights and Activations

Yoshua Bengio

Open PDF Google Scholar

citations

Cite Score

60

AI summary

This paper introduces Quantized Neural Networks (QNNs), training neural networks with low precision weights/activations, evaluated on MNIST, CIFAR-10, SVHN, and ImageNet, achieving comparable accuracy to 32-bit counterparts, also demonstrating a binary matrix multiplication GPU kernel for faster QNN execution.

Main Contributions

Introduces a method to train Quantized-Neural-Networks (QNNs) with low precision weights and activations at run-time and during gradient computation.
Demonstrates the possibility of training BNNs on MNIST, CIFAR-10, and SVHN, achieving near state-of-the-art results.
Presents results on the ImageNet dataset using binary weights/activations and quantized versions.
Shows that quantized gradients can be used with only 6-bits with small accuracy degradation.
Shows that with 4-bit weights and activations Recurrent QNNs achieve similar accuracies as their 32-bit floating point counterparts on the Penn Treebank dataset.

Abstract

We introduce a method to train Quantized Neural Networks (QNNs) neural networks with extremely low precision (e.g., 1-bit) weights and activations, at run-time. At train- time the quantized weights and activations are used for computing the parameter gradients. During the forward pass, QNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations. As a result, power consumption is expected to be drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to their 32-bit counterparts. For example, our quantized version of AlexNet with 1-bit weights and 2-bit activations achieves 51% top-1 accuracy. Moreover, we quantize the parameter gradients to 6-bits as well which enables gradients computation using only bit-wise opera- tion. Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits. Last but not least, we programmed a binary matrix multiplication GPU kernel with which it is possible to run our MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The QNN code is available online.

Citation Graph

Loading graph...

References [71]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[3]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[4]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[5]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[6]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[7]Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

S. Ioffe, Christian Szegedy - 2015

18 papers in library cite

Very good paper! Similar feel as ResNets: simple idea, elegant. Not too mathy

[8]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[9]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[10]Human-Level Control Through Deep Reinforcement Learning

V. Mnih - 2015

9 papers in library cite

Maybe the first instance of RL with NNs! I am impressed that it seems quite straightforward to implement (maybe tricky to get working).

[11]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[12]Understanding the Difficulty of Training Deep Feedforward Neural Networks

Yoshua Bengio - 2010

20 papers in library cite

Nice but underwhelming results (they still underperform vs. pretraining). I also didn't really like the way it's written. It's not bad, it's just a bit clunky. Worth the read though.

[13]Mastering the Game of Go With Deep Neural Networks and Tree Search

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. V. D. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, Ilya Sutskever, T. Lillicrap, M. Leach, Koray Kavukcuoglu, T. Graepel, Demis Hassabis - 2016

5 papers in library cite

Surprisingly simple. Years ago when I first heard of AlphaGo it seemed such a different/far away concept - "wow, it's AI!". Now that I understand this, it sounds like something I could have worked on (if I had my current knowledge)

[14]Deep Neural Networks for Acoustic Modeling in Speech Recognition

Geoffrey Hinton - 2012

21 papers in library cite

The core of the paper itself is a bit boring and doesn't introduce anything new (just RBMs and DBNs again) but I am giving this a 4 because it's probably the best explanation of RBMs and DBNs I've read so far.

[15]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[16]Regularization of Neural Networks Using Dropconnect

L. Wan, M. Zeiler, S. Zhang, Rob Fergus - 2013

8 papers in library cite

I feel that the method is very complex and does not improve much on top of regular dropout.

[17]Deeply-Supervised Nets

Chen Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu - 2014

8 papers in library cite

Amazing potential, very nice idea, but bad writing.

[18]Maxout Networks

Yoshua Bengio - 2013

17 papers in library cite

A bit hard to understand, but very nice idea.

[19]Deep Convolutional Neural Networks for LVCSR

T. Sainath, Abdel Rahman Mohamed, Brian Kingsbury, Bhuvana Ramabhadran - 2013

2 papers in library cite

Nice to see CNNs for other things other than images, but I am biased against speech recognition and CNNs. Very good that the paper is short though.

[20]Torch7: A Matlab-Like Environment for Machine Learning

Clement Farabet - 2011

5 papers in library cite

Very nice paper! I am impressed that they used Lua, and the API sounds very simple. Plus, it sounds like they didn't even try that hard to get good performance (vs. Theano which was very tryhard)

[21]Theano: New Features and Speed Improvements

F. Bastien, P. Lamblin, Razvan Pascanu, James Bergstra, I. Goodfellow, A. Bergeron, A. Bouchard, N. Nicolas, Yoshua Bengio - 2012

13 papers in library cite

The paper itself is ok, but it seems like they just wanted to make a paper to say "hey, ours is so much better now after these improvements, we crushed Torch7". A bit tryhard tbh.

[22]Improving the Speed of Neural Networks on Cpus

Vincent Vanhoucke, A. Senior, Mark Z. Mao - 2011

4 papers in library cite

It's good and it's interesting, but I don't think it adds a ton. Good read though. I read it because I thought it discussed quantization (which it does, but not in the sense of making NNs smaller)

[23]Context Dependent Recurrent Neural Network Language Model

Tomas Mikolov, Geoffrey Zweig - 2012

12 papers in library cite

Nothing too interesting, just using the context of the RNN.

[24]Fast and Robust Neural Network Joint Models for Statistical Machine Translation

Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, John Makhoul - 2014

9 papers in library cite

Simple and very nice that it showed that NNs can stand on their own for NMT.

[25]Pylearn2: A Machine Learning Research Library

Yoshua Bengio - 2013

5 papers in library cite

The paper itself is ok, but I think it had little impact. The idea of having an extensible and easy to use framework is nice (and is something that I also try to do in my projects), but seems like it didn't catch on and was only used by Bengio's lab.

[26]Theano: A CPU and GPU Math Expression Compiler

James Bergstra, O. Breuleux, F. Bastien, P. Lamblin, Razvan Pascanu, G. Desjardins, J. Turian, D. W. Farley, Yoshua Bengio - 2010

22 papers in library cite

Very nice framework. Symbolic programming is very nice. However, I think that this had very little impact and was mostly used by Bengio's lab.

[27]Deep Compression: Compressing Deep Neural Network With Pruning, Trained Quantization and Huffman Coding

S. Han, H. Mao, W. J. Dally - 2015

3 papers in library cite

Very well cited

[28]Deep Compression: Compressing Deep Neural Network With Pruning, Trained Quantization and Huffman Coding

S. Han, H. Mao, W. J. Dally - 2015

3 papers in library cite

Very well cited

[29]Fitnets: Hints for Thin Deep Nets

A. Romero, Nicolas Ballas, S. E. Kahou, A. Chassang, C. Gatta, Yoshua Bengio - 2015

5 papers in library cite

Fitnets, referenced by resnets + bengio

[30]Deep Learning With Limited Numerical Precision

S. Gupta, A. Agrawal, Karthik Gopalakrishnan, P. Narayanan - 2015

3 papers in library cite

[31]Compressing Neural Networks With the Hashing Trick

Weizhu Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, Yanru Chen - 2015

2 papers in library cite

Network compression?

[32]Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

Yoshua Bengio, N. Leonard, Aaron Courville - 2013

3 papers in library cite

Conditional Computation

[33]Inceptionism: Going Deeper Into Neural Networks

A. Mordvintsev, Christopher Olah, M. Tyka - 2015

2 papers in library cite

It's a blog post, but there are 4 papers there that I would like to read

[34]Practical Variational Inference for Neural Networks

Alex Graves - 2011

8 papers in library cite

[35]Deep Learning Using Linear Support Vector Machines

Y. Tang - 2013

2 papers in library cite

[36]Deep Learning With Cots HPC Systems

A. Coates, B. Huval, Tianle Wang, D. Wu, Bryan Catanzaro, N. Andrew - 2013

2 papers in library cite

[37]Large-Scale FPGA-based Convolutional Networks

Clement Farabet, Yann Lecun, Koray Kavukcuoglu, E. Culurciello, B. Martini, P. Akselrod, S. Talay - 2011

2 papers in library cite

[38]Learning Both Weights and Connections for Efficient Neural Network

S. Han, J. Pool, J. Tran, W. Dally - 2015

2 papers in library cite

[39]NeuFlow: A Runtime Reconfigurable Dataflow Processor for Vision

Clement Farabet, B. Martini, B. Corda, P. Akselrod, E. Culurciello, Yann Lecun - 2011

2 papers in library cite

[40]Neural Networks for Machine Learning

Geoffrey Hinton - 2012

2 papers in library cite

[41]Spatially-Sparse Convolutional Neural Networks

B. Graham - 2014

2 papers in library cite

[42]Xnor-net: Imagenet Classification Using Binary Convolutional Neural Networks

M. Rastegari, V. Ordonez, Joseph Redmon, Ali Farhadi - 2016

2 papers in library cite

[43]Analysis of High-Performance Floating-Point Arithmetic on FPGAs

G. Govindu, L. Zhuo, S. Choi, V. Prasanna - 2004

1 paper in library cites

[44]Asic Implementation of Random Number Generators Using sr Latches and Its Evaluation

N. Torii, H. Kokubo, D. Yamamoto, K. Itoh, M. Takenaka, T. Matsumoto - 2016

1 paper in library cites

[45]Backpropagation for Energy-Efficient Neuromorphic Computing

S. K. Esser, R. Appuswamy, P. Merolla, J. V. Arthur, D. S. Modha - 2015

1 paper in library cites

Missing year

[46]Binarized Neural Networks for Language Modeling

W. Zheng, Y. Tang

1 paper in library cites

[47]Binaryconnect: Training Deep Neural Networks With Binary Weights During Propagations

M. Courbariaux, Yoshua Bengio, J. P. David - 2015

1 paper in library cites

Missing year

[48]Bitwise Neural Networks

M. Kim, P. Smaragdis

1 paper in library cites

[49]Compressing Deep Convolutional Networks Using Vector Quantization

Y. Gong, L. Liu, Michael Yang, L. Bourdev - 2014

1 paper in library cites

[50]Computing’s Energy Problem (And What We Can Do About It)

M. Horowitz - 2014

1 paper in library cites

[51]Convolutional Neural Networks Using Logarithmic Data Representation

D. Miyashita, E. H. Lee, B. Murmann - 2016

1 paper in library cites

[52]Dadiannao: A Machine-Learning Supercomputer

Yanru Chen, T. Luo, Shuming Liu, S. Zhang, Luheng He, J. Wang, Lei Li, T. Chen, Zhiwei Xu, N. Sun - 2014

1 paper in library cites

[53]Deep Neural Networks Are Robust to Weight Binarization and Other Non-Linear Distortions

P. Merolla, R. Appuswamy, J. Arthur, S. K. Esser, D. Modha - 2016

1 paper in library cites

[54]Diannao: A Small-Footprint High-Throughput Accelerator for Ubiquitous Machine-Learning

T. Chen, Z. Du, N. Sun, J. Wang, Chiyu Wu, Yanru Chen, O. Temam - 2014

1 paper in library cites

[55]Dorefa-Net: Training Low Bitwidth convolutional Neural Networks With Low Bitwidth Gradients

Shuyan Zhou, Z. Ni, Xinyu Zhou, H. Wen, Yonghui Wu, Y. Zou - 2016

1 paper in library cites

[56]Efficient and Accurate Approximations of Nonlinear Convolutional Networks

X. Zhang, James Zou, X. Ming, K. He, Jian Sun - 2015

1 paper in library cites

[57]Expectation backpropagation: Parameter-Free Training of Multilayer Neural networks With Continuous or Discrete Weights

D. Soudry, I. Hubara, R. Meir - 2014

1 paper in library cites

[58]Fast Inverse Square Root

C. Lomont - 2003

1 paper in library cites

[59]Fixed-Point Feedforward Deep Neural Network Design Using Weights+ 1, 0, and- 1

K. Hwang, W. Sung - 2014

1 paper in library cites

[60]Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree

Chen Yu Lee, P. W. Gallagher, Zhuowen Tu - 2015

1 paper in library cites

[61]Hardware-Oriented Approximation of Convolutional Neural Networks

P. Gysel, M. Motamedi, S. Ghiasi - 2016

1 paper in library cites

[62]Lasagne: First Release

S. Dieleman, J. Schlter, Colin Raffel, E. Olson, S. K. Snderby, D. Nouri, D. Maturana, M. Thoma, E. Battenberg, J. Kelly, J. D. Fauw, M. Heilman, Diogo149, B. Mcfee, H. Weideman, Takacsg84, Peterderivaz, Jon, Instagibbs, D. K. Rasul, Congliu, Britefury, J. Degrave - 2015

1 paper in library cites

[63]Neuflow: Dataflow Vision Processing System-on-a-Chip

P. H. Pham, D. Jelaca, Clement Farabet, B. Martini, Yann Lecun, E. Culurciello - 2012

1 paper in library cites

[64]Neural Net Works With Few Multiplications

Zongyu Lin, M. Courbariaux, R. Memisevic, Yoshua Bengio - 2015

1 paper in library cites

[65]Neural Net Works With Few Multiplications

Zongyu Lin, M. Courbariaux, R. Memisevic, Yoshua Bengio - 2015

1 paper in library cites

[66]Recurrent Neural Networks With Limited Numerical Precision

J. Ott, Zongyu Lin, Y. Z. Zhang, S. C. Liu, Yoshua Bengio - 2016

1 paper in library cites

[67]Reduction of Quantizing Noise by Use of Feedback

H. Spang, P. Schultheiss - 1962

1 paper in library cites

[68]Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks With Discrete Synapses

C. Baldassi, A. Ingrosso, C. Lucibello, L. Saglietti, R. Zecchina - 2015

1 paper in library cites

[69]Training Binary Multilayer neural Networks for Image Classification Using Expectation Backpropgation

Zhoujun Cheng, D. Soudry, Z. Mao, Z. Lan - 2015

1 paper in library cites

[70]Training Deep Neural Networks With Low Precision Multiplications

M. Courbariaux, Yoshua Bengio, J. P. David - 2014

1 paper in library cites

[71]Yodann: An Ultra-Low Power Convolutional Neural Network Accelerator Based on Binary Weights

R. Andri, L. Cavigelli, D. Rossi, L. Benini - 2016

1 paper in library cites

Cited by

0

papers in your library

Cites

33

papers in your library

Read

on November 20, 2025

Wow, I expected so much more from this. It is a good paper in the sense that the method makes sense, but at the same time it is so hard to read and full of jargon - and it is not even short!!!

Tags

Paper Aliases

No aliases