AI summary

This paper introduces the wake-sleep algorithm, an unsupervised learning method for multilayer networks using stochastic neurons; it uses “recognition” and “generative” connections to learn economical representations by minimizing a description length objective, demonstrating capabilities on toy problems and handwritten digit recognition.

Abstract

An unsupervised learning algorithm for a multilayer network of stochastic neurons is described. Bottom-up “recognition” connections convert the input into representations in successive hidden layers and top-down “generative” connections reconstruct the representation in one layer from the representation in the layer above. In the “wake” phase, neurons are driven by recognition connections, and generative connections are adapted to increase the probability that they would reconstruct the correct activity vector in the layer below. In the "sleep" phase, neurons are driven by generative connections and recognition connections are adapted to increase the probability that they would produce the correct activity vector in the layer above. Supervised learning algorithms for multilayer neural networks face two problems: They require a teacher to specify the desired output of the network and they require some method of communicating error information to all of the connections. The wake-sleep algorithm avoids both these problems. When there is no external teaching signal to be matched, some other goal is required to force the hidden units to extract underlying structure. In the wake-sleep algorithm the goal is to learn representations that are economical to describe but allow the input to be reconstructed accurately. We can quantify this goal by imagining a communication game in which each vector of raw sensory inputs is communicated to a receiver by first sending its hidden representation and then sending the difference between the input vector and its top-down reconstruction from the hidden representation. The aim of learning is to minimize the “description length” which is the total number of bits that would be required to communicate the input vectors in this way [1]. No communication actually takes place, but minimizing the description length that would be required forces the network to learn economical representations that capture the underlying regularities in the data [2].