AI summary

This paper explores two types of bias in back-propagation networks to minimize hidden units, applying weight decay methods (hyperbolic, exponential) on parity and speech recognition tasks, and shows that introducing biases decreases the number of hidden units but also the convergence rate.

Abstract

Rumelhart (1987), has proposed a method for choosing minimal or "simple" representations during learning in Back-propagation networks. This approach can be used to (a) dynamically select the number of hidden units, (b) construct a representation that is appropriate for the problem and (c) thus improve the generalization ability of Back-propagation networks. The method Rumelhart suggests involves adding penalty terms to the usual error function. In this paper we introduce Rumelhart's minimal networks idea and compare two possible biases on the weight search space. These biases are compared in both simple counting problems and a speech recognition problem. In general, the constrained search does seem to minimize the number of hidden units required with an expected increase in local minima.