AI summary

This paper introduces a novel approach to train probabilistic neural networks for statistical language modeling, utilizing importance sampling to address the curse of dimensionality. The method significantly speeds up training by reducing the number of network passes, achieving comparable performance to full-gradient models on the Brown corpus.

Abstract

Our previous work on statistical language modeling introduced the use of probabilistic feedforward neural networks to help dealing with the curse of dimensionality. Training this model by maximum likelihood however requires for each example to perform as many network passes as there are words in the vocabulary. Inspired by the contrastive divergence model, we propose and evaluate sampling-based methods which require network passes only for the observed "positive example" and a few sampled negative example words. A very significant speed-up is obtained with an adaptive importance sampling.