AI summary

This paper presents a structured language model (SLM) with a connectionist model component that uses neural networks and distributed word representations to improve language modeling, achieving significant perplexity reduction and moderate word error rate reduction on the UPENN treebank and WSJ corpora.

Abstract

We investigate the performance of the Structured Language Model when one of its components is modeled by a connectionist model. Using a connectionist model and a distributed representation of the items in the history makes the component able to use much longer contexts than possible with currently used interpolated or back- off models, both because of the inherent capability of the connec- tionist model to fight the data sparseness problem, and because of the only sub-linear growth in the model size when increasing the context length. Experiments show significant improvement in perplexity and moderate reduction in word error rate over the base- line SLM results on the UPENN treebank and Wall Street Journal (WSJ) corpora respectively. The results also show that the proba- bility distribution obtained by our model is much less correlated to regular N-grams than the baseline SLM model.