AI summary

This paper introduces optimized backing-off distributions for M-gram language modeling, derived from theoretical approaches, achieving a 10% improvement in perplexity and a 5% reduction in word error rate on the Verbmobil and Wall-Street-Journal corpora.

Abstract

In stochastic language modeling, backing-off is a widely used method to cope with the sparse data problem. In case of unseen events this method backs off to a less specific distribution. In this paper we propose to use distributions which are especially optimized for the task of backing-off. Two different theoretical derivations lead to distributions which are quite different from the probability distributions that are usually used for backing off. Experiments show an improvement of about 10% in terms of perplexity and 5% in terms of word error rate.