AI summary

The paper introduces a distributed infrastructure to train language models up to 2 trillion tokens using a new smoothing method called Stupid Backoff, achieving improvements in machine translation quality as measured by the BLEU score.

Abstract

This paper reports on the benefits of large-scale statistical language modeling in machine translation. A distributed infrastructure is proposed which we use to train on up to 2 trillion tokens, resulting in language models having up to 300 billion n-grams. It is capable of providing smoothed probabilities for fast, single-pass decoding. We introduce a new smoothing method, dubbed Stupid Backoff, that is inexpensive to train on large data sets and approaches the quality of Kneser-Ney Smoothing as the amount of training data increases.