AI summary

The paper introduces Meteor Universal, a new MT evaluation metric. It leverages bitext to automatically extract linguistic resources like paraphrase tables and function word lists. A universal parameter set, trained on multiple languages, is used. Experiments on Russian and Hindi show significant improvements over BLEU.

Abstract

This paper describes Meteor Universal, released for the 2014 ACL Workshop on Statistical Machine Translation. Meteor Universal brings language specific evaluation to previously unsupported target languages by (1) automatically extracting linguistic resources (paraphrase tables and function word lists) from the bitext used to train MT systems and (2) using a universal parameter set learned from pooling human judgments of translation quality from several language directions. Meteor Universal is shown to significantly outperform baseline BLEU on two new languages, Russian (WMT13) and Hindi (WMT14).