Papperoni

2016

Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch

Open PDF Google Scholar

citations

Cite Score

84

AI summary

This paper introduces a method for open-vocabulary neural machine translation by encoding rare words as sequences of subword units using byte pair encoding, achieving improvements of up to 1.3 BLEU on the WMT 15 translation tasks English-German and English-Russian.

Main Contributions

The paper introduces a method for open-vocabulary neural machine translation by encoding rare words as sequences of subword units.
The paper adapts byte pair encoding (BPE), a compression algorithm, to the task of word segmentation, allowing for the representation of an open vocabulary through a fixed-size vocabulary of variable-length character sequences.
The paper shows that open-vocabulary neural machine translation is possible by encoding (rare) words via subword units.
The paper finds that the proposed architecture is simpler and more effective than using large vocabularies and back-off dictionaries.
The paper achieves improvements of up to 1.3 BLEU on the WMT 15 translation tasks English-German and English-Russian.

Abstract

Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character n-gram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English German and English Russian by up to 1.1 and 1.3 BLEU, respectively.

Citation Graph

Loading graph...

References [36]

Sort:

Filter:

[1]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[2]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[3]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[4]Effective Approaches to Attention-Based Neural Machine Translation

T. Luong, H. Pham, Christopher D. Manning - 2015

15 papers in library cite

Good paper, but very derivative. Attention methods start getting very complicated... I understand why Transformers took over TBH

[5]Adadelta: An Adaptive Learning Rate Method

Matthew D. Zeiler - 2012

13 papers in library cite

Cool contribution. Simple and straightforward, and not too mathy. Based on ADAGRAD.

[6]On the Difficulty of Training Recurrent Neural Networks

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio - 2013

21 papers in library cite

It starts very mathy but in the end there are some very nice contributions! You don't actually need to understand the math to know what's going on in the end.

[7]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[8]On Using Very Large Target Vocabulary for Neural Machine Translation

Yoshua Bengio - 2014

12 papers in library cite

It's nice, but it starts getting a bit into the realm of "yeah, that seems like a minor improvement". It's nice that they use the importance sampling stuff from the previous paper though - I thought it had completely vanished :)

[9]Addressing the Rare Word Problem in Neural Machine Translation

T. Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, Wojciech Zaremba - 2014

14 papers in library cite

The method was very poorly explained. It was also worse than a paper released sooner, and more complicated. Overall not that good.

[10]Subword Language Modeling With Neural Networks

Tomas Mikolov, Ilya Sutskever, A. Deoras, H. S. Le, S. Kombrink, Jan Cernocky - 2012

7 papers in library cite

Doesn't add a ton of contribution other than saying that subword models can perform better than char-level models. Probably the most important thing here is early use of subwords.

[11]Moses: Open Source Toolkit for Statistical Machine Translation

P. Koehn, H. Hoang, Alexandra Birch, Chris Callison Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, E. Herbst - 2007

8 papers in library cite

[12]Character-Aware Neural Language Models

Yoon Kim, Yacine Jernite, D. Sontag, Alexander M. Rush - 2016

7 papers in library cite

[13]Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

W. Ling, C. Dyer, A. W. Black, I. Trancoso, R. Fermandez, S. Amir, L. Marujo, T. Luis - 2015

5 papers in library cite

[14]A Simple, Fast, and Effective Reparameterization of IBM Model 2

C. Dyer, V. Chahuneau, Noah A. Smith - 2013

4 papers in library cite

[15]Better Word Representations With Recursive Neural Networks for Morphology

T. Luong, Richard Socher, Christopher D. Manning - 2013

4 papers in library cite

[16]A New Algorithm for Data Compression

P. Gage - 1994

3 papers in library cite

[17]Modelling Out-of-Vocabulary Words for Robust Speech Recognition

I. Bazzi - 2002

3 papers in library cite

[18]Character-Based Neural Machine Translation

M. R. C. Jussa, J. A. R. Fonollosa - 2016

2 papers in library cite

[19]Compositional Morphology for Word Representations and Language Modelling

J. A. Botha, Phil Blunsom - 2014

2 papers in library cite

[20]Results of the WMT15 Metrics Shared Task

M. Stanojevic, A. Kamran, P. Koehn, O. Bojar - 2015

2 papers in library cite

[21]Variable-Length Word Encodings for Neural Translation Models

R. Chitnis, J. Denero - 2015

2 papers in library cite

[22]A Joint Dependency Model of Morphological and Syntactic Structure for Statistical Machine Translation

R. Sennrich, B. Haddow - 2015

1 paper in library cites

[23]Can We Translate Letters?

D. Vilar, J. T. Peter, Hermann Ney - 2007

1 paper in library cites

[24]Character-Based Pivot Translation for Under-Resourced Languages and Domains

J. Tiedemann - 2012

1 paper in library cites

[25]Character-Based PSMT for Closely Related Languages

J. Tiedemann - 2009

1 paper in library cites

[26]chrF: Character N-Gram F-Score for Automatic MT Evaluation

M. Popovic - 2015

1 paper in library cites

[27]Empirical Methods for Compound Splitting

P. Koehn, K. Knight - 2003

1 paper in library cites

[28]Improving SMT Quality With Morpho-Syntactic Analysis

S. Niessen, Hermann Ney - 2000

1 paper in library cites

[29]Integrating an Unsupervised Transliteration Model Into Statistical Machine Translation

N. Durrani, H. Sajjad, H. Hoang, P. Koehn - 2014

1 paper in library cites

[30]Machine Translation Without Words Through Substring Alignment

Graham Neubig, T. Watanabe, S. Mori, T. Kawahara - 2012

1 paper in library cites

[31]Morphology-Aware Statistical Machine Translation Based on Morphs Induced in an Unsupervised Manner

S. Virpioja, J. J. Vayrynen, M. Creutz, M. Sadeniemi - 2007

1 paper in library cites

[32]The Edinburgh/JHU Phrase-based Machine Translation Systems for WMT 2015

B. Haddow, M. Huck, Alexandra Birch, N. Bogoychev, P. Koehn - 2015

1 paper in library cites

[33]Unsupervised Discovery of Morphemes

M. Creutz, K. Lagus - 2002

1 paper in library cites

[34]Unsupervised Morphology Rivals Supervised Morphology for Arabic MT

D. Stallard, Jacob Devlin, M. Kayser, Y. K. Lee, R. Barzilay - 2012

1 paper in library cites

[35]Unsupervised Multilingual Learning for Morphological Segmentation

B. Snyder, R. Barzilay - 2008

1 paper in library cites

[36]Word hy-phen-a-tion by Com-Put-Er

F. M. Liang - 1983

1 paper in library cites

Cited by

22

papers in your library

Cites

10

papers in your library

Read

on July 3, 2025

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

Tags

Paper Aliases

No aliases