Papperoni

2016

Multilingual Language Processing From Bytes

Amarnag Subramanya

Open PDF Google Scholar

citations

Cite Score

14

AI summary

This paper introduces Byte-to-Span (BTS), an LSTM-based model that reads text as bytes and outputs span annotations, and is able to analyze text in many languages with a single model. The multilingual models are compact, but produce results similar to or better than the state-of-the-art in Part-of-Speech tagging and Named Entity Recognition.

Main Contributions

The paper introduces Byte-to-Span (BTS), an LSTM-based model that reads text as bytes and outputs span annotations.
The model can analyze text in many languages with a single model.
The multilingual models are very compact.
The model produces results similar to or better than the state-of-the-art in Part-of-Speech tagging and Named Entity Recognition.
The model learns language-independent representations at deeper levels.

Abstract

We describe an LSTM-based model which we call Byte-to-Span (BTS) that reads text as bytes and outputs span annotations of the form [start, length, label] where start positions, lengths, and labels are separate entries in our vocabulary. Because we operate directly on unicode bytes rather than language-specific words or characters, we can analyze text in many languages with a single model. Due to the small vocabulary size, these multilingual models are very compact, but produce results similar to or better than the state-of-the-art in Part-of-Speech tagging and Named Entity Recognition that use only the provided training datasets (no external data sources). Our models are learning "from scratch" in that they do not rely on any elements of the standard pipeline in Natural Language Processing (including tokenization), and thus can run in standalone fashion on raw text.

Citation Graph

Loading graph...

References [33]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[3]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[4]Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors

Geoffrey E. Hinton, N. Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2012

25 papers in library cite

Dropout, super impactful. The idea that you are training many estimators at once is also very nice.

[5]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[6]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[7]A Framework for Learning Predictive Structures From Multiple Tasks and Unlabeled Data

Rie Kubota Ando, Tong Zhang - 2005

10 papers in library cite

Very nice and clever way of solving the problem of semi-supervised learning, and makes a lot of sense. I give them more credit for formalizing the concept. The methodology is a bit boring.

[8]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[9]The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training

Pascal Vincent - 2009

5 papers in library cite

Very nice analysis of why supervised pretraining works!

[10]Character-Level Convolutional Networks for Text Classification

X. Zhang, J. Zhao, Yann Lecun - 2015

7 papers in library cite

LeCun + Character CNNs

[11]Towards End-to-End Speech Recognition With Recurrent Neural Networks

Alex Graves, Navdeep Jaitly - 2014

2 papers in library cite

Graves, speech recog

[12]End-to-End Attention-Based Large Vocabulary Speech Recognition

D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, Yoshua Bengio - 2015

2 papers in library cite

Large Vocab speeck recog + Bengio

[13]Listen, Attend and Spell

W. Chan, Navdeep Jaitly, Quoc V. Le, Oriol Vinyals - 2015

4 papers in library cite

Speech recog + attention

[14]Character-Aware Neural Language Models

Yoon Kim, Yacine Jernite, D. Sontag, Alexander M. Rush - 2016

7 papers in library cite

[15]Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

J. Lafferty, Andrew Mccallum, F. C. Pereira - 2001

6 papers in library cite

[16]Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

W. Ling, C. Dyer, A. W. Black, I. Trancoso, R. Fermandez, S. Amir, L. Marujo, T. Luis - 2015

5 papers in library cite

[17]Improved Transition-Based Parsing by Modeling Characters Instead of Words With LSTMs

M. Ballesteros, C. Dyer, Noah A. Smith - 2015

3 papers in library cite

[18]Named Entity Recognition Through classifier Combination

R. Florian, A. Ittycheriah, H. Jing, Tong Zhang - 2003

3 papers in library cite

[19]Named Entity Recognition With Character-Level Models

Dan Klein, J. Smarr, H. Nguyen, Christopher D. Manning - 2003

2 papers in library cite

[20]Variable-Length Word Encodings for Neural Translation Models

R. Chitnis, J. Denero - 2015

2 papers in library cite

[21]A Universal Part-of-Speech Tagset

Slav Petrov, Dipanjan Das, R. Mcdonald - 2011

1 paper in library cites

[22]Bidirectional lstm-crf Models for Sequence Tagging

Zhongqiang Huang, Weixin Xu, K. Yu - 2015

1 paper in library cites

[23]Boosting Named Entity Recognition With Neural Character Embeddings

C. D. Santos, V. Guimaraes, R. Niteroi, R. D. Janeiro - 2015

1 paper in library cites

[24]Char_align: A Program for Aligning Parallel Texts at the Character Level

K. W. Church - 1993

1 paper in library cites

[25]Chinese and Japanese Word Segmentation Using Word-Level and Character-Level Information

T. Nakagawa - 2004

1 paper in library cites

[26]Effective Identification of Source Code Authors Using Byte-Level Information

G. Frantzeskou, E. Stamatatos, S. Gritzalis, S. Katsikas - 2006

1 paper in library cites

[27]From Speech to Letters-Using a Novel Neural Network Architecture for Grapheme Based Asr

F. Eyben, M. Wollmer, B. Schuller, Alex Graves - 2009

1 paper in library cites

[28]Language Independent Authorship Attribution Using Character Level Language Models

F. Peng, Dale Schuurmans, Shijie Wang, V. Keselj - 2003

1 paper in library cites

[29]Learning Multilingual Named Entity Recognition From wikipedia

J. Nothman, N. Ringland, W. Radford, T. Murphy, J. R. Curran - 2013

1 paper in library cites

[30]Lexicon Infused Phrase Embeddings for Named Entity Resolution

A. Passos, V. Kumar, Andrew Mccallum - 2014

1 paper in library cites

[31]Low Resource Dependency Parsing: Cross-Lingual Parameter Sharing in a Neural Network Parser

L. Duong, T. Cohn, S. Bird, P. Cook - 2015

1 paper in library cites

[32]Named Entity Extraction Using Ad-Aboost

X. Carreras, L. Marques, L. Padro - 2002

1 paper in library cites

[33]Part-of-Speech Tagging From 97% to 100%: Is It Time for Some Linguistics?

Christopher D. Manning - 2011

1 paper in library cites

Cited by

2

papers in your library

Cites

13

papers in your library

Read

on October 28, 2025

Nothing impressive... Their main innovation was the byte stuff. Plus the old-school NLP stuff makes it boring.

Tags

Paper Aliases

No aliases