Papperoni

2017

Semi-Supervised Sequence Tagging With Bidirectional Language Models

M. E. Peters, W. Ammar, C. Bhagavatula, Russell Power

Open PDF Google Scholar

citations

Cite Score

35

AI summary

This paper introduces TagLM, a semi-supervised sequence tagging model using bidirectional language models (LMs). It achieves state-of-the-art results on CoNLL 2003 NER and CoNLL 2000 Chunking datasets by incorporating pre-trained context embeddings from LMs into token representations.

Main Contributions

Proposes TagLM, a semi-supervised approach that uses pre-trained bidirectional language models to augment token representations in sequence tagging models.
Achieves state-of-the-art results on the CoNLL 2003 NER task, surpassing previous systems that use other forms of transfer or joint learning.
Establishes a new state-of-the-art result on the CoNLL 2000 Chunking task.
Demonstrates that using both forward and backward LM embeddings boosts performance over using only forward LM embeddings.
Shows that domain-specific pre-training is unnecessary by applying a LM trained in the news domain to scientific papers.

Abstract

Pre-trained word embeddings learned from unlabeled text have become a standard component of neural network architectures for NLP tasks. However, in most cases, the recurrent network that operates on word-level representations to produce context sensitive representations is trained on relatively little labeled data. In this paper, we demonstrate a general semi-supervised approach for adding pre-trained context embeddings from bidirectional language models to NLP systems and apply it to sequence labeling tasks. We evaluate our model on two standard datasets for named entity recognition (NER) and chunking, and in both cases achieve state of the art results, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers.

Citation Graph

Loading graph...

References [45]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[4]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[5]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[6]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[7]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[8]On the Properties of Neural Machine Translation: Encoder-Decoder Approaches

Kyunghyun Cho, B. V. Merrienboer, D. Bahdanau, Yoshua Bengio - 2014

9 papers in library cite

This just builds upon other work, comparing and analyzing previous architectures. This faces a specific problem of the time: long sentences. Not too relevant today.

[9]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[10]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[11]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[12]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[13]Visualizing and Understanding Recurrent Networks

Li Fei Fei - 2015

3 papers in library cite

Very nice! The results were a bit underwhelming though. Nice visualizations but few explanations. Good nonetheless.

[14]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[15]One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

C. Chelba, Tomas Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, Tony Robinson - 2013

13 papers in library cite

It's somewhat shallow, but I can see the importance of this paper.

[16]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[17]context2vec: Learning Generic Context Embedding With Bidirectional LSTM

O. Melamud, J. Goldberger, Ido Dagan - 2016

5 papers in library cite

I liked the methodology a lot but the paper reads like an MSc course project.

[18]Fast and Robust Neural Network Joint Models for Statistical Machine Translation

Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, John Makhoul - 2014

9 papers in library cite

Simple and very nice that it showed that NNs can stand on their own for NMT.

[19]Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling

H. Sak, A. W. Senior, F. Beaufays - 2014

5 papers in library cite

Only 5 pages

[20]Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

J. Lafferty, Andrew Mccallum, F. C. Pereira - 2001

6 papers in library cite

[21]A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

K. Hashimoto, Caiming Xiong, Y. Tsuruoka, Richard Socher - 2016

5 papers in library cite

[22]Assessing the Ability of lstms to Learn Syntax-Sensitive Dependencies

Tal Linzen, E. Dupoux, Y. Goldberg - 2016

5 papers in library cite

[23]Statistical Machine Translation

P. Koehn - 2010

5 papers in library cite

[24]A High-Performance Semi-Supervised Learning Method for Text Chunking

Rie Kubota Ando, Tong Zhang - 2005

4 papers in library cite

[25]Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

E. F. T. K. Sang, F. D. Meulder - 2003

4 papers in library cite

[26]Neural Architectures for Named Entity Recognition

G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, C. Dyer - 2016

4 papers in library cite

[27]Semi-Supervised Sequential Labeling and Segmentation Using Gigaword Scale Unlabeled Data

J. Suzuki, H. Isozaki - 2008

4 papers in library cite

[28]Text Classification From Labeled and Unlabeled Documents Using Em

K. Nigam, A. K. Mccallum, Sebastian Thrun, T. Mitchell - 2000

4 papers in library cite

[29]Combining Labeled and Unlabeled Data With Co-Training

A. Blum, T. Mitchell - 1998

3 papers in library cite

[30]Deep Multi-Task Learning With Low Level Tasks Supervised at Lower Layers

A. Sogaard, Y. Goldberg - 2016

3 papers in library cite

[31]Design Challenges and Misconceptions in Named Entity Recognition

L. A. Ratinov, Dan Roth - 2009

3 papers in library cite

[32]End-to-End Sequence Labeling via Bi-Directional LSTM-CNNS-CRF

X. Ma, Eduard Hovy - 2016

3 papers in library cite

[33]Introduction to the CoNLL-2000 Shared Task Chunking

E. F. T. K. Sang, S. Buchholz - 2000

2 papers in library cite

[34]Limitations of Co-Training for Natural Language Learning From Large Datasets

D. Pierce, C. Cardie - 2001

2 papers in library cite

[35]Named Entity Recognition With Bidirectional LSTM-CNNs

J. Chiu, E. Nichols - 2016

2 papers in library cite

[36]Semi-Supervised Sequence Modeling With Syntactic Topic Models

Wentao Li, Andrew Mccallum - 2005

2 papers in library cite

[37]A Bidirectional Recurrent Neural Language Model for Machine Translation

A. Peris, F. Casacuberta - 2015

1 paper in library cites

[38]Bidirectional Language Model for Handwriting Recognition

V. Frinken, A. Fornes, J. Llados, J. M. Ogier - 2012

1 paper in library cites

[39]Discriminative Training Methods for Hidden Markov Models: Theory and Experiments With Perceptron Algorithms

Michael Collins - 2002

1 paper in library cites

[40]Joint Entity Recognition and Disambiguation

G. Luo, X. Huang, Chin Yew Lin, Z. Nie - 2015

1 paper in library cites

[41]Semi-Supervised Learning and Domain Adaptation in Natural Language Processing

A. Sogaard - 2013

1 paper in library cites

[42]Semi-Supervised Structured Output Learning Based on a Hybrid Generative and Discriminative Approach

J. Suzuki, A. Fujino, H. Isozaki - 2007

1 paper in library cites

[43]Syntax-Based Semi-Supervised Named Entity Tagging

B. Mohit, R. Hwa - 2005

1 paper in library cites

[44]The AI2 System at SemEval-2017 Task 10 (ScienceIE): Semi-Supervised End-to-End Entity and Relation Extraction

W. Ammar, M. E. Peters, C. Bhagavatula, Russell Power - 2017

1 paper in library cites

[45]Transfer Learning for Sequence Tagging With Hierarchical Recurrent Networks

Zhilin Yang, Ruslan Salakhutdinov, W. W. Cohen - 2017

1 paper in library cites

Cited by

5

papers in your library

Cites

19

papers in your library

Read

on October 24, 2025

Nothing too different from other LSTM + Attention papers

Tags

Paper Aliases

No aliases