Papperoni

2018

Character-Level Language Modeling With Deeper Self-Attention

R. A. Rfou, D. Choe, Noah Constant, M. Guo, Llion Jones

Open PDF Google Scholar

citations

Cite Score

23

AI summary

This paper introduces a character-level language model using a deep (64-layer) transformer with self-attention. It achieves state-of-the-art results on text8 and enwik8 datasets by incorporating auxiliary losses at intermediate layers and positions, demonstrating the effectiveness of deep transformers for character-level language modeling.

Main Contributions

Demonstrates that a deep transformer model can outperform RNN variants in character-level language modeling.
Achieves state-of-the-art results on text8 and enwik8 benchmarks with 1.13 and 1.06 bits per character, respectively.
Shows the importance of adding auxiliary losses at intermediate network layers and sequence positions for training deep transformer models.
Introduces a 64-layer transformer architecture, significantly deeper than previous transformer networks.
Replaces the sinusoidal timing signal with a learned per-layer positional embedding to improve performance in deep networks.

Abstract

LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model (Vaswani et al. 2017) with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.

Citation Graph

Loading graph...

References [46]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

S. Ioffe, Christian Szegedy - 2015

18 papers in library cite

Very good paper! Similar feel as ResNets: simple idea, elegant. Not too mathy

[4]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[5]Layer Normalization

Jimmy Lei Ba, R. Kiros, Geoffrey E. Hinton - 2016

14 papers in library cite

Very nice! At first I had a little bit of prejudice because it seemed way too mathy, but actually the math is easy to follow and the results are very nice.

[6]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[7]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[8]Backpropagation Through Time: What It Does and How to Do It

P. Werbos - 1990

9 papers in library cite

Amazing tutorial! Very pragmatic. Explains very basic concepts and focus on implementation

[9]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[10]Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Noam Shazeer, Azalia Mirhoseini, K. Maziarz, A. Davis, Quoc Le, Geoffrey Hinton, Jeffrey Dean - 2017

9 papers in library cite

It's nice, but there's an important section in the middle about batch sizes that I don't quite understand. Not sure if I am missing some background knowledge or if they explain it poorly, and seems foundational to their main method... Either way, I did understand the methodology of the paper, and they have nice results :)

[11]End-to-End Memory Networks

S. Sukhbaatar, A. Szlam, Jason Weston, Rob Fergus - 2015

18 papers in library cite

This was so surprising! This is very similar to transformers and RAG. Who knew?!

[12]LSTM neural Networks for Language Modeling

M. Sundermeyer, R. Schluter, Hermann Ney - 2010

7 papers in library cite

It reads like an undergrad project - doesn't add anything new.

[13]Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies

Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber - 2001

16 papers in library cite

Wow, this is so much better than the other paper - I should have read it sooner. It's concise and not too abstract, and also gives very good context on RNN problems and how to solve them.

[14]Memory Networks

Jason Weston, S. Chopra, Antoine Bordes - 2015

18 papers in library cite

The first half of the paper (when they discuss the concept in a very abstract way) is amazing. However, the actual methodology was very convoluted - I did not like it. I thought that Neural Turing Machines were inspired in this, but actually they are contemporary... So anyway, the concept is nice, execution is not.

[15]A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Yarin Gal - 2015

9 papers in library cite

The methodology is very simple and effective, but maybe it was too simple for a paper, so they put A TON of math on top of it... Unnecessary really.

[16]Extensions of Recurrent Neural Network Language Model

Tomas Mikolov, S. Kombrink, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2011

16 papers in library cite

Doesn't add much.

[17]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[18]One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

C. Chelba, Tomas Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, Tony Robinson - 2013

13 papers in library cite

It's somewhat shallow, but I can see the importance of this paper.

[19]Learning to Generate Reviews and Discovering Sentiment

Alec Radford, R. Jozefowicz, Ilya Sutskever - 2017

8 papers in library cite

Very impressive results. We can see that they were onto something (later became GPT2)

[20]Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context

U. Khandelwal, He He, P. Qi, Dan Jurafsky - 2018

2 papers in library cite

Very nice paper and an analysis that makes sense! People thought that LSTMs had "infinite memory", but this paper shows that it is not so!

[21]Improving Neural Language Models With a Continuous Cache

E. Grave, Armand Joulin, Nicolas Usunier - 2016

7 papers in library cite

This was a surprise to me - I expected this to suck. However, they can provide a simple and intuitive way of improving LMs - nice

[22]Subword Language Modeling With Neural Networks

Tomas Mikolov, Ilya Sutskever, A. Deoras, H. S. Le, S. Kombrink, Jan Cernocky - 2012

7 papers in library cite

Doesn't add a ton of contribution other than saying that subword models can perform better than char-level models. Probably the most important thing here is early use of subwords.

[23]Character-Level Convolutional Networks for Text Classification

X. Zhang, J. Zhao, Yann Lecun - 2015

7 papers in library cite

LeCun + Character CNNs

[24]Language Modeling With Gated Convolutional Networks

Yann N. Dauphin, A. Fan, Michael Auli, D. Grangier - 2016

8 papers in library cite

[25]Regularizing and Optimizing 1stm Language Models

S. Merity, Nitish Shirish Keskar, Richard Socher - 2017

6 papers in library cite

SotA LM in 2017

[26]Gated Feedback Recurrent Neural Networks

J. Chung, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2015

3 papers in library cite

Youshua Bengio + Gated Networks

[27]Neural Machine Translation in Linear Time

N. Kalchbrenner, L. Espeholt, K. Simonyan, A. V. D. Oord, Alex Graves, Koray Kavukcuoglu - 2016

5 papers in library cite

Bytenet - Also "linear time" caught my attention

[28]Recurrent Highway Networks

J. G. Zilly, R. K. Srivastava, J. Koutnik, Jürgen Schmidhuber - 2016

6 papers in library cite

Recurrent highway networks - maybe read after the original highway network

[29]Recurrent Batch Normalization

T. Cooijmans, Nicolas Ballas, C. Laurent, Aaron Courville - 2016

3 papers in library cite

Method for implementing BN for RNNs

[30]Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations

David Krueger, T. Maharaj, J. Kramar, M. Pezeshki, Nicolas Ballas, N. R. Ke, A. G. A. P. Goyal, Yoshua Bengio, Hugo Larochelle, Aaron Courville - 2016

3 papers in library cite

Zoneout: variation of dropout, but with very little citations

[31]Dynamic Evaluation of Neural Sequence Models

B. Krause, E. Kahembwe, I. Murray, S. Renals - 2017

3 papers in library cite

[32]Multiplicative LSTM for Sequence Modelling

B. Krause, L. Lu, I. Murray, S. Renals - 2016

3 papers in library cite

[33]Fast-Slow Recurrent Neural Networks

A. Mujika, F. Meier, A. Steger - 2017

2 papers in library cite

[34]Hierarchical multiscale recurrent Neural Networks

J. Chung, S. Ahn, Yoshua Bengio - 2016

2 papers in library cite

[35]Independently Recurrent Neural Network (Indrnn): Building a Longer and Deeper RNN

Shanda Li, Wentao Li, C. Cook, C. Zhu, Y. Gao - 2018

2 papers in library cite

[36]Architectural Complexity Measures of Recurrent Neural Networks

S. Zhang, Yonghui Wu, T. Che, Zongyu Lin, R. Memisevic, Ruslan R. Salakhutdinov, Yoshua Bengio - 2016

1 paper in library cites

[37]Byte-Level Machine Reading Across Morphologically Varied Languages

T. Kenter, Llion Jones, D. Hewlett - 2018

1 paper in library cites

[38]Cmix

B. Knol - 2017

1 paper in library cites

[39]Frustratingly Short Attention Spans in Neural Language Modeling

M. Daniluk, Tim Rocktaschel, J. Welbl, Sebastian Riedel - 2017

1 paper in library cites

[40]Improving Gans Using Optimal Transport

T. Salimans, Haowei Zhang, Alec Radford, D. N. Metaxas - 2018

1 paper in library cites

[41]Large Text Compression Benchmark

M. Mahoney - 2009

1 paper in library cites

[42]Sparse Attentive Backtracking: Long-Range Credit Assignment in Recurrent Networks

N. R. Ke, A. G. A. P. Goyal, O. Bilaniuk, J. Binas, L. Charlin, C. Pal, Yoshua Bengio - 2017

1 paper in library cites

[43]Surprisal-Driven Feedback in Recurrent Networks

K. M. Rocki - 2016

1 paper in library cites

[44]Unbiasing Truncated Back Propagation Through Time

C. Tallec, Y. Ollivier - 2017

1 paper in library cites

[45]Unitary Evolution Recurrent Neural Networks

M. Arjovsky, A. Shah, Yoshua Bengio - 2015

1 paper in library cites

[46]Very Deep Convolutional Networks for Natural Language Processing

Alexis Conneau, Holger Schwenk, L. Barrault, Yann Lecun - 2016

1 paper in library cites

Cited by

6

papers in your library

Cites

30

papers in your library

Read

on November 16, 2025

It's good, and almost a 4, but I think it is a bit boring. Plus character LMs are not meta.

Tags

Paper Aliases

No aliases