Papperoni

2014

Neural Turing Machines

Alex Graves, G. Wayne, Ivo Danihelka

Open PDF Google Scholar

citations

Cite Score

66

AI summary

This paper introduces Neural Turing Machines (NTM), coupling neural networks with external memory, interacting via attentional processes. NTMs are differentiable and trainable with gradient descent, showing they can infer algorithms like copying, sorting, and associative recall, outperforming LSTMs on algorithmic tasks.

Main Contributions

Introduces the Neural Turing Machine (NTM) architecture, combining neural networks with external memory.
Demonstrates that NTMs can be trained end-to-end using gradient descent due to their differentiable nature.
Shows NTMs can infer simple algorithms such as copying, sorting, and associative recall.
Presents experimental results showing that NTMs learn faster and generalize better than LSTMs on the tested algorithmic tasks.
Introduces content-based and location-based addressing mechanisms for interacting with the external memory.

Abstract

We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine or Von Neumann architecture but is differentiable end-to-end, allowing it to be efficiently trained with gradient descent. Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.

Citation Graph

Loading graph...

References [42]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[3]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[4]Neural Networks and Physical Systems With Emergent Collective Computational Abilities

J. J. Hopfield - 1982

8 papers in library cite

Very theory heavy and doesn't seem to add much... Famous because it introduced the "Hopfield network" - "At a time when research in neural networks was facing a lull, Hopfield's paper injected a fresh perspective by demonstrating how a network of simple, interconnected units could exhibit complex computational abilities as a collective, emergent property."

[5]Speech Recognition With Deep Recurrent Neural Networks

Geoffrey Hinton - 2013

13 papers in library cite

Small, simple, nice results. First usage of deep BiLSTM?

[6]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[7]Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies

Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber - 2001

16 papers in library cite

Wow, this is so much better than the other paper - I should have read it sooner. It's concise and not too abstract, and also gives very good context on RNN problems and how to solve them.

[8]Learning to Learn Using Gradient Descent

Sepp Hochreiter, A. Steven Younger, Peter R. Conwell - 2001

4 papers in library cite

I don't see how this is meta-learning.

[9]Generating Text With Recurrent Neural Networks

Ilya Sutskever, James Martens, Geoffrey E. Hinton - 2011

13 papers in library cite

Pleasant paper but results are underwhelming. They use RNNs for character-level modeling, which is different. They also use the hessian-free method proposed by Martens, but don't go too deep into how it works, which is nice because otherwise it would be very mathy. Other papers cite this more as an example of usage rather than an actual milestone.

[10]Learning Distributed Representations of Concepts

Geoffrey E. Hinton - 1986

13 papers in library cite

 Probably seminal, but a bit boring overall

[11]Towards End-to-End Speech Recognition With Recurrent Neural Networks

Alex Graves, Navdeep Jaitly - 2014

2 papers in library cite

Graves, speech recog

[12]Recursive Distributed Representations

J. B. Pollack - 1990

7 papers in library cite

[13]Parallel Distributed Processing

D. E. Rumelhart, J. L. Mcclelland, P. R. Group - 1986

15 papers in library cite

[14]Semantic Compositionality Through Recursive Matrix-Vector Spaces

Richard Socher, B. Huval, Christopher D. Manning, Andrew Y. Ng - 2012

7 papers in library cite

[15]Learning Context-Free Grammars: Capabilities and Limitations of a Recurrent Neural Network With an External Stack Memory

S. Das, C. Giles, G. Sun - 1992

5 papers in library cite

[16]Connectionism and Cognitive Architecture: A Critical Analysis

J. A. Fodor, Z. W. Pylyshyn - 1988

2 papers in library cite

[17]Holographic Reduced Representation: Distributed Representation for Cognitive Structures

T. A. Plate - 2003

2 papers in library cite

[18]Three Models for the Description of Language

N. Chomsky - 1956

2 papers in library cite

[19]A General Framework for Adaptive Processing of Data Structures

Paolo Frasconi, M. Gori, A. Sperduti - 1998

1 paper in library cites

[20]Banishing the Homunculus: Making Working Memory Work

T. E. Hazy, M. J. Frank, R. C. O'reilly - 2006

1 paper in library cites

[21]Boltzcons: Dynamic Symbol Structures in a Connectionist Network

D. S. Touretzky - 1990

1 paper in library cites

[22]Cellular Basis of Working Memory

P. S. G. Rakic - 1995

1 paper in library cites

[23]Computation: finite and Infinite Machines

M. L. Minsky - 1967

1 paper in library cites

[24]Continuous Attractors and Oculomotor Control

H. S. Seung - 1998

1 paper in library cites

[25]First Draft of a Report on the Edvac

J. V. Neumann - 1945

1 paper in library cites

[26]How to Build a Brain: A Neural Architecture for Biological Cognition

C. Eliasmith - 2013

1 paper in library cites

[27]Hyperdimensional Computing: An Introduction to Computing in Distributed Representation With High-Dimensional Random Vectors

P. Kanerva - 2009

1 paper in library cites

[28]Machine Learning: A Probabilistic Perspective

Kevin P. Murphy - 2012

1 paper in library cites

A. Baddeley, M. Eysenck, M. Anderson - 2009

1 paper in library cites

[30]Memory and the Computational Brain: Why Cognitive Science Will Transform Neuroscience

C. R. Gallistel, A. P. King - 2009

1 paper in library cites

[31]On the Computational Power of Neural Nets

H. T. Siegelmann, E. D. Sontag - 1995

1 paper in library cites

[32]Simple Substrates for Complex Cognition

Peter Dayan - 2008

1 paper in library cites

[33]Synaptic Basis of Cortical Persistent Activity: The Importance of Nmda Receptors to Working Memory

X. J. Wang - 1999

1 paper in library cites

[34]Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems

P. Smolensky - 1990

1 paper in library cites

[35]The Algebraic Mind: Integrating Connectionism and Cognitive Science

G. F. Marcus - 2003

1 paper in library cites

[36]The Cognitive Revolution: A Historical Perspective

George A. Miller - 2003

1 paper in library cites

[37]The Evolution of the Language Faculty: Clarifications and Implications

W. Fitch, M. D. Hauser, N. Chomsky - 2005

1 paper in library cites

[38]The Importance of Mixed Selectivity in Complex Cognitive Tasks

M. Rigotti, O. Barak, M. R. Warden, X. J. Wang, N. D. Daw, E. K. Miller, S. Fusi - 2013

1 paper in library cites

[39]The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information

George A. Miller - 1956

1 paper in library cites

[40]The Nature of the Language Faculty and Its Implications for Evolution of Language (Reply to Fitch, Hauser, and Chomsky)

R. Jackendoff, S. Pinker - 2005

1 paper in library cites

[41]The Problem of Rapid Variable Creation

R. F. Hadley - 2009

1 paper in library cites

[42]Time Constraints and Resource Sharing in Adults' Working Memory Spans

P. Barrouillet, S. Bernardin, V. Camos - 2004

1 paper in library cites

Cited by

18

papers in your library

Cites

13

papers in your library

Read

on October 13, 2025

This paper is amazing. If someone told me that NNs could use and address memory by position I wouldn't believe it worked. Very nice, but it's a shame that it's just a toy example.

Tags

Paper Aliases

No aliases