Papperoni

2015

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

M. Abadi, Akshat Agarwal, P. Barham, E. Brevdo, Ziru Chen, C. Citro, G. Corrado, A. Davis, Jeffrey Dean, M. Devin, Sanjay Ghemawat, I. Goodfellow, A. Harp, Geoffrey Irving, M. Isard, Y. Jia, R. Jozefowicz, Lukasz Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray, Christopher Olah, M. Schuster, J. Shlens, B. Steiner, Ilya Sutskever, K. Talwar, P. Tucker, Vincent Vanhoucke, V. Vasudevan, F. Viegas, Oriol Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, Xiaoqiang Zheng

Open PDF Google Scholar

citations

Cite Score

88

AI summary

This paper introduces TensorFlow, a flexible interface and implementation for expressing and executing machine learning algorithms on heterogeneous distributed systems. It supports training and inference for deep neural networks, achieving scalability across diverse platforms from mobile devices to large-scale clusters. The TensorFlow API and reference implementation were released as open-source.

Main Contributions

Introduces TensorFlow, an interface for expressing machine learning algorithms.
Presents an implementation for executing algorithms on heterogeneous systems.
Demonstrates flexibility across a wide variety of algorithms, including training and inference for deep neural networks.
Describes the TensorFlow API and an implementation built at Google.
Releases the TensorFlow API and a reference implementation as an open-source package.

Abstract

TensorFlow [1] is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms. A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards. The system is flexible and can be used to express a wide variety of algorithms, including training and inference algorithms for deep neural network models, and it has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields, including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction, and computational drug discovery. This paper describes the TensorFlow interface and an implementation of that interface that we have built at Google. The TensorFlow API and a reference implementation were released as an open-source package under the Apache 2.0 license in November, 2015 and are available at www.tensorflow.org.

Citation Graph

Loading graph...

References [56]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[3]Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

S. Ioffe, Christian Szegedy - 2015

18 papers in library cite

Very good paper! Similar feel as ResNets: simple idea, elegant. Not too mathy

[4]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[5]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[6]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[7]Caffe: Convolutional Architecture for Fast Feature Embedding

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, Ross Girshick, S. Guadarrama, Trevor Darrell - 2014

12 papers in library cite

Nothing new really, but worth the read. It's nice because it's the precursor to current AI frameworks + has a Python interface. Also good that model representation is separate from implementation

[8]TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

M. Abadi, Akshat Agarwal, P. Barham, E. Brevdo, Ziru Chen, C. Citro, G. Corrado, A. Davis, Jeffrey Dean, M. Devin, Sanjay Ghemawat, I. Goodfellow, A. Harp, Geoffrey Irving, M. Isard, Y. Jia, R. Jozefowicz, Lukasz Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray, Christopher Olah, M. Schuster, J. Shlens, B. Steiner, Ilya Sutskever, K. Talwar, P. Tucker, Vincent Vanhoucke, V. Vasudevan, F. Viegas, Oriol Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, Xiaoqiang Zheng - 2015

11 papers in library cite

This should be the golden standard to what framework papers should be. It's large, but it's not boring at all. Explains the core concepts while not going too deep as to describe unimportant things; explains design decisions and shortcomings... overall amazing

[9]Large-Scale Video Classification With Convolutional Neural Networks

A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, Li Fei Fei - 2014

2 papers in library cite

I liked it a lot. It's nothing "wow", but a very nice approach, and apparently the first o apply CNN to video, which is nice :)

[10]Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

Matei Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. Mccauley, M. J. Franklin, S. Shenker, Ion Stoica - 2012

2 papers in library cite

Very nice paper. Interesting throughout, and introduced Spark! Very very nice.

[11]Large Scale Distributed Deep Networks

Jeffrey Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Quoc V. Le, Mark Z. Mao, Marc'aurelio Ranzato, A. Senior, P. Tucker, K. Yang, Andrew Y. Ng - 2012

16 papers in library cite

Good paper, nice algorithm. Nothing too crazy, but I understand the impact. I think the work to create the system was larger than the algorithm itself.

[12]Pointer Networks

Oriol Vinyals, M. Fortunato, Navdeep Jaitly - 2015

10 papers in library cite

Cool concept. Nice that it works and can find good solutions for TSP.

[13]Building High-Level Features Using Large Scale Unsupervised Learning

Quoc V. Le, M. A. Ranzato, R. Monga, M. Devin, K. Chen, G. S. Corrado, Jeffrey Dean, Andrew Y. Ng - 2012

10 papers in library cite

Very nice and very early work - seems very simple but very insightful to use an autoencoder to detect objects. Also, very similar to the neocognitron :)

[14]One Weird Trick for Parallelizing Convolutional Neural Networks

Alex Krizhevsky - 2014

3 papers in library cite

Very nice paper. Good new approach to training and explains the idea and rationale well. Not too many citations but used by SotA CNNs at the time.

[15]Grammar as a Foreign Language

Geoffrey Hinton - 2015

9 papers in library cite

It's a nice paper showing that attention can be used for parsing. However, parsing is boring and is very derivative. Good paper nonetheless.

[16]On Rectified Linear Units for Speech Processing

Matthew D. Zeiler, M. A. Ranzato, R. Monga, M. Mao, K. Yang, Quoc Le, P. Nguyen, A. Senior, Vincent Vanhoucke, Jeffrey Dean - 2013

3 papers in library cite

It's boring and the only thing it brings to the table is usage of ReLU in speech recog (which is unsurprising since it was already used everywhere else)

[17]Theano: A CPU and GPU Math Expression Compiler

James Bergstra, O. Breuleux, F. Bastien, P. Lamblin, Razvan Pascanu, G. Desjardins, J. Turian, D. W. Farley, Yoshua Bengio - 2010

22 papers in library cite

Very nice framework. Symbolic programming is very nice. However, I think that this had very little impact and was mostly used by Bengio's lab.

[18]Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups

Geoffrey E. Hinton, L. Deng, D. Yu, George E. Dahl, A. Mohamed, Navdeep Jaitly, A. Senior, Vincent Vanhoucke, P. Nguyen, T. N. Sainath, Brian Kingsbury - 2012

8 papers in library cite

The core of the paper itself is a bit boring and doesn't introduce anything new (just RBMs and DBNs again) but I am giving this a 4 because it's probably the best explanation of RBMs and DBNs I've read so far.

[19]The MNIST Database of Handwritten Digits

Yann Lecun - 1998

8 papers in library cite

Not a paper - it's actually a dataset

[20]Devise: A Deep Visual-Semantic Embedding Model

Andrea Frome, G. S. Corrado, J. Shlens, Samy Bengio, Jeffrey Dean, Tomas Mikolov, Marc'aurelio Ranzato - 2013

4 papers in library cite

Identify objects

[21]Multiple Object Recognition With Visual Attention

Jimmy Lei Ba, V. Mnih, Koray Kavukcuoglu - 2014

4 papers in library cite

Cited by DRAW

[22]Multi-Digit Number Recognition From Street View Imagery Using Deep Convolutional Neural Networks

Ian J. Goodfellow, Y. Bulatov, J. Ibarz, S. Arnoud, V. Shet - 2014

4 papers in library cite

Goodfellow, street number recog

[23]Move Evaluation in Go Using Deep Convolutional Neural Networks

C. Maddison, A. Huang, Ilya Sutskever, D. Silver - 2015

2 papers in library cite

Precursor to AlphaGO?

[24]Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

Benjamin Recht, C. Re, S. Wright, F. Niu - 2011

6 papers in library cite

[25]NVIDIA CUBLAS Library

Nvidia - 2009

3 papers in library cite

[26]Algorithm 679: A Set of Level 3 Basic Linear Algebra Subprograms

J. J. Dongarra, J. D. Croz, I. S. Duff, S. Hammarling - 1990

2 papers in library cite

[27]Ciel: A Universal Execution Engine for Distributed Data-Flow Computing

D. G. Murray, M. Schwarzkopf, C. Smowton, S. Smith, A. Madhavapeddy, S. Hand - 2011

2 papers in library cite

[28]Cuda-Convnet

Alex Krizhevsky - 2014

2 papers in library cite

[29]Dryad: Distributed Data-Parallel Programs From Sequential Building Blocks

M. Isard, M. Budiu, Y. Yu, A. Birrell, D. Fetterly - 2007

2 papers in library cite

Missing author listMissing year

[30]Eigen, a C++ Template Library for Linear Algebra

2 papers in library cite

[31]FlumeJava: Easy, Efficient Data-Parallel Pipelines

C. Chambers, A. Raniwala, F. Perry, S. Adams, R. R. Henry, R. Bradshaw, N. Weizenbaum - 2010

2 papers in library cite

[32]Multilingual Acoustic Models Using Distributed Deep Neural Networks

Georg Heigold, Vincent Vanhoucke, A. Senior, P. Nguyen, M. A. Ranzato, M. Devin, Jeffrey Dean - 2013

2 papers in library cite

[33]Project adam: Building an Efficient and Scalable Deep Learning Training System

T. M. Chilimbi, Y. Suzue, J. Apacible, K. Kalyanaraman - 2014

2 papers in library cite

[34]Torch: A Modular Machine Learning Software Library

Ronan Collobert, Samy Bengio, J. Mariethoz - 2002

2 papers in library cite

[35]An Introduction to Computational Networks and the Computational Network Toolkit

D. Yu, A. Eversole, M. Seltzer, K. Yao, Zhongqiang Huang, B. Guenter, O. Kuchaiev, Y. Z. Zhang, F. Seide, Haiming Wang - 2014

1 paper in library cites

[36]Annual Review of Computer Science Vol. 1

Arvind, D. E. Culler - 1986

1 paper in library cites

Missing year

[37]Chainer: A Powerful, Flexible and Intuitive Framework of Neural Networks

S. Tokui

1 paper in library cites

[38]cuDNN: Efficient Primitives for Deep Learning

S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, Bryan Catanzaro, E. Shelhamer - 2014

1 paper in library cites

[39]Dandelion: A Compiler and Runtime for Heterogeneous Systems

C. J. Rossbach, Y. Yu, J. Currey, J. Martin, D. Fetterly - 2013

1 paper in library cites

[40]Executing a Program on the MIT Tagged-Token Dataflow Architecture

Arvind, R. S. Nikhil - 1990

1 paper in library cites

[41]Frame-by-Frame Language Identification in Short Utterances Using Deep Neural Networks

J. G. Dominguez, I. L. Moreno, P. J. Moreno, J. G. Rodriguez - 2015

1 paper in library cites

[42]Global code Motion/Global Value Numbering

C. Click - 1995

1 paper in library cites

[43]Google Turning Its Lucrative Web Search Over to AI Machines

Jack Clark - 2015

1 paper in library cites

[44]Google Voice Search: Faster and More Accurate

H. Sak, A. Senior, K. Rao, F. Beaufays, J. Schalkwyk - 2015

1 paper in library cites

[45]Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines

J. R. Kelley, C. Barnes, A. Adams, S. Paris, F. Durand, S. Amarasinghe - 2013

1 paper in library cites

[46]How Google Translate Squeezes Deep Learning Onto a Phone

O. Good - 2015

1 paper in library cites

[47]Improving Photo Search

C. R. Rosenberg - 2013

1 paper in library cites

[48]Large-Scale Cluster Management at Google with Borg

A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, J. Wilkes - 2015

1 paper in library cites

[49]Massively Multitask Networks for Drug Discovery

B. Ramsundar, S. Kearnes, P. Riley, D. Webster, D. Konerding, V. Pande - 2015

1 paper in library cites

[50]Massively Parallel Methods for Deep Reinforcement Learning

A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. D. Maria, V. Panneershelvam, M. Suleyman, C. Beattie - 2015

1 paper in library cites

[51]Naiad: A Timely Dataflow System

D. G. Murray, F. Mcsherry, R. Isaacs, M. Isard, P. Barham, M. Abadi - 2013

1 paper in library cites

Missing year

[52]Parameter Server

M. Li, D. Andersen, A. Smola

1 paper in library cites

[53]Pedestrian Detection With a Large-Field-of-View Deep Network

A. Angelova, Alex Krizhevsky, Vincent Vanhoucke - 2015

1 paper in library cites

[54]Speech Recognition and Deep Learning

Vincent Vanhoucke - 2015

1 paper in library cites

Missing year

[55]The CIFAR-10 Dataset

Alex Krizhevsky, V. Nair, Geoffrey Hinton

1 paper in library cites

[56]The Neural Networks Behind Google Voice Transcription

F. Beaufays - 2015

1 paper in library cites

Cited by

11

papers in your library

Cites

23

papers in your library

Read

on August 2, 2025

This should be the golden standard to what framework papers should be. It's large, but it's not boring at all. Explains the core concepts while not going too deep as to describe unimportant things; explains design decisions and shortcomings... overall amazing

Tags

Paper Aliases

No aliases