Papperoni

2019

Green AI

Oren Etzioni

Open PDF Google Scholar

citations

Cite Score

57

AI summary

This paper advocates for Green AI by making efficiency a key evaluation metric for AI research alongside accuracy and proposes reporting the financial cost of developing, training, and running models to enable greener and more inclusive AI practices.

Main Contributions

Advocates for making efficiency a key evaluation metric for AI research alongside accuracy.
Proposes reporting the financial cost of developing, training, and running models to establish baselines for investigating increasingly efficient methods.
Aims to make AI greener and more inclusive, enabling broader participation in high-quality research.
Introduces the concept of Floating Point Operations (FPO) as a hardware-agnostic measure for evaluating the amount of work performed by a computational process.
Encourages researchers to report budget/accuracy curves during training to allow for wiser model selection and highlight the stability of different approaches.

Abstract

The computations required for deep learning research have been doubling every few months, resulting in an estimated 300,000x increase from 2012 to 2018 [2]. These computations have a surprisingly large carbon footprint [40]. Ironically, deep learning was inspired by the human brain, which is remarkably energy efficient. Moreover, the financial cost of the computations can make it difficult for academics, students, and researchers, in particular those from emerging economies, to engage in deep learning research. This position paper advocates a practical solution by making efficiency an evaluation criterion for research alongside accuracy and related measures. In addition, we propose reporting the financial cost or “price tag" of developing, training, and running models to provide baselines for the investigation of increasingly efficient methods. Our goal is to make AI both greener and more inclusive-enabling any inspired undergraduate with a laptop to write high-quality research papers. Green Al is an emerging focus at the Allen Institute for AI.

Citation Graph

Loading graph...

References [51]

Sort:

Filter:

[1]Deep Residual Learning for Image Recognition

K. He, X. Zhang, S. Ren, Jian Sun - 2016

20 papers in library cite

This is simply amazing. Very very simple idea, totally revolutionary. No maths, just "it works!". Amazing.

[2]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[3]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[4]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[5]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[6]ImageNet: A Large-Scale Hierarchical Image Database

J. Deng, W. Dong, Richard Socher, L. J. Li, K. Li, Li Fei Fei - 2009

28 papers in library cite

Very nice idea and huge impact!

[7]Mastering the Game of Go With Deep Neural Networks and Tree Search

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. V. D. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, Ilya Sutskever, T. Lillicrap, M. Leach, Koray Kavukcuoglu, T. Graepel, Demis Hassabis - 2016

5 papers in library cite

Surprisingly simple. Years ago when I first heard of AlphaGo it seemed such a different/far away concept - "wow, it's AI!". Now that I understand this, it sounds like something I could have worked on (if I had my current knowledge)

[8]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[9]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[10]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[11]XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Z. Dai, Yining Yang, J. Carbonell, Ruslan Salakhutdinov, Quoc V. Le - 2019

11 papers in library cite

The method is nice and the results are very good but this paper is just soooo hard to follow...

[12]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[13]Energy and Policy Considerations for Deep Learning in NLP

E. Strubell, A. Ganesh, Andrew Mccallum - 2019

3 papers in library cite

Very nice push and maybe the first one to actually raise a flag - also, very contemporary and relevant discussion

[14]SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A. Wang, Y. Pruksachatkun, Nikita Nangia, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2019

15 papers in library cite

Nothing too surprising, just getting a bunch of stuff from different places and putting it all together. At least they do a good analysis of the benchmark vs. existing methodologies.

[15]You Only Look Once: Unified, Real-Time Object Detection

Joseph Redmon, S. Divvala, Ross Girshick, Ali Farhadi - 2016

1 paper in library cites

[16]Aggregated Residual Transformations for Deep Neural Networks

Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, K. He - 2017

3 papers in library cite

SOTA vision

[17]Exploring the Limits of Weakly Supervised Pretraining

D. Mahajan, Ross Girshick, V. Ramanathan, K. He, M. Paluri, Yiwei Li, A. Bharambe, Laurens Van Der Maaten - 2018

2 papers in library cite

CNN + transfer learning

[18]Defending Against Neural Fake News

Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, F. Roesner, Yejin Choi - 2019

5 papers in library cite

[19]On the State of the Art of Evaluation in Neural Language Models

G. Melis, C. Dyer, Phil Blunsom - 2018

6 papers in library cite

SotA LM in 2017

[20]MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

A. G. Howard, M. Zhu, Berlin Chen, D. Kalenichenko, Wenyi Wang, T. Weyand, M. Andreetto, Hartwig Adam - 2017

1 paper in library cites

[21]Squeeze-and-Excitation Networks

Jiaxi Hu, L. Shen, G. Sun - 2018

1 paper in library cites

[22]Mastering the Game of Go Without Human Knowledge

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Yanru Chen, T. Lillicrap, F. Hui, L. Sifre, G. V. D. Driessche, T. Graepel, Demis Hassabis - 2017

7 papers in library cite

[23]Mastering Chess and Shogi by Self-Play With a General Reinforcement Learning Algorithm

D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, Demis Hassabis - 2017

5 papers in library cite

[24]Algorithms for Hyper-Parameter Optimization

J. S. Bergstra, R. Bardenet, Yoshua Bengio, B. Kegl - 2011

3 papers in library cite

[25]Show Your Work: Improved Reporting of Experimental Results

J. Dodge, Suchin Gururangan, D. Card, Richard Schwartz, Noah A. Smith - 2019

3 papers in library cite

[26]The Unreasonable Effectiveness of Data

A. Halevy, P. Norvig, Fernando Pereira - 2009

3 papers in library cite

[27]An Analysis of Deep Neural Network Models for Practical Applications

A. Canziani, A. Paszke, E. Culurciello - 2017

2 papers in library cite

[28]Neural Architecture Search With Reinforcement Learning

Barret Zoph, Quoc V. Le - 2017

2 papers in library cite

[29]Pruning Convolutional Neural Networks for Resource Efficient Inference

P. Molchanov, S. Tyree, T. Karras, T. Aila, J. Kautz - 2017

2 papers in library cite

[30]Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

C. Sun, A. Shrivastava, Shivalika Singh, Aman Gupta - 2017

2 papers in library cite

[31]Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors

J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Zbigniew Wojna, Yueqi Song, S. Guadarrama, K. Murphy - 2017

2 papers in library cite

[32]Xnor-net: Imagenet Classification Using Binary Convolutional Neural Networks

M. Rastegari, V. Ordonez, Joseph Redmon, Ali Farhadi - 2016

2 papers in library cite

[33]AI and Compute

Dario Amodei, Danny Hernandez - 2018

1 paper in library cites

[34]Autopilot of Cement Plants for Reduction of Fuel Consumption and Emissions

P. Acharyya, S. D. Rosario, R. Flor, R. Joshi, Dustin Li, R. Linares, Haowei Zhang - 2019

1 paper in library cites

[35]Cramming More Components Onto Integrated Circuits

G. E. Moore - 1965

1 paper in library cites

[36]Data-Efficient Reinforcement Learning With Probabilistic Model-Predictive Control

S. Kamthe, M. P. Deisenroth - 2018

1 paper in library cites

[37]Deep Learning for Wildlife Conservation and Restoration Efforts

C. Duhart, G. Dublon, B. Mayton, G. Davenport, J. A. Paradiso - 2019

1 paper in library cites

[38]Dual Path Networks

Yanru Chen, Jeffrey Li, H. Xiao, X. Jin, Shuicheng Yan, J. Feng - 2017

1 paper in library cites

[39]Early Fusion for Goal Directed Robotic Vision

A. Walsman, Yonatan Bisk, S. Gabriel, Diganta Misra, Y. Artzi, Yejin Choi, D. Fox - 2019

1 paper in library cites

[40]Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization

Lei Li, K. Jamieson, G. Desalvo, A. Rostamizadeh, A. Talwalkar - 2017

1 paper in library cites

[41]Learning Time/Memory-Efficient Deep Architectures With Budgeted Super Networks

T. Veniat, L. Denoyer - 2018

1 paper in library cites

[42]MobileNetV2: Inverted Residuals and Linear Bottlenecks

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L. C. Chen - 2018

1 paper in library cites

[43]MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks

A. Gordon, E. Eban, O. Nachum, Berlin Chen, H. Wu, T. J. Yang, E. Choi - 2018

1 paper in library cites

[44]Open Loop Hyperparameter Optimization and Determinantal Point Processes

J. Dodge, K. Jamieson, Noah A. Smith - 2017

1 paper in library cites

[45]ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

N. Ma, X. Zhang, H. T. Zheng, Jian Sun - 2018

1 paper in library cites

[46]ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

X. Zhang, Xinyu Zhou, M. Lin, Jian Sun - 2018

1 paper in library cites

[47]SoPa: Bridging CNNs, RNNs, and Weighted Finite-State Machines

Richard Schwartz, S. Thomson, Noah A. Smith - 2018

1 paper in library cites

[48]Sparse Networks From Scratch: Faster Training Without Losing Performance

Tim Dettmers, Luke Zettlemoyer - 2019

1 paper in library cites

[49]Ssd: Single Shot multibox Detector

Weizhou Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, S. Reed, C. Y. Fu, A. C. Berg - 2016

1 paper in library cites

[50]Tackling Climate Change With Machine Learning

D. Rolnick, P. L. Donti, L. H. Kaack, K. Kochanski, A. Lacoste, K. Sankaran, A. S. Ross, N. M. Dupont, N. Jaques, A. W. Brown, A. Luccioni, T. Maharaj, E. D. Sherwin, S. K. Mukkavilli, K. P. Kording, C. Gomes, Andrew Y. Ng, Demis Hassabis, John C. Platt, F. Creutzig, J. Chayes, Yoshua Bengio - 2019

1 paper in library cites

[51]The AI Index 2018 Annual Report. Al Index Steering Committee, Human-Centered Al Initiative, Stanford University

Y. Shoham, R. Perrault, E. Brynjolfsson, Jack Clark, J. Manyika, J. C. Niebles, T. Lyons, J. Etchemendy, Z. Bauer - 2018

1 paper in library cites

Cited by

4

papers in your library

Cites

21

papers in your library

Read

on November 27, 2025

I like the push a lot! Specially when it comes to make research more democratic, which I personally benefit from. Also, AllenAI seems like they do good, honest work!

Tags

Paper Aliases

No aliases