Papperoni

2018

Winner's Curse? On Pace, Progress, and Empirical Rigor

Alex Wiltschko

Open PDF Google Scholar

citations

Cite Score

10

AI summary

This paper explores potential issues that may be arising in the field of Machine Learning, as the rate of empirical advancement may not have been matched by a consistent increase in the level of empirical rigor across the field as a whole.

Main Contributions

Highlights examples where progress has actually been slowed as a result of the lack of empirical rigor.
Offers thoughts on incentive structures currently at play.
Gives suggestions as seeds for discussions on productive change.
Highlights the importance of tuning methodology, sliced analysis, ablation studies, and sanity checks.

Abstract

The field of ML is distinguished both by rapid innovation and rapid dissemination of results. While the pace of progress has been extraordinary by any measure, in this paper we explore potential issues that we believe to be arising as a result. In particular, we observe that the rate of empirical advancement may not have been matched by consistent increase in the level of empirical rigor across the field as a whole. This short position paper highlights examples where progress has actually been slowed as a result, offers thoughts on incentive structures currently at play, and gives suggestions as seeds for discussions on productive change.

Citation Graph

Loading graph...

References [19]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]Human-Level Control Through Deep Reinforcement Learning

V. Mnih - 2015

9 papers in library cite

Maybe the first instance of RL with NNs! I am impressed that it seems quite straightforward to implement (maybe tricky to get working).

[3]No Free Lunch Theorems for Optimization

David H. Wolpert, William G. Macready - 1997

1 paper in library cites

Nice read, but it gets a bit tough to follow after Section 4. Very nice results thought... I missed some practical implications of this. Like, results are interesting but so what? Seems like this expanded into some pretty interesting interpretations, but they don't discuss much (there is one paragraph that does)

[4]Hidden Technical Debt in Machine Learning Systems

Dan Dennison - 2014

1 paper in library cites

It's very good for raising the flag, and several things they describe I have personally experienced. However, as I have already plenty of experience with this, I think it's not too surprising.

[5]Deep Reinforcement Learning That Matters

P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, D. Meger - 2017

2 papers in library cite

Reproducibility review of RL methods

[6]Are Gans Created Equal? A Large-Scale Study

M. Lucic, K. Kurach, M. Michalski, Sylvain Gelly, O. Bousquet - 2017

2 papers in library cite

Reproducibility study

[7]On the State of the Art of Evaluation in Neural Language Models

G. Melis, C. Dyer, Phil Blunsom - 2018

6 papers in library cite

SotA LM in 2017

[8]Practical Bayesian Optimization of Machine Learning Algorithms

J. Snoek, Hugo Larochelle, R. P. Adams - 2012

9 papers in library cite

[9]Mastering the Game of Go Without Human Knowledge

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Yanru Chen, T. Lillicrap, F. Hui, L. Sifre, G. V. D. Driessche, T. Graepel, Demis Hassabis - 2017

7 papers in library cite

[10]Regularized Evolution for Image classifier Architecture Search

Y. H. Q. V. Le, E. Real, A. Aggarwal - 2018

3 papers in library cite

[11]Sequential Model-Based Optimization for General Algorithm configuration

Frank Hutter, H. H. Hoos, K. L. Brown - 2011

3 papers in library cite

Missing year

[12]Machine Learning: The High-Interest Credit Card of Technical Debt

A. Anonymous

2 papers in library cite

[13]Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling

C. Rikelme, G. Tucker, J. Snoek - 2018

1 paper in library cites

[14]Google Vizier: A Service for Black-Box Optimization

D. Golovin, B. Solnik, S. Moitra, G. Kochanski, J. E. Karro, D. Sculley - 2017

1 paper in library cites

[15]Intricacies of Collins' Parsing Model

D. M. Bikel - 2004

1 paper in library cites

Missing year

[16]Nips Test of Time Award Presentation

A. Rahimi, Benjamin Recht

1 paper in library cites

[17]Open Scholarship and Peer Review: A Time for Experimentation

D. Soergel, A. Saunders, Andrew Mccallum - 2013

1 paper in library cites

Missing year

[18]The Machine Learning Bubble?

M. Mozer

1 paper in library cites

[19]The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction

E. Breck, S. Cai, E. Nielsen, M. Salib, D. Sculley - 2017

1 paper in library cites

Cited by

1

papers in your library

Cites

7

papers in your library

Read

on November 21, 2025

Well, it's okay-ish. It's a good push but as an opinion piece is very mid.

Tags

Paper Aliases

No aliases