Papperoni

2014

Hidden Technical Debt in Machine Learning Systems

Dan Dennison

Open PDF Google Scholar

citations

Cite Score

55

AI summary

This paper examines the technical debt incurred in machine learning systems, highlighting ML-specific risk factors such as boundary erosion, entanglement, and data dependencies, and emphasizes the need for better abstractions, testing, and design patterns to maintain long-term system health.

Main Contributions

Identifies ML-specific risk factors contributing to technical debt.
Explores system-level anti-patterns in ML system design.
Emphasizes the need for better abstractions and testing methodologies.
Highlights the importance of team culture in managing technical debt.
Provides insights into measuring and paying off technical debt in ML systems.

Abstract

Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt, we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns.

Citation Graph

Loading graph...

References [17]

Sort:

Filter:

[1]Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising

Leon Bottou, J. Peters, J. Q. Candela, D. X. Charles, D. M. Chickering, E. Portugaly, D. Ray, Patrice Simard, E. Snelson - 2013

2 papers in library cite

"Down to earth mathematical explanations" good read to get examples of explaining maths, but 50+ pages

Missing year

[2]Machine Learning: The High-Interest Credit Card of Technical Debt

A. Anonymous

2 papers in library cite

[3]Project adam: Building an Efficient and Scalable Deep Learning Training System

T. M. Chilimbi, Y. Suzue, J. Apacible, K. Kalyanaraman - 2014

2 papers in library cite

[4]Scaling Distributed Machine Learning With the Parameter Server

M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, B. Su - 2014

2 papers in library cite

[5]Ad Click Prediction: A View From the Trenches

H. B. Mcmahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, E. Davydov, D. Golovin, S. Chikkerur, D. Liu, M. Wattenberg, A. M. Hrafnkelsson, T. Boulos, J. Kubica - 2013

1 paper in library cites

[6]Antipatterns: Refactoring Software, Architectures, and Projects in Crisis

W. J. Brown, H. W. Mccormick, T. J. Mowbray, R. C. Malveau - 1998

1 paper in library cites

Missing year

M. Fowler

1 paper in library cites

[8]Detecting Adversarial Advertisements in the Wild

D. Sculley, M. E. Otey, M. Pohl, B. Spitznagel, J. Hainsworth, Y. Zhou - 2011

1 paper in library cites

[9]Google's Hybrid Approach to Research

A. Spector, P. Norvig, Slav Petrov - 2012

1 paper in library cites

[10]Photon: Fault-Tolerant and Scalable Joining of Continuous Data Streams

R. Ananthanarayanan, V. Basker, S. Das, Aman Gupta, H. Jiang, T. Qiu, A. Reznichenko, D. Ryabkov, M. Singh, S. Venkataraman - 2013

1 paper in library cites

[11]Refactoring: Improving the Design of Existing Code

M. Fowler - 1999

1 paper in library cites

[12]Scalable Hands-Free Transfer Learning for Online Advertising

B. Dalessandro, Deli Chen, T. Raeder, C. Perlich, M. H. Williams, F. Provost - 2014

1 paper in library cites

[13]Scaling Big Data Mining Infrastructure: The Twitter Experience

Junyang Lin, D. Ryaboy - 2013

1 paper in library cites

[14]Searching for Build Debt: Experiences Managing Technical Debt at google

J. D. Morgenthaler, M. Gridnev, R. Sauciuc, S. Bhansali - 2012

1 paper in library cites

[15]SEC Charges Knight Capital With Violations of Market Access Rule

Securities, E. Commission - 2013

1 paper in library cites

Missing year

[16]The Challenges of Building Machine Learning Tools for the Masses

A. Zheng

1 paper in library cites

[17]The Epoch-Greedy Algorithm for Multi-Armed Bandits With Side Information

John Langford, Tong Zhang - 2008

1 paper in library cites

Cited by

1

papers in your library

Cites

1

papers in your library

Read

on November 26, 2025

It's very good for raising the flag, and several things they describe I have personally experienced. However, as I have already plenty of experience with this, I think it's not too surprising.

Tags

Paper Aliases

No aliases