Papperoni

2012

Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

Matei Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. Mccauley, M. J. Franklin, S. Shenker, Ion Stoica

Open PDF Google Scholar

citations

Cite Score

77

AI summary

This paper introduces Resilient Distributed Datasets (RDDs), a distributed memory abstraction for fault-tolerant in-memory cluster computing, implemented in Spark. It achieves a 20x speedup over Hadoop in iterative applications and speeds up an analytics report by 40x. It enables interactive queries on a 1 TB dataset with 5-7s latency.

Main Contributions

Introduces Resilient Distributed Datasets (RDDs), a distributed memory abstraction for fault-tolerant in-memory cluster computing.
RDDs enable efficient data reuse in a broad range of applications.
RDDs provide fault tolerance by logging the transformations used to build a dataset (its lineage) rather than the actual data.
Implemented RDDs in a system called Spark.
Spark outperforms Hadoop by up to 20x in iterative applications.

Abstract

We present Resilient Distributed Datasets (RDDs), a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner. RDDs are motivated by two types of applications that current computing frameworks handle inefficiently: iterative algorithms and interactive data mining tools. In both cases, keeping data in memory can improve performance by an order of magnitude. To achieve fault tolerance efficiently, RDDs provide a restricted form of shared memory, based on coarsegrained transformations rather than fine-grained updates to shared state. However, we show that RDDs are expressive enough to capture a wide class of computations, including recent specialized programming models for iterative jobs, such as Pregel, and new applications that these models do not capture. We have implemented RDDs in a system called Spark, which we evaluate through a variety of user applications and benchmarks.

Citation Graph

Loading graph...

References [33]

Sort:

Filter:

[1]MapReduce: Simplified Data Processing on Large Clusters

Jeffrey Dean, Sanjay Ghemawat - 2004

4 papers in library cite

Amazing paper that discusses how MapReduce works! Very simple, and really nice to read something not related to AI. A shame that it's off-topic.

[2]Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

Matei Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. Mccauley, M. J. Franklin, S. Shenker, Ion Stoica - 2012

2 papers in library cite

Very nice paper. Interesting throughout, and introduced Spark! Very very nice.

[3]Ciel: A Universal Execution Engine for Distributed Data-Flow Computing

D. G. Murray, M. Schwarzkopf, C. Smowton, S. Smith, A. Madhavapeddy, S. Hand - 2011

2 papers in library cite

[4]Dryad: Distributed Data-Parallel Programs From Sequential Building Blocks

M. Isard, M. Budiu, Y. Yu, A. Birrell, D. Fetterly - 2007

2 papers in library cite

[5]FlumeJava: Easy, Efficient Data-Parallel Pipelines

C. Chambers, A. Raniwala, F. Perry, S. Adams, R. R. Henry, R. Bradshaw, N. Weizenbaum - 2010

2 papers in library cite

[6]The Anatomy of a Large-Scale Hypertextual Web Search Engine

S. Brin, L. Page - 1998

2 papers in library cite

[7]A First Order Approximation to the Optimum Checkpoint Interval

J. W. Young - 1974

1 paper in library cites

Missing author listMissing year

1 paper in library cites

[9]Caching Function Calls Using Precise Dependencies

A. Heydon, R. Levin, Y. Yu - 2000

1 paper in library cites

[10]Comet: Batched Stream Processing for Data Intensive Distributed Computing

Bingxiang He, Michael Yang, Z. Guo, R. Chen, B. Su, Wuwei Lin, L. Zhou - 2010

1 paper in library cites

[11]Database Management Systems

R. Ramakrishnan, J. Gehrke - 2003

1 paper in library cites

[12]Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling

Matei Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, Ion Stoica - 2010

1 paper in library cites

[13]Design and Evaluation of a Real-Time URL Spam Filtering Service

K. Thomas, C. Grier, J. Ma, V. Paxson, Dawn Song - 2011

1 paper in library cites

[14]Disk-Locality in Datacenter Computing Considered Irrelevant

G. Ananthanarayanan, A. Ghodsi, S. Shenker, Ion Stoica - 2011

1 paper in library cites

[15]Distributed Shared Memory: A Survey of Issues and Algorithms

B. Nitzberg, V. Lo - 1991

1 paper in library cites

[16]DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language

Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, J. Currey - 2008

1 paper in library cites

[17]HaLoop: Efficient Iterative Data Processing on Large Clusters

Y. Bu, B. Howe, M. Balazinska, M. D. Ernst - 2010

1 paper in library cites

[18]Incoop: MapReduce for Incremental Computations

P. Bhatotia, A. Wieder, R. Rodrigues, U. A. Acar, R. Pasquin - 2011

1 paper in library cites

[19]Large-Scale Incremental Processing Using Distributed Transactions and Notifications

D. Peng, F. Dabek - 2010

1 paper in library cites

[20]Lineage Retrieval for Scientific Data Processing: A Survey

R. Bose, J. Frew - 2005

1 paper in library cites

[21]Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center

B. Hindman, Andy Konwinski, Matei Zaharia, A. Ghodsi, A. D. Joseph, R. H. Katz, S. Shenker, Ion Stoica - 2011

1 paper in library cites

[22]Nectar: Automatic Management of Data and Computation in Datacenters

P. K. Gunda, L. Ravindranath, C. A. Thekkath, Y. Yu, L. Zhuang - 2010

1 paper in library cites

[23]On Availability of Intermediate Data in Cloud Computations

S. Y. Ko, I. Hoque, B. Cho, I. Gupta - 2009

1 paper in library cites

[24]Piccolo: Building Fast, Distributed Programs With Partitioned Tables

Russell Power, Jeffrey Li - 2010

1 paper in library cites

[25]Pregel: A System for Large-Scale Graph Processing

G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, G. Czajkowski - 2010

1 paper in library cites

[26]Provenance in Databases: Why, How, and Where

J. Cheney, L. Chiticariu, W. C. Tan - 2009

1 paper in library cites

[27]R2: An Application-Level Kernel for Record and Replay

Z. Guo, Xinpeng Wang, Jie Tang, Xiaodong Liu, Zhiwei Xu, M. Wu, M. F. Kaashoek, Zhengyou Zhang - 2008

1 paper in library cites

Missing author listMissing year

1 paper in library cites

[29]Scaling the Mobile Millennium System in the Cloud

T. Hunter, T. Moldovan, Matei Zaharia, S. Merzgui, J. Ma, M. J. Franklin, P. Abbeel, A. M. Bayen - 2011

1 paper in library cites

[30]Stateful Bulk Processing for Incremental Analytics

D. Logothetis, C. Olston, B. Reed, K. C. Webb, K. Yocum - 2010

1 paper in library cites

[31]The Case for RAMClouds: Scalable High-Performance Storage Entirely in DRAM

J. Ousterhout, P. Agrawal, D. Erickson, C. Kozyrakis, J. Leverich, D. Mazieres, S. Mitra, A. Narayanan, G. Parulkar, M. Rosenblum, S. M. Rumble, E. Stratmann, R. Stutsman - 2010

1 paper in library cites

[32]The Elements of Statistical Learning: Data Mining, Inference, and Prediction

T. Hastie, R. Tibshirani, J. Friedman - 2009

1 paper in library cites

[33]Twister: A Runtime for Iterative Mapreduce

J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S. H. Bae, Jiezhong Qiu, G. Fox - 2010

1 paper in library cites

Cited by

2

papers in your library

Cites

2

papers in your library

Read

on August 3, 2025

Very nice paper. Interesting throughout, and introduced Spark! Very very nice.

Tags

Paper Aliases

No aliases