Papperoni

2008

Utility Data Annotation With Amazon Mechanical Turk

Alexander Sorokin, David Forsyth

Open PDF Google Scholar

citations

Cite Score

33

AI summary

This paper presents a framework for outsourcing data annotation tasks to Amazon Mechanical Turk, demonstrating that this approach yields good quality, cheap, and fast annotations for various image labeling problems, and outlines strategies for task specification and pricing.

Main Contributions

Demonstrates how to efficiently outsource image annotation to Amazon Mechanical Turk.
Shows that annotations produced via MT are high-quality, cost-effective, and fast.
Introduces strategies for defining, pricing, and ensuring the quality of annotation tasks.
Discusses four different annotation protocols, including coarse object segmentation, polygonal labeling, and 14-point human landmark labeling.
Presents empirical results from five annotation experiments, collecting 3861 labels for 982 images at a total cost of US$59.

Abstract

We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced.

Citation Graph

Loading graph...

References [28]

Sort:

Filter:

[1]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[2]Labelme: A Database and Web-Based Tool for Image Annotation

Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, William T. Freeman - 2008

10 papers in library cite

It's a good paper overall but not worth the read. They just describe the platform (which for the time may have been a paradigm shift from in-house datasets). Maybe the basis for Amazon MT?

[3]One-Shot Learning of Object Categories

Li Fei Fei, Rob Fergus, Pietro Perona - 2006

5 papers in library cite

It's a nice problem they are trying to solve, though I doubt the effectiveness. Maybe it was really relevant in a time where they didn't have data. Problem is it focuses too much on the methodology, and I am not too interested in it.

[4]Caltech-256 Object Category Dataset

Greg Griffin, Alex Holub, Pietro Perona - 2007

9 papers in library cite

Boring read, but at least they were somewhat impactful.

[5]Labeling Images With a Computer Game

Luis Von Ahn, Laura Dabbish - 2004

5 papers in library cite

It's alright. TBH I expected a bit more. It's a good idea but I feel that they were very optimist around people playing - and it didn't work

[6]Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments

G. B. Huang, M. Ramesh, T. Berg, E. L. Miller - 2007

5 papers in library cite

Seems relevant but I am tired of reading papers about datasets, especially the early ones.

[7]Histograms of Oriented Gradients for Human Detection

N. Dalal, B. Triggs - 2005

12 papers in library cite

[8]The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results

Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, Andrew Zisserman - 2007

7 papers in library cite

[9]The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results

Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, Andrew Zisserman - 2007

7 papers in library cite

[10]Introduction to a Large-Scale General Purpose Ground Truth Database: Methodology, Annotation Tool and Benchmarks

B. Yao, X. Yang, S. Zhu - 2007

3 papers in library cite

[11]Learning to Detect Objects in Images via a Sparse, Part-Based Representation

Sandhini Agarwal, A. Awan, Dan Roth - 2004

3 papers in library cite

[12]A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics

D. Martin, C. Fowlkes, D. Tal, Jitendra Malik - 2001

2 papers in library cite

[13]Actions as Space-Time Shapes

M. Blank, L. Gorelick, E. Shechtman, M. Irani, R. Basri - 2005

2 papers in library cite

[14]Peekaboom: A Game for Locating Objects in Images

Luis Von Ahn, Rosanne Liu, M. Blum - 2006

2 papers in library cite

[15]A Trainable System for Object Detection

C. Papageorgiou, T. Poggio - 2000

1 paper in library cites

Missing author listMissing year

[16]Amazon Mechanical Turk

1 paper in library cites

[17]An Introduction to Evaluating Biometric Systems

P. J. Phillips, A. Martin, C. Wilson, M. Przybocki - 2000

1 paper in library cites

[18]Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection

P. N. Belhumeur, J. P. Hespanha, D. J. Kriegman - 1997

1 paper in library cites

Missing author list

2008

1 paper in library cites

[20]Evaluation of Localized Semantics: Data, Methodology, and Experiments

K. Barnard, Q. Fan, R. Swaminathan, A. Hoogs, R. Collins, P. Rondot, J. Kaufhold - 2008

1 paper in library cites

Missing author list

[21]Imageparsing

2008

1 paper in library cites

Missing year

[22]Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues

D. Martin, C. Fowlkes, Jitendra Malik

1 paper in library cites

[23]Learning to Parse Images of Articulated Bodies

D. Ramanan - 2007

1 paper in library cites

Missing author listMissing year

[24]Linguistic data Consortium

1 paper in library cites

[25]Recovering Human Body Configurations: Combining Segmentation and Recognition

G. Mori, Xiang Ren, A. Efros, Jitendra Malik - 2004

1 paper in library cites

[26]The cmu Pose, Illumination, and Expression(pie) Database

T. Sim, S. Baker, M. Bsat - 2002

1 paper in library cites

Missing author listMissing year

[27]The Pascal Visual Object Classes Challenge 2008

1 paper in library cites

[28]Who's in the Picture?

T. L. Berg, A. C. Berg, J. Edwards, David Forsyth - 2004

1 paper in library cites

Cited by

2

papers in your library

Cites

5

papers in your library

Read

on February 2, 2026

It's an alright guide to using Amazon MT. It's good that they lay the foundations to the methodology, but nothing too amazing

Tags

Paper Aliases

No aliases