Papperoni

2015

The Pascal Visual Object Classes Challenge: A Retrospective

Mark Everingham, S. M. Ali Eslami, Luc Van Gool, C. Williams, John Winn, Andrew Zisserman

citations

Cite Score

AI summary

This paper reviews the PASCAL VOC challenges from 2008-2012, introducing novel evaluation methods such as bootstrapping, normalized average precision, and clustering to analyze algorithm performance on datasets, track community progress, and identify areas for improvement in object classification, detection, segmentation, and action classification.

Main Contributions

Introduced novel evaluation methods: bootstrapping for significance, normalized average precision for cross-class comparison, and clustering for visualizing algorithm performance.
Provided a comprehensive review of the PASCAL VOC challenges from 2008-2012, covering classification, detection, segmentation, action classification, and person layout.
Analyzed the community's progress over time using error identification methods, highlighting both improvements and areas where progress was not monotonic.
Discussed lessons learned from organizing the challenge and offered recommendations for future challenge design.
Investigated the potential of 'super-classifiers' by combining submitted methods, showing significant performance gains and complementarity.

Abstract

The PASCAL Visual Object Classes (VOC) challenge consists of two components: (i) a publicly available dataset of images together with ground truth annotation and standardised evaluation software; and (ii) an annual competition and workshop. There are five challenges: classification, detection, segmentation, action classification, and person layout. In this paper we provide a review of the challenge from 2008-2012. The paper is intended for two audiences: algorithm designers, researchers who want to see what the state of the art is, as measured by performance on the VOC datasets, along with the limitations and weak points of the current generation of algorithms; and, challenge designers, who want to see what we as organisers have learnt from the process and our recommendations for the organisation of future challenges. To analyse the performance of submitted algorithms on the VOC datasets we introduce a number of novel evaluation methods: a bootstrapping method for determining whether differences in the performance of two algorithms are significant or not; a normalised average precision so that performance can be compared across classes with different proportions of positive instances; a clustering method for visualising the performance across multiple algorithms so that the hard and easy images can be identified; and the use of a joint classifier over the submitted algorithms in order to measure their complementarity and combined performance. We also analyse the community’s progress through time using the methods of Hoiem et al (2012) to identify the types of occurring errors. We conclude the paper with an appraisal of the aspects of the challenge that worked well, and those that could be improved in future challenges.

Citation Graph

Loading graph...

References [46]

Sort:

Filter:

[1]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite