AI summary

This paper introduces a model compression technique that trains compact neural nets to mimic ensemble selection by labeling a large unlabeled dataset, or generating synthetic cases using MUNGE, to achieve comparable performance with significantly smaller and faster models.

Abstract

Often the best performing supervised learning models are ensembles of hundreds or thousands of base-level classifiers. Unfortunately, the space required to store this many classifiers, and the time required to execute them at run-time, prohibits their use in applications where test sets are large (e.g. Google), where storage space is at a premium (e.g. PDAs), and where computational power is limited (e.g. hearing aids). We present a method for "compressing" large, complex ensembles into smaller, faster models, usually without significant loss in performance.