Sciweavers

AUSAI
2003
Springer

Choosing Learning Algorithms Using Sign Tests with High Replicability

13 years 9 months ago
Choosing Learning Algorithms Using Sign Tests with High Replicability
An important task in machine learning is determining which learning algorithm works best for a given data set. When the amount of data is small the same data needs to be used repeatedly in order to get a reasonable estimate of the accuracy of the learning algorithms. This results in violations of assumptions on which standard tests are based and makes it hard to design a good test. In this article, we investigate sign tests to address the problem of choosing the best of two learning algorithms when only a small data set is available. Sign tests are conceptually simple and no assumption about underlying distributions is required. We show that simplistic sample generation can lead to flawed test outcomes. Furthermore, we identify a test that performs well based on Type I error (showing a difference between algorithms when there is none), power (showing a difference when it indeed exists) and replicability. Replicability is a novel measure of a quality of a test that gives an indicatio...
Remco R. Bouckaert
Added 06 Jul 2010
Updated 06 Jul 2010
Type Conference
Year 2003
Where AUSAI
Authors Remco R. Bouckaert
Comments (0)