 
Summary: Ordering and Finding the Best of K > 2
Supervised Learning Algorithms
Olcay Taner Yildiz and Ethem Alpaydin, Senior Member, IEEE
AbstractGiven a data set and a number of supervised learning algorithms, we would like to find the algorithm with the smallest
expected error. Existing pairwise tests allow a comparison of two algorithms only; range tests and ANOVA check whether multiple
algorithms have the same expected error and cannot be used for finding the smallest. We propose a methodology, the MultiTest
algorithm, whereby we order supervised learning algorithms taking into account 1) the result of pairwise statistical tests on expected
error (what the data tells us), and 2) our prior preferences, e.g., due to complexity. We define the problem in graphtheoretic terms and
propose an algorithm to find the "best" learning algorithm in terms of these two criteria, or in the more general case, order learning
algorithms in terms of their "goodness." Simulation results using five classification algorithms on 30 data sets indicate the utility of the
method. Our proposed method can be generalized to regression and other loss functions by using a suitable pairwise test.
Index TermsMachine learning, classifier design and evaluation, experimental design.
æ
1 INTRODUCTION
IN machine learning literature, there exist several super
vised learning algorithms, and for any application, we
need to find the algorithm that generalizes the best, i.e., the
one with the smallest probability of misclassifying an
instance unseen during training. To check for a statistically
significant difference, the algorithms are trained and tested
