Summary: LETTER Communicated by David Wolpert
No Free Lunch for Early Stopping
Yaser S. Abu-Mostafa
Bell Laboratories, Lucent Technologies, 600 Mountain Ave., Rm. 2C265, Murray Hill,
NJ 07974, U.S.A.
Learning Systems Group, California Institute of Technology, MC 13693, Pasadena,
CA 91125, U.S.A.
We show that with a uniform prior on models having the same training
error, early stopping at some fixed training error above the training error
minimum results in an increase in the expected generalization error.
Early stopping of training is one of the methods that aim to prevent over-
training due to too powerful a model class, noisy training examples, or a
small training set. We study early stopping at a predetermined training er-
ror level. If there is no prior information other than the training examples,
all models with the same training error should be equally likely to be cho-
sen as the early stopping solution. When this is the case, we show that for
general linear models, early stopping at any training error level above the