 
Summary: Bin Model for Neural Networks
Yaser AbuMostafay, Xubo Songz
y Dept. of Electrical Engineering and Dept. of Computer Science
California Institute of Technology
Pasedena, CA 91125 USA
z Dept. of Electrical Engineering
California Institute of Technology
Pasedena, CA 91125 USA
Abstract We propose a theoretical framework for the modeling of learning machines,
such as neural networks, which we call the bin model, in which we consider a function as
a Bernoulli distribution. Using this model, we study the issues related to generalization,
such as the expected test error given a certain training error, and the expected test error
given the best training error. Noise in the data is also captured by the bin model, and the
effect of noise on generalization is quantified.
1 Introduction
In learning, what we have is a finite and often noisy data set. To approximate the target function
embodied in the data set, we choose a class of functions, e.g., the functions that can be implemented
by a feedforward neural network with certain architecture, as our candidates. Each candidate function
approximates the target function to certain precision. From the candidate functions, we will choose the
one that fits the data well according to some criterion. We have only access to training error. How well
