skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Learning rate and attractor size of the single-layer perceptron

Abstract

We study the simplest possible order one single-layer perceptron with two inputs, using the delta rule with online learning, in order to derive closed form expressions for the mean convergence rates. We investigate the rate of convergence in weight space of the weight vectors corresponding to each of the 14 out of 16 linearly separable rules. These vectors follow zigzagging lines through the piecewise constant vector field to their respective attractors. Based on our studies, we conclude that a single-layer perceptron with N inputs will converge in an average number of steps given by an Nth order polynomial in (t/l), where t is the threshold, and l is the size of the initial weight distribution. Exact values for these averages are provided for the five linearly separable classes with N=2. We also demonstrate that the learning rate is determined by the attractor size, and that the attractors of a single-layer perceptron with N inputs partition R{sup N}+R{sup N}.

Authors:
;  [1];  [2]
  1. Department of Mathematics, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801 (United States)
  2. (United States)
Publication Date:
OSTI Identifier:
21072391
Resource Type:
Journal Article
Resource Relation:
Journal Name: Physical Review. E, Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics; Journal Volume: 75; Journal Issue: 2; Other Information: DOI: 10.1103/PhysRevE.75.026704; (c) 2007 The American Physical Society; Country of input: International Atomic Energy Agency (IAEA)
Country of Publication:
United States
Language:
English
Subject:
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS; ATTRACTORS; CONVERGENCE; DISTRIBUTION; LAYERS; LEARNING; POLYNOMIALS; VECTOR FIELDS; VECTORS

Citation Formats

Singleton, Martin S., Huebler, Alfred W., and Department of Physics, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801. Learning rate and attractor size of the single-layer perceptron. United States: N. p., 2007. Web. doi:10.1103/PHYSREVE.75.026704.
Singleton, Martin S., Huebler, Alfred W., & Department of Physics, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801. Learning rate and attractor size of the single-layer perceptron. United States. doi:10.1103/PHYSREVE.75.026704.
Singleton, Martin S., Huebler, Alfred W., and Department of Physics, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801. Thu . "Learning rate and attractor size of the single-layer perceptron". United States. doi:10.1103/PHYSREVE.75.026704.
@article{osti_21072391,
title = {Learning rate and attractor size of the single-layer perceptron},
author = {Singleton, Martin S. and Huebler, Alfred W. and Department of Physics, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801},
abstractNote = {We study the simplest possible order one single-layer perceptron with two inputs, using the delta rule with online learning, in order to derive closed form expressions for the mean convergence rates. We investigate the rate of convergence in weight space of the weight vectors corresponding to each of the 14 out of 16 linearly separable rules. These vectors follow zigzagging lines through the piecewise constant vector field to their respective attractors. Based on our studies, we conclude that a single-layer perceptron with N inputs will converge in an average number of steps given by an Nth order polynomial in (t/l), where t is the threshold, and l is the size of the initial weight distribution. Exact values for these averages are provided for the five linearly separable classes with N=2. We also demonstrate that the learning rate is determined by the attractor size, and that the attractors of a single-layer perceptron with N inputs partition R{sup N}+R{sup N}.},
doi = {10.1103/PHYSREVE.75.026704},
journal = {Physical Review. E, Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics},
number = 2,
volume = 75,
place = {United States},
year = {Thu Feb 15 00:00:00 EST 2007},
month = {Thu Feb 15 00:00:00 EST 2007}
}
  • A single-layer perceptron divides the input signal space into two regions separated by a hyperplane. In many applications, however, the training signal of the adaptive algorithm represents more complicated decision regions that may not be linearly separable. For these cases, it is usually not obvious how the adaptive algorithm of a single-layer perceptron will perform, in terms of its convergence properties and the optimum location of the hyperplane boundary. In this paper, the authors determine the stationary points of Rosenblatt's learning algorithm for a single-layer perceptron and two nonseparable models of the training data. The analysis is based on amore » system identification formulation of the training signal, and the perceptron input signals are modeled as independent Gaussian sequences. Expressions for the corresponding performance function are also derived, and computer simulations are presented that verify the analytical results.« less
  • An accelerated learning algorithm (ABP--adaptive back propagation) is proposed for the supervised training of multilayer perceptron networks. The learning algorithm is inspired from the principle of forced dynamics'' for the total error functional. The algorithm updates the weights in the direction of steepest descent, but with a learning rate a specific function of the error and of the error gradient norm. This specific form of this function is chosen such as to accelerate convergence. Furthermore, ABP introduces no additional tuning'' parameters found in variants of the backpropagation algorithm. Simulation results indicate a superior convergence speed for analog problems only, asmore » compared to other competing methods, as well as reduced sensitivity to algorithm step size parameter variations.« less
  • A neural network system, utilizing the multi-layer perceptron approach has been applied to distinguish between partial discharge pulse shapes that arc characteristic of cavities and electrical trees. The neural network was found to be capable of recognizing the differences in PD Pulses produced by single cavity and electrical tree discharge sources. It also could differentiate between the discharge pulse forms emanating from electrical trees of different lengths; likewise, it was able to recognize changes in the shape of the discharge pulses with time due to aging effects. However, as these recognition capabilities relate only to comparisons of single discharge sourcesmore » on a one-to-one basis, the application of neural networks to PD pulse shape recognition on actual cables, where a number of different discharge sources may be discharging simultaneously, is quite premature at this time without more detailed exploratory work on complex discharge patterns.« less
  • A feed-forward three-layer perceptron was trained by high impedance fault, fault-like load, and normal load current patterns, using the back-propagation training algorithm. This paper reports that the neural network parameters were embodied in a high impedance arcing faults detection algorithm, which uses a simple preprocessing technique to prepare the information input to the network. The algorithm was tested by traces of normal load current disturbed by currents of faults on dry and wet soil, an arc welder, computers, and fluorescent lights.
  • Purpose: Dynamic tracking of moving organs, such as lung and liver tumors, under radiation therapy requires prediction of organ motions prior to delivery. The shift of moving organ may change a lot due to huge transform of respiration at different periods. This study aims to reduce the influence of that changes using adjustable training signals and multi-layer perceptron neural network (ASMLP). Methods: Respiratory signals obtained using a Real-time Position Management(RPM) device were used for this study. The ASMLP uses two multi-layer perceptron neural networks(MLPs) to infer respiration position alternately and the training sample will be updated with time. Firstly, amore » Savitzky-Golay finite impulse response smoothing filter was established to smooth the respiratory signal. Secondly, two same MLPs were developed to estimate respiratory position from its previous positions separately. Weights and thresholds were updated to minimize network errors according to Leverberg-Marquart optimization algorithm through backward propagation method. Finally, MLP 1 was used to predict 120∼150s respiration position using 0∼120s training signals. At the same time, MLP 2 was trained using 30∼150s training signals. Then MLP is used to predict 150∼180s training signals according to 30∼150s training signals. The respiration position is predicted as this way until it was finished. Results: In this experiment, the two methods were used to predict 2.5 minute respiratory signals. For predicting 1s ahead of response time, correlation coefficient was improved from 0.8250(MLP method) to 0.8856(ASMLP method). Besides, a 30% improvement of mean absolute error between MLP(0.1798 on average) and ASMLP(0.1267 on average) was achieved. For predicting 2s ahead of response time, correlation coefficient was improved from 0.61415 to 0.7098.Mean absolute error of MLP method(0.3111 on average) was reduced by 35% using ASMLP method(0.2020 on average). Conclusion: The preliminary results demonstrate that the ASMLP respiratory prediction method is more accurate than MLP method and can improve the respiration forecast accuracy.« less