Recursive least-squares learning algorithms for neural networks
- Los Alamos National Lab., NM (USA)
- Washington Univ., Seattle, WA (USA). Dept. of Electrical Engineering
This paper presents the development of a pair of recursive least squares (RLS) algorithms for online training of multilayer perceptrons, which are a class of feedforward artificial neural networks. These algorithms incorporate second order information about the training error surface in order to achieve faster learning rates than are possible using first order gradient descent algorithms such as the generalized delta rule. A least squares formulation is derived from a linearization of the training error function. Individual training pattern errors are linearized about the network parameters that were in effect when the pattern was presented. This permits the recursive solution of the least squares approximation, either via conventional RLS recursions or by recursive QR decomposition-based techniques. The computational complexity of the update is in the order of (N{sup 2}), where N is the number of network parameters. This is due to the estimation of the N {times} N inverse Hessian matrix. Less computationally intensive approximations of the RLS algorithms can be easily derived by using only block diagonal elements of this matrix, thereby partitioning the learning into independent sets. A simulation example is presented in which a neural network is trained to approximate a two dimensional Gaussian bump. In this example, RLS training required an order of magnitude fewer iterations on average (527) than did training with the generalized delta rule (6331). 14 refs., 3 figs.
- Research Organization:
- Los Alamos National Lab., NM (USA)
- Sponsoring Organization:
- DOE/MA
- DOE Contract Number:
- W-7405-ENG-36
- OSTI ID:
- 6698032
- Report Number(s):
- LA-UR-90-2358; CONF-9007117--3; ON: DE90014926
- Country of Publication:
- United States
- Language:
- English
Similar Records
Dynamic gradient descent learning algorithms for enhanced empirical modeling of power plants
Discrete mathematics for computer scientists and mathematicians: 2nd ed