Online evolutionary neural architecture search for multivariate non-stationary time series forecasting

Lyu, Zimeng; Ororbia, Alexander; Desell, Travis

doi:10.1016/j.asoc.2023.110522

Online evolutionary neural architecture search for multivariate non-stationary time series forecasting

Journal Article · Thu Jun 15 00:00:00 EDT 2023 · Applied Soft Computing

DOI:https://doi.org/10.1016/j.asoc.2023.110522· OSTI ID:2422444

^[1]; Ororbia, Alexander ^[2]; Desell, Travis ^[2]

Rochester Inst. of Technology, Rochester, NY (United States); OSTI
Rochester Inst. of Technology, Rochester, NY (United States)

Time series forecasting (TSF) is one of the most important tasks in data science. TSF models are usually pre-trained with historical data and then applied on future unseen datapoints. However, real-world time series data is usually non-stationary and models trained offline usually face problems from data drift. Models trained and designed in an offline fashion can not quickly adapt to changes quickly or be deployed in real-time. To address these issues, this work presents the Online NeuroEvolution-based Neural Architecture Search (ONE-NAS) algorithm, which is a novel neural architecture search method capable of automatically designing and dynamically training recurrent neural networks (RNNs) for online forecasting tasks. Without any pre-training, ONE-NAS utilizes populations of RNNs that are continuously updated with new network structures and weights in response to new multivariate input data. ONE-NAS is tested on real-world, large-scale multivariate wind turbine data as well as the univariate Dow Jones Industrial Average (DJIA) dataset. These results demonstrate that ONE-NAS outperforms traditional statistical time series forecasting methods, including online linear regression, fixed long short-term memory (LSTM) and gated recurrent unit (GRU) models trained online, as well as state-of-the-art, online ARIMA strategies. Additionally, results show that utilizing multiple populations of RNNs which are periodically repopulated provide significant performance improvements, allowing this online neural network architecture design and training to be successful.

View Accepted Manuscript (DOE)

Research Organization:: Microbeam Technologies, Inc., Grand Forks, ND (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE Office of Fossil Energy (FE)

Grant/Contract Number:: FE0031547

OSTI ID:: 2422444

Journal Information:: Applied Soft Computing, Journal Name: Applied Soft Computing Journal Issue: C Vol. 145; ISSN 1568-4946

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (30)

Exponential smoothing: The state of the art Gardner, Everette S. Journal of Forecasting, Vol. 4, Issue 1 https://doi.org/10.1002/for.3980040103	journal	January 1985
Minimal gated unit for recurrent neural networks Zhou, Guo-Bing; Wu, Jianxin; Zhang, Chen-Lin International Journal of Automation and Computing, Vol. 13, Issue 3 https://doi.org/10.1007/s11633-016-1006-2	journal	June 2016
Catastrophic forgetting in connectionist networks French, R. Trends in Cognitive Sciences, Vol. 3, Issue 4 https://doi.org/10.1016/S1364-6613(99)01294-2	journal	April 1999
Time series forecasting of COVID-19 transmission in Canada using LSTM networks Chimmula, Vinay Kumar Reddy; Zhang, Lei Chaos, Solitons & Fractals, Vol. 135 https://doi.org/10.1016/j.chaos.2020.109864	journal	June 2020
Application of the ARIMA model on the COVID-2019 epidemic dataset Benvenuto, Domenico; Giovanetti, Marta; Vassallo, Lazzaro Data in Brief, Vol. 29 https://doi.org/10.1016/j.dib.2020.105340	journal	April 2020
Online sequential extreme learning machine with kernels for nonstationary time series prediction Wang, Xinying; Han, Min Neurocomputing, Vol. 145 https://doi.org/10.1016/j.neucom.2014.05.068	journal	December 2014
Online learning: A comprehensive survey Hoi, Steven C. H.; Sahoo, Doyen; Lu, Jing Neurocomputing, Vol. 459 https://doi.org/10.1016/j.neucom.2021.04.112	journal	October 2021
Financial time series forecasting model based on CEEMDAN and LSTM Cao, Jian; Li, Zhi; Li, Jian Physica A: Statistical Mechanics and its Applications, Vol. 519 https://doi.org/10.1016/j.physa.2018.11.061	journal	April 2019
Estimating Properties of Autoregressive Forecasts Stine, Robert A. Journal of the American Statistical Association, Vol. 82, Issue 400 https://doi.org/10.1080/01621459.1987.10478542	journal	December 1987
Modelling and forecasting daily surgical case volume using time series analysis Zinouri, Nazanin; Taaffe, Kevin M.; Neyens, David M. Health Systems, Vol. 7, Issue 2 https://doi.org/10.1080/20476965.2017.1390185	journal	January 2018
State of Health Estimation for Lithium-ion Batteries Based on Fusion of Autoregressive Moving Average Model and Elman Neural Network Chen, Zheng; Xue, Qiao; Xiao, Renxin IEEE Access, Vol. 7 https://doi.org/10.1109/ACCESS.2019.2930680	journal	January 2019
Automated Machine Learning for COVID-19 Forecasting Tetteroo, Jaco; Baratchi, Mitra; Hoos, Holger H. IEEE Access, Vol. 10 https://doi.org/10.1109/ACCESS.2022.3202220	journal	January 2022
Learning to Drive in the Open Racing Car Simulator Using Online Neuroevolution Cardamone, Luigi; Loiacono, Daniele; Lanzi, Pier Luca IEEE Transactions on Computational Intelligence and AI in Games, Vol. 2, Issue 3 https://doi.org/10.1109/TCIAIG.2010.2052102	journal	September 2010
Multivariate Chaotic Time Series Online Prediction Based on Improved Kernel Recursive Least Squares Algorithm Han, Min; Zhang, Shuhui; Xu, Meiling IEEE Transactions on Cybernetics, Vol. 49, Issue 4 https://doi.org/10.1109/TCYB.2018.2789686	journal	April 2019
Real-Time Neuroevolution in the NERO Video Game Stanley, K. O.; Bryant, B. D.; Miikkulainen, R. IEEE Transactions on Evolutionary Computation, Vol. 9, Issue 6 https://doi.org/10.1109/TEVC.2005.856210	journal	December 2005
Online Incremental Machine Learning Platform for Big Data-Driven Smart Traffic Management Nallaperuma, Dinithi; Nawaratne, Rashmika; Bandaragoda, Tharindu IEEE Transactions on Intelligent Transportation Systems, Vol. 20, Issue 12 https://doi.org/10.1109/TITS.2019.2924883	journal	December 2019
Adaptation Strategies for Automated Machine Learning on Evolving Data Celik, Bilge; Vanschoren, Joaquin IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43, Issue 9 https://doi.org/10.1109/TPAMI.2021.3062900	journal	September 2021
Modeling and forecasting pelagic fish production using univariate and multivariate ARIMA models Tsitsika, Efthymia V.; Maravelias, Christos D.; Haralabous, John Fisheries Science, Vol. 73, Issue 5 https://doi.org/10.1111/j.1444-2906.2007.01426.x	journal	October 2007
Large Scale Online Multiple Kernel Regression with Application to Time-Series Prediction Sahoo, Doyen; Hoi, Steven C. H.; Li, Bin ACM Transactions on Knowledge Discovery from Data, Vol. 13, Issue 1 https://doi.org/10.1145/3299875	journal	January 2019
Investigating recurrent neural network memory structures using neuro-evolution Ororbia, Alexander; ElSaid, AbdElRahman; Desell, Travis GECCO '19: Genetic and Evolutionary Computation Conference, Proceedings of the Genetic and Evolutionary Computation Conference https://doi.org/10.1145/3321707.3321795	conference	July 2019
Improving neuroevolutionary transfer learning of deep recurrent neural networks through network-aware adaptation ElSaid, AbdElRahman; Karns, Joshua; Lyu, Zimeng Proceedings of the 2020 Genetic and Evolutionary Computation Conference https://doi.org/10.1145/3377930.3390193	conference	June 2020
Neuroevolution of recurrent neural networks for time series forecasting of coal-fired power plant operating parameters Lyu, Zimeng; Patwardhan, Shuchita; Stadem, David Proceedings of the Genetic and Evolutionary Computation Conference Companion https://doi.org/10.1145/3449726.3463196	conference	July 2021
Robust and Adaptive Online Time Series Prediction with Long Short-Term Memory Yang, Haimin; Pan, Zhisong; Tao, Qing Computational Intelligence and Neuroscience, Vol. 2017 https://doi.org/10.1155/2017/9478952	journal	January 2017
Evolving Neural Networks through Augmenting Topologies Stanley, Kenneth O.; Miikkulainen, Risto Evolutionary Computation, Vol. 10, Issue 2 https://doi.org/10.1162/106365602320169811	journal	June 2002
Training with Noise is Equivalent to Tikhonov Regularization Bishop, Chris M. Neural Computation, Vol. 7, Issue 1 https://doi.org/10.1162/neco.1995.7.1.108	journal	January 1995
Long Short-Term Memory Hochreiter, Sepp; Schmidhuber, Jürgen Neural Computation, Vol. 9, Issue 8 https://doi.org/10.1162/neco.1997.9.8.1735	journal	November 1997
Statistical and Machine Learning forecasting methods: Concerns and ways forward Makridakis, Spyros; Spiliotis, Evangelos; Assimakopoulos, Vassilios PLOS ONE, Vol. 13, Issue 3 https://doi.org/10.1371/journal.pone.0194889	journal	March 2018
Online ARIMA Algorithms for Time Series Prediction Liu, Chenghao; Hoi, Steven C. H.; Zhao, Peilin Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30, Issue 1 https://doi.org/10.1609/aaai.v30i1.10257	journal	February 2016
Online Forecasting and Anomaly Detection Based on the ARIMA Model Kozitsin, Viacheslav; Katser, Iurii; Lakontsev, Dmitry Applied Sciences, Vol. 11, Issue 7 https://doi.org/10.3390/app11073194	journal	April 2021
Time Series Data Modeling Using Advanced Machine Learning and AutoML Alsharef, Ahmad; Sonia, Sonia; Kumar, Karan Sustainability, Vol. 14, Issue 22 https://doi.org/10.3390/su142215292	journal	November 2022

Similar Records

Predictive Skill of Deep Learning Models Trained on Limited Sequence Data

Technical Report · Thu Oct 01 00:00:00 EDT 2020 · OSTI ID:1688570

The predictive skill of convolutional neural networks models for disease forecasting

Journal Article · Thu Jul 08 20:00:00 EDT 2021 · PLoS ONE · OSTI ID:1806600

Related Subjects

97 MATHEMATICS AND COMPUTING
Computer Science
Neural architecture search
Neuroevolution
Online learning
Recurrent neural networks
Time series forecasting

Online evolutionary neural architecture search for multivariate non-stationary time series forecasting

Citation Formats

References (30)

Similar Records

Related Subjects