Enhancing Streamflow Forecast and Extracting Insights Using Long-Short Term Memory Networks With Data Integration at Continental Scales
Abstract
Abstract Recent observations with varied schedules and types (moving average, snapshot, or regularly spaced) can help to improve streamflow forecasts, but it is challenging to integrate them effectively. Based on a long short‐term memory (LSTM) streamflow model, we tested multiple versions of a flexible procedure we call data integration (DI) to leverage recent discharge measurements to improve forecasts. DI accepts lagged inputs either directly or through a convolutional neural network unit. DI ubiquitously elevated streamflow forecast performance to unseen levels, reaching a record continental‐scale median Nash‐Sutcliffe Efficiency coefficient value of 0.86. Integrating moving‐average discharge, discharge from the last few days, or even average discharge from the previous calendar month could all improve daily forecasts. Directly using lagged observations as inputs was comparable in performance to using the convolutional neural network unit. Importantly, we obtained valuable insights regarding hydrologic processes impacting LSTM and DI performance. Before applying DI, the base LSTM model worked well in mountainous or snow‐dominated regions, but less well in regions with low discharge volumes (due to either low precipitation or high precipitation‐energy synchronicity) and large interannual storage variability. DI was most beneficial in regions with high flow autocorrelation: it greatly reduced baseflow bias in groundwater‐dominated western basinsmore »
- Authors:
-
[1];
[2];
[1]
- Pennsylvania State Univ., University Park, PA (United States)
- Pennsylvania State Univ., University Park, PA (United States); Stanford Univ., Stanford, CA (United States)
- Publication Date:
- Research Org.:
- Univ. of California, Davis, CA (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC)
- OSTI Identifier:
- 1803068
- Alternate Identifier(s):
- OSTI ID: 1664500
- Grant/Contract Number:
- SC0016605; DE‐SC0016605
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Water Resources Research
- Additional Journal Information:
- Journal Volume: 56; Journal Issue: 9; Journal ID: ISSN 0043-1397
- Publisher:
- American Geophysical Union (AGU)
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 54 ENVIRONMENTAL SCIENCES; Environmental Sciences & Ecology; Marine & Freshwater Biology; Water Resources
Citation Formats
Feng, Dapeng, Fang, Kuai, and Shen, Chaopeng. Enhancing Streamflow Forecast and Extracting Insights Using Long-Short Term Memory Networks With Data Integration at Continental Scales. United States: N. p., 2020.
Web. doi:10.1029/2019wr026793.
Feng, Dapeng, Fang, Kuai, & Shen, Chaopeng. Enhancing Streamflow Forecast and Extracting Insights Using Long-Short Term Memory Networks With Data Integration at Continental Scales. United States. https://doi.org/10.1029/2019wr026793
Feng, Dapeng, Fang, Kuai, and Shen, Chaopeng. Sun .
"Enhancing Streamflow Forecast and Extracting Insights Using Long-Short Term Memory Networks With Data Integration at Continental Scales". United States. https://doi.org/10.1029/2019wr026793. https://www.osti.gov/servlets/purl/1803068.
@article{osti_1803068,
title = {Enhancing Streamflow Forecast and Extracting Insights Using Long-Short Term Memory Networks With Data Integration at Continental Scales},
author = {Feng, Dapeng and Fang, Kuai and Shen, Chaopeng},
abstractNote = {Abstract Recent observations with varied schedules and types (moving average, snapshot, or regularly spaced) can help to improve streamflow forecasts, but it is challenging to integrate them effectively. Based on a long short‐term memory (LSTM) streamflow model, we tested multiple versions of a flexible procedure we call data integration (DI) to leverage recent discharge measurements to improve forecasts. DI accepts lagged inputs either directly or through a convolutional neural network unit. DI ubiquitously elevated streamflow forecast performance to unseen levels, reaching a record continental‐scale median Nash‐Sutcliffe Efficiency coefficient value of 0.86. Integrating moving‐average discharge, discharge from the last few days, or even average discharge from the previous calendar month could all improve daily forecasts. Directly using lagged observations as inputs was comparable in performance to using the convolutional neural network unit. Importantly, we obtained valuable insights regarding hydrologic processes impacting LSTM and DI performance. Before applying DI, the base LSTM model worked well in mountainous or snow‐dominated regions, but less well in regions with low discharge volumes (due to either low precipitation or high precipitation‐energy synchronicity) and large interannual storage variability. DI was most beneficial in regions with high flow autocorrelation: it greatly reduced baseflow bias in groundwater‐dominated western basins and also improved peak prediction for basins with dynamical surface water storage, such as the Prairie Potholes or Great Lakes regions. However, even DI cannot elevate performance in high‐aridity basins with 1‐day flash peaks. Despite this limitation, there is much promise for a deep‐learning‐based forecast paradigm due to its performance, automation, efficiency, and flexibility.},
doi = {10.1029/2019wr026793},
journal = {Water Resources Research},
number = 9,
volume = 56,
place = {United States},
year = {Sun Jun 21 00:00:00 EDT 2020},
month = {Sun Jun 21 00:00:00 EDT 2020}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
The Soil Moisture Active Passive (SMAP) Mission
journal, May 2010
- Entekhabi, Dara; Njoku, Eni G.; O'Neill, Peggy E.
- Proceedings of the IEEE, Vol. 98, Issue 5
Deep learning
journal, May 2015
- LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey
- Nature, Vol. 521, Issue 7553
Linear Inversion of Band-Limited Reflection Seismograms
journal, October 1986
- Santosa, Fadil; Symes, William W.
- SIAM Journal on Scientific and Statistical Computing, Vol. 7, Issue 4
Estimating Wetland Connectivity to Streams in the Prairie Pothole Region: An Isotopic and Remote Sensing Approach
journal, February 2018
- Brooks, J. R.; Mushet, D. M.; Vanderhoof, M. K.
- Water Resources Research, Vol. 54, Issue 2
Reliable long-range ensemble streamflow forecasts: Combining calibrated climate forecasts with a conceptual runoff model and a staged error model: LONG-RANGE ENSEMBLE STREAMFLOW FORECASTS
journal, October 2016
- Bennett, James C.; Wang, Q. J.; Li, Ming
- Water Resources Research, Vol. 52, Issue 10
Conceptual Framework for the National Flood Interoperability Experiment
journal, October 2016
- Maidment, David R.
- JAWRA Journal of the American Water Resources Association, Vol. 53, Issue 2
The fan of influence of streams and channel feedbacks to simulated land surface water and carbon dynamics: CHANNEL FEEDBACKS TO SIMULATED LAND SURFACE DYNAMICS
journal, February 2016
- Shen, Chaopeng; Riley, William J.; Smithgall, Kurt R.
- Water Resources Research, Vol. 52, Issue 2
The SWOT Mission and Its Capabilities for Land Hydrology
journal, October 2015
- Biancamaria, Sylvain; Lettenmaier, Dennis P.; Pavelsky, Tamlin M.
- Surveys in Geophysics, Vol. 37, Issue 2
Intermittent Surface Water Connectivity: Fill and Spill Vs. Fill and Merge Dynamics
journal, October 2016
- Leibowitz, Scott G.; Mushet, David M.; Newton, Wesley E.
- Wetlands, Vol. 36, Issue S2
A comparison of performance of several artificial intelligence methods for forecasting monthly discharge time series
journal, August 2009
- Wang, Wen-Chuan; Chau, Kwok-Wing; Cheng, Chun-Tian
- Journal of Hydrology, Vol. 374, Issue 3-4
Estimation of baseflow recession constants
journal, August 1996
- Vogel, Richard M.; Kroll, Charles N.
- Water Resources Management, Vol. 10, Issue 4
Assimilation of GRACE Terrestrial Water Storage Observations into a Land Surface Model for the Assessment of Regional Flood Potential
journal, November 2015
- Reager, John; Thomas, Alys; Sproles, Eric
- Remote Sensing, Vol. 7, Issue 11
A process-based, distributed hydrologic model based on a large-scale method for surface–subsurface coupling
journal, December 2010
- Shen, Chaopeng; Phanikumar, Mantha S.
- Advances in Water Resources, Vol. 33, Issue 12
Improving Budyko curve-based estimates of long-term water partitioning using hydrologic signatures from GRACE: PREDICTING DEPARTURE FROM BUDYKO USING GRACE
journal, July 2016
- Fang, Kuai; Shen, Chaopeng; Fisher, Joshua B.
- Water Resources Research, Vol. 52, Issue 7
HESS Opinions: Incubating deep-learning-powered hydrologic science advances as a community
journal, January 2018
- Shen, Chaopeng; Laloy, Eric; Elshorbagy, Amin
- Hydrology and Earth System Sciences, Vol. 22, Issue 11
A Transdisciplinary Review of Deep Learning Research and Its Relevance for Water Resources Scientists
journal, November 2018
- Shen, Chaopeng
- Water Resources Research, Vol. 54, Issue 11
Developing a Long Short-Term Memory (LSTM) based model for predicting water table depth in agricultural areas
journal, June 2018
- Zhang, Jianfeng; Zhu, Yan; Zhang, Xiaoping
- Journal of Hydrology, Vol. 561
Bias correction of simulated historical daily streamflow at ungauged locations by using independently estimated flow duration curves
journal, January 2018
- Farmer, William H.; Over, Thomas M.; Kiang, Julie E.
- Hydrology and Earth System Sciences, Vol. 22, Issue 11
A strategy to overcome adverse effects of autoregressive updating of streamflow forecasts
journal, January 2015
- Li, M.; Wang, Q. J.; Bennett, J. C.
- Hydrology and Earth System Sciences, Vol. 19, Issue 1
Analytical model of seasonal climate impacts on snow hydrology: Continuous snowpacks
journal, October 2009
- Woods, Ross A.
- Advances in Water Resources, Vol. 32, Issue 10
Integration of soil moisture remote sensing and hydrologic modeling using data assimilation
journal, December 1998
- Houser, Paul R.; Shuttleworth, W. James; Famiglietti, James S.
- Water Resources Research, Vol. 34, Issue 12
Toward improved identification of hydrological models: A diagnostic evaluation of the “ abcd ” monthly water balance model for the conterminous United States : DIAGNOSTIC EVALUATION OF “
journal, August 2010
- Martinez, Guillermo F.; Gupta, Hoshin V.
- Water Resources Research, Vol. 46, Issue 8
Long Short-Term Memory
journal, November 1997
- Hochreiter, Sepp; Schmidhuber, Jürgen
- Neural Computation, Vol. 9, Issue 8
Assimilation of GRACE Terrestrial Water Storage Data into a Land Surface Model: Results for the Mississippi River Basin
journal, June 2008
- Zaitchik, Benjamin F.; Rodell, Matthew; Reichle, Rolf H.
- Journal of Hydrometeorology, Vol. 9, Issue 3
Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling
journal, October 2009
- Gupta, Hoshin V.; Kling, Harald; Yilmaz, Koray K.
- Journal of Hydrology, Vol. 377, Issue 1-2
A process-based diagnostic approach to model evaluation: Application to the NWS distributed hydrologic model: PROCESS-BASED DIAGNOSTIC EVALUATION OF HYDROLOGIC MODEL
journal, September 2008
- Yilmaz, Koray K.; Gupta, Hoshin V.; Wagener, Thorsten
- Water Resources Research, Vol. 44, Issue 9
Quantifying storage changes in regional Great Lakes watersheds using a coupled subsurface-land surface process model and GRACE, MODIS products
journal, September 2014
- Niu, Jie; Shen, Chaopeng; Li, Shu-Guang
- Water Resources Research, Vol. 50, Issue 9
Integrating different types of information into hydrological model parameter estimation: Application to ungauged catchments and land use scenario analysis: INTEGRATING DIFFERENT TYPES OF INFORMATION
journal, June 2012
- Bulygina, Nataliya; Ballard, Caroline; McIntyre, Neil
- Water Resources Research, Vol. 48, Issue 6
Long short term memory (LSTM) recurrent neural network (RNN) for discharge level prediction and forecast in Cimandiri river, Indonesia
journal, July 2019
- Sudriani, Y.; Ridwansyah, I.; A. Rustini, H.
- IOP Conference Series: Earth and Environmental Science, Vol. 299
Development of a large-sample watershed-scale hydrometeorological data set for the contiguous USA: data set characteristics and assessment of regional variability in hydrologic model performance
journal, January 2015
- Newman, A. J.; Clark, M. P.; Sampson, K.
- Hydrology and Earth System Sciences, Vol. 19, Issue 1
Quantifying the effects of data integration algorithms on the outcomes of a subsurface–land surface processes model
journal, September 2014
- Shen, Chaopeng; Niu, Jie; Fang, Kuai
- Environmental Modelling & Software, Vol. 59
Near-Real-Time Forecast of Satellite-Based Soil Moisture Using Long Short-Term Memory with an Adaptive Data Integration Kernel
journal, March 2019
- Fang, Kuai; Shen, Chaopeng
- Journal of Hydrometeorology, Vol. 21, Issue 3
Evaluating controls on coupled hydrologic and vegetation dynamics in a humid continental climate watershed using a subsurface-land surface processes model: EVALUATING CONTROLS ON HYDROLOGIC AND VEGETATION DYNAMICS
journal, May 2013
- Shen, Chaopeng; Niu, Jie; Phanikumar, Mantha S.
- Water Resources Research, Vol. 49, Issue 5
The Value of SMAP for Long-Term Soil Moisture Estimation With the Help of Deep Learning
journal, April 2019
- Fang, Kuai; Pan, Ming; Shen, Chaopeng
- IEEE Transactions on Geoscience and Remote Sensing, Vol. 57, Issue 4
Real-Time Data Assimilation for Operational Ensemble Streamflow Forecasting
journal, June 2006
- Vrugt, Jasper A.; Gupta, Hoshin V.; Nualláin, BreanndánÓ
- Journal of Hydrometeorology, Vol. 7, Issue 3
Prolongation of SMAP to Spatiotemporally Seamless Coverage of Continental U.S. Using a Deep Learning Neural Network
journal, November 2017
- Fang, Kuai; Shen, Chaopeng; Kifer, Daniel
- Geophysical Research Letters, Vol. 44, Issue 21
Correcting Errors in Streamflow Forecast Ensemble Mean and Spread
journal, February 2008
- Wood, Andrew W.; Schaake, John C.
- Journal of Hydrometeorology, Vol. 9, Issue 1
Patterns of similarity of seasonal water balances: A window into streamflow variability over a range of time scales
journal, July 2014
- Berghuijs, Wouter R.; Sivapalan, Murugesu; Woods, Ross A.
- Water Resources Research, Vol. 50, Issue 7
The quantity and quality of information in hydrologic models
journal, January 2015
- Nearing, Grey S.; Gupta, Hoshin V.
- Water Resources Research, Vol. 51, Issue 1
Physics Guided RNNs for Modeling Dynamical Systems: A Case Study in Simulating Lake Temperature Profiles
book, May 2019
- Jia, Xiaowei; Willard, Jared; Karpatne, Anuj
- Proceedings of the 2019 SIAM International Conference on Data Mining
Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data
journal, October 2017
- Karpatne, Anuj; Atluri, Gowtham; Faghmous, James H.
- IEEE Transactions on Knowledge and Data Engineering, Vol. 29, Issue 10
Deep Learning with a Long Short-Term Memory Networks Approach for Rainfall-Runoff Simulation
journal, October 2018
- Hu, Caihong; Wu, Qiang; Li, Hui
- Water, Vol. 10, Issue 11
Rainfall–runoff modelling using Long Short-Term Memory (LSTM) networks
journal, January 2018
- Kratzert, Frederik; Klotz, Daniel; Brenner, Claire
- Hydrology and Earth System Sciences, Vol. 22, Issue 11
A Multiscale Ensemble Filtering System for Hydrologic Data Assimilation. Part I: Implementation and Synthetic Experiment
journal, June 2009
- Pan, Ming; Wood, Eric F.; McLaughlin, Dennis B.
- Journal of Hydrometeorology, Vol. 10, Issue 3
River flow forecasting through conceptual models part I — A discussion of principles
journal, April 1970
- Nash, J. E.; Sutcliffe, J. V.
- Journal of Hydrology, Vol. 10, Issue 3
Temporal connectivity in a prairie pothole complex
journal, March 2003
- Leibowitz, Scott G.; Vining, Kevin C.
- Wetlands, Vol. 23, Issue 1
Not higher but more often
journal, February 2015
- Hirsch, Robert M.; Archfield, Stacey A.
- Nature Climate Change, Vol. 5, Issue 3
Benchmarking of a Physically Based Hydrologic Model
journal, August 2017
- Newman, Andrew J.; Mizukami, Naoki; Clark, Martyn P.
- Journal of Hydrometeorology, Vol. 18, Issue 8
Real-Time Flood Forecasting and Information System for the State of Iowa
journal, March 2017
- Krajewski, Witold F.; Ceynar, Daniel; Demir, Ibrahim
- Bulletin of the American Meteorological Society, Vol. 98, Issue 3
A standard approach to baseflow separation using the Lyne and Hollick filter
journal, January 2013
- Ladson, Ar; Brown, R.; Neal, B.
- Australian Journal of Water Resources, Vol. 17, Issue 1
Climate, soil water storage, and the average annual water balance
journal, July 1994
- Milly, P. C. D.
- Water Resources Research, Vol. 30, Issue 7
Application of Long Short-Term Memory (LSTM) Neural Network for Flood Forecasting
journal, July 2019
- Le, Xuan-Hien; Ho, Hung Viet; Lee, Giha
- Water, Vol. 11, Issue 7, p. 1387
Deep learning in neural networks: An overview
journal, January 2015
- Schmidhuber, Jürgen
- Neural Networks, Vol. 61
Hydrological data assimilation with the ensemble Kalman filter: Use of streamflow observations to update states in a distributed hydrological model
journal, October 2008
- Clark, Martyn P.; Rupp, David E.; Woods, Ross A.
- Advances in Water Resources, Vol. 31, Issue 10
Benchmarking a Catchment-Aware Long Short-Term Memory Network (LSTM) for Large-Scale Hydrological Modeling
text, January 2019
- Kratzert, Frederik; Klotz, Daniel; Shalev, Guy
- Unpublished
Rainfall-Runoff modelling using Long-Short-Term-Memory (LSTM) networks
text, January 2018
- Kratzert, Frederik; Klotz, Daniel; Brenner, Claire
- EarthArXiv
Physics Guided Recurrent Neural Networks For Modeling Dynamical Systems: Application to Monitoring Water Temperature And Quality In Lakes
preprint, January 2018
- Jia, Xiaowei; Karpatne, Anuj; Willard, Jared
- arXiv
NeuralHydrology -- Interpreting LSTMs in Hydrology
text, January 2019
- Kratzert, Frederik; Herrnegger, Mathew; Klotz, Daniel
- arXiv