Near-Real-Time Forecast of Satellite-Based Soil Moisture Using Long Short-Term Memory with an Adaptive Data Integration Kernel

Fang, Kuai; Shen, Chaopeng

doi:10.1175/JHM-D-19-0169.1

Title: Near-Real-Time Forecast of Satellite-Based Soil Moisture Using Long Short-Term Memory with an Adaptive Data Integration Kernel

Full Record
Other Related Research

Abstract

Nowcasts, or near-real-time (NRT) forecasts, of soil moisture based on the Soil Moisture Active and Passive (SMAP) mission could provide substantial value for a range of applications including hazards monitoring and agricultural planning. To provide such a NRT forecast with high fidelity, we enhanced a time series deep learning architecture, long short-term memory (LSTM), with a novel data integration (DI) kernel to assimilate the most recent SMAP observations as soon as they become available. The kernel is adaptive in that it can accommodate irregular observational schedules. Testing over the CONUS, this NRT forecast product showcases predictions with unprecedented accuracy when evaluated against subsequent SMAP retrievals. It showed smaller error than NRT forecasts reported in the literature, especially at longer forecast latency. The comparative advantage was due to LSTM’s structural improvements, as well as its ability to utilize more input variables and more training data. The DI-LSTM was compared to the original LSTM model that runs without data integration, referred to as the projection model here. We found that the DI procedure removed the autocorrelated effects of forcing errors and errors due to processes not represented in the inputs, for example, irrigation and floodplain/lake inundation, as well as mismatches due tomore »« less

Authors:

Fang, Kuai ^[1];

^[1]

Department of Civil and Environmental Engineering, The Pennsylvania State University, University Park, Pennsylvania

Publication Date:: Sun Mar 01 00:00:00 EST 2020

Research Org.:: Pennsylvania State Univ., University Park, PA (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Biological and Environmental Research (BER); National Science Foundation (NSF)

OSTI Identifier:: 1602597

Alternate Identifier(s):: OSTI ID: 1593951

Grant/Contract Number:: SC0016605; 1832294

Resource Type:: Published Article

Journal Name:: Journal of Hydrometeorology

Additional Journal Information:: Journal Name: Journal of Hydrometeorology Journal Volume: 21 Journal Issue: 3; Journal ID: ISSN 1525-755X

Publisher:: American Meteorological Society

Country of Publication:: United States

Language:: English

Subject:: 54 ENVIRONMENTAL SCIENCES

Citation Formats


                    Fang, Kuai, and Shen, Chaopeng. Near-Real-Time Forecast of Satellite-Based Soil Moisture Using Long Short-Term Memory with an Adaptive Data Integration Kernel.  United States: N. p., 2020. 
Web.  doi:10.1175/JHM-D-19-0169.1.

Copy to clipboard


                    Fang, Kuai, & Shen, Chaopeng. Near-Real-Time Forecast of Satellite-Based Soil Moisture Using Long Short-Term Memory with an Adaptive Data Integration Kernel.  United States.  https://doi.org/10.1175/JHM-D-19-0169.1

Copy to clipboard


                    Fang, Kuai, and Shen, Chaopeng. Sun .  
"Near-Real-Time Forecast of Satellite-Based Soil Moisture Using Long Short-Term Memory with an Adaptive Data Integration Kernel".  United States.  https://doi.org/10.1175/JHM-D-19-0169.1.

Copy to clipboard


                    
@article{osti_1602597,

  title        = {Near-Real-Time Forecast of Satellite-Based Soil Moisture Using Long Short-Term Memory with an Adaptive Data Integration Kernel},

  author       = {Fang, Kuai and Shen, Chaopeng},

  abstractNote = {Nowcasts, or near-real-time (NRT) forecasts, of soil moisture based on the Soil Moisture Active and Passive (SMAP) mission could provide substantial value for a range of applications including hazards monitoring and agricultural planning. To provide such a NRT forecast with high fidelity, we enhanced a time series deep learning architecture, long short-term memory (LSTM), with a novel data integration (DI) kernel to assimilate the most recent SMAP observations as soon as they become available. The kernel is adaptive in that it can accommodate irregular observational schedules. Testing over the CONUS, this NRT forecast product showcases predictions with unprecedented accuracy when evaluated against subsequent SMAP retrievals. It showed smaller error than NRT forecasts reported in the literature, especially at longer forecast latency. The comparative advantage was due to LSTM’s structural improvements, as well as its ability to utilize more input variables and more training data. The DI-LSTM was compared to the original LSTM model that runs without data integration, referred to as the projection model here. We found that the DI procedure removed the autocorrelated effects of forcing errors and errors due to processes not represented in the inputs, for example, irrigation and floodplain/lake inundation, as well as mismatches due to unseen forcing conditions. The effects of this purely data-driven DI kernel are discussed for the first time in the geosciences. Furthermore, this work presents an upper-bound estimate for the random component of the SMAP retrieval error.},

  doi          = {10.1175/JHM-D-19-0169.1},

  journal      = {Journal of Hydrometeorology},

  number       = 3,

  volume       = 21,

  place        = {United States},

  year         = {Sun Mar 01 00:00:00 EST 2020},

  month        = {Sun Mar 01 00:00:00 EST 2020}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Publisher's Version of Record
https://doi.org/10.1175/JHM-D-19-0169.1

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 39 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

Evaluating the Potential and Challenges of an Uncertainty Quantification Method for Long Short–Term Memory Models for Soil Moisture Predictions

Journal Article Fang, Kuai ; Kifer, Daniel ; Lawson, Kathryn ; ... - Water Resources Research

Abstract Recently, recurrent deep networks have shown promise to harness newly available satellite‐sensed data for long‐term soil moisture projections. However, to be useful in forecasting, deep networks must also provide uncertainty estimates. Here we evaluated Monte Carlo dropout with an input‐dependent data noise term (MCD+N), an efficient uncertainty estimation framework originally developed in computer vision, for hydrologic time series predictions. MCD+N simultaneously estimates a heteroscedastic input‐dependent data noise term (a trained error model attributable to observational noise) and a network weight uncertainty term (attributable to insufficiently constrained model parameters). Although MCD+N has appealing features, many heuristic approximations were employed duringmore »« less
https://doi.org/10.1029/2020wr028095

Full Text Available
Prolongation of SMAP to Spatiotemporally Seamless Coverage of Continental U.S. Using a Deep Learning Neural Network

Journal Article Fang, Kuai ; Shen, Chaopeng ; Kifer, Daniel ; ... - Geophysical Research Letters

The Soil Moisture Active Passive (SMAP) mission has delivered valuable sensing of surface soil moisture since 2015. However, it has a short time span and irregular revisit schedules. Utilizing a state-of-the-art time series deep learning neural network, Long Short-Term Memory (LSTM), we created a system that predicts SMAP level-3 moisture product with atmospheric forcings, model-simulated moisture, and static physiographic attributes as inputs. The system removes most of the bias with model simulations and improves predicted moisture climatology, achieving small test root-mean-square errors (<0.035) and high-correlation coefficients >0.87 for over 75% of Continental United States, including the forested southeast. As themore »« less
Cited by 119
https://doi.org/10.1002/2017gl075619

Full Text Available
Enhancing Streamflow Forecast and Extracting Insights Using Long-Short Term Memory Networks With Data Integration at Continental Scales

Journal Article Feng, Dapeng ; Fang, Kuai ; Shen, Chaopeng - Water Resources Research

Abstract Recent observations with varied schedules and types (moving average, snapshot, or regularly spaced) can help to improve streamflow forecasts, but it is challenging to integrate them effectively. Based on a long short‐term memory (LSTM) streamflow model, we tested multiple versions of a flexible procedure we call data integration (DI) to leverage recent discharge measurements to improve forecasts. DI accepts lagged inputs either directly or through a convolutional neural network unit. DI ubiquitously elevated streamflow forecast performance to unseen levels, reaching a record continental‐scale median Nash‐Sutcliffe Efficiency coefficient value of 0.86. Integrating moving‐average discharge, discharge from the last few days,more »« less
https://doi.org/10.1029/2019wr026793

Full Text Available
Development of a spatially complete floodplain map of the conterminous United States using random forest

Journal Article Woznicki, Sean A. ; Baynes, Jeremy ; Panlasigui, Stephanie ; ... - Science of the Total Environment

Floodplains perform several important ecosystem services, including storing water during precipitation events and reducing peak flows, thus reducing flooding of downstream communities. Understanding the relationship between flood inundation and floodplains is critical for ecosystem and community health and well-being, as well as targeting floodplain and riparian restoration. Many communities in the United States, particularly those in rural areas, lack inundation maps due to the high cost of flood modeling. Only 60% of the conterminous United States has Flood Insurance Rate Maps (FIRMs) through the U.S. Federal Emergency Management Agency (FEMA). We developed a 30- meter resolution flood inundation map ofmore »« less
https://doi.org/10.1016/j.scitotenv.2018.07.353
Unraveling the 2021 Central Tennessee flood event using a hierarchical multi-model inundation modeling framework

Journal Article Gangrade, Sudershan ; Ghimire, Ganesh R. ; Kao, Shih-Chieh ; ... - Journal of Hydrology

Flood prediction systems need hierarchical atmospheric, hydrologic, and hydraulic models to predict rainfall, runoff, streamflow, and floodplain inundation. The accuracy of such systems depends on the error propagation through the modeling chain, sensitivity to input data, and choice of models. In this study, we used multiple precipitation forcings (hindcast and forecast) to drive hydrologic and hydrodynamic models to analyze the impacts of various drivers on the estimates of flood inundation depth and extent. We implement this framework to unravel the August 2021 extreme flooding event that occurred in Central Tennessee, USA. We used two radar-based quantitative precipitation estimates (STAGE4 andmore »« less
https://doi.org/10.1016/j.jhydrol.2023.130157

Full Text Available

Similar Records