A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States
Abstract
The Weather Research and Forecasting Hydrological (WRF-Hydro) system is a state-of-the-art numerical model that models the entire hydrological cycle based on physical principles. As with other hydrological models, WRF-Hydro parameterizes many physical processes. Hence, WRF-Hydro needs to be calibrated to optimize its output with respect to observations for the application region. When applied to a relatively large domain, both WRF-Hydro simulations and calibrations require intensive computing resources and are best performed on multimode, multicore high-performance computing (HPC) systems. Typically, each physics-based model requires a calibration process that works specifically with that model and is not transferrable to a different process or model. The parameter estimation tool (PEST) is a flexible and generic calibration tool that can be used in principle to calibrate any of these models. In its existing configuration, however, PEST is not designed to work on the current generation of massively parallel HPC clusters. To address this issue, we ported the parallel PEST to HPCs and adapted it to work with WRF-Hydro. The porting involved writing scripts to modify the workflow for different workload managers and job schedulers, as well as to connect the parallel PEST to WRF-Hydro. To test the operational feasibility and the computational benefits ofmore »
- Authors:
-
- Argonne National Lab. (ANL), Lemont, IL (United States)
- Publication Date:
- Research Org.:
- Argonne National Laboratory (ANL), Argonne, IL (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- OSTI Identifier:
- 1557999
- Grant/Contract Number:
- AC02-06CH11357
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Geoscientific Model Development (Online)
- Additional Journal Information:
- Journal Name: Geoscientific Model Development (Online); Journal Volume: 12; Journal Issue: 8; Journal ID: ISSN 1991-9603
- Publisher:
- European Geosciences Union
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 54 ENVIRONMENTAL SCIENCES
Citation Formats
Wang, Jiali, Wang, Cheng, Rao, Vishwas, Orr, Andrew, Yan, Eugene, and Kotamarthi, Rao. A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States. United States: N. p., 2019.
Web. doi:10.5194/gmd-12-3523-2019.
Wang, Jiali, Wang, Cheng, Rao, Vishwas, Orr, Andrew, Yan, Eugene, & Kotamarthi, Rao. A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States. United States. https://doi.org/10.5194/gmd-12-3523-2019
Wang, Jiali, Wang, Cheng, Rao, Vishwas, Orr, Andrew, Yan, Eugene, and Kotamarthi, Rao. Tue .
"A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States". United States. https://doi.org/10.5194/gmd-12-3523-2019. https://www.osti.gov/servlets/purl/1557999.
@article{osti_1557999,
title = {A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States},
author = {Wang, Jiali and Wang, Cheng and Rao, Vishwas and Orr, Andrew and Yan, Eugene and Kotamarthi, Rao},
abstractNote = {The Weather Research and Forecasting Hydrological (WRF-Hydro) system is a state-of-the-art numerical model that models the entire hydrological cycle based on physical principles. As with other hydrological models, WRF-Hydro parameterizes many physical processes. Hence, WRF-Hydro needs to be calibrated to optimize its output with respect to observations for the application region. When applied to a relatively large domain, both WRF-Hydro simulations and calibrations require intensive computing resources and are best performed on multimode, multicore high-performance computing (HPC) systems. Typically, each physics-based model requires a calibration process that works specifically with that model and is not transferrable to a different process or model. The parameter estimation tool (PEST) is a flexible and generic calibration tool that can be used in principle to calibrate any of these models. In its existing configuration, however, PEST is not designed to work on the current generation of massively parallel HPC clusters. To address this issue, we ported the parallel PEST to HPCs and adapted it to work with WRF-Hydro. The porting involved writing scripts to modify the workflow for different workload managers and job schedulers, as well as to connect the parallel PEST to WRF-Hydro. To test the operational feasibility and the computational benefits of this first-of-its-kind HPC-enabled parallel PEST, we developed a case study using a flood in the midwestern United States in 2013. Results on a problem involving the calibration of 22 parameters show that on the same computing resources used for parallel WRF-Hydro, the HPC-enabled parallel PEST can speed up the calibration process by a factor of up to 15 compared with commonly used PEST in sequential mode. The speedup factor is expected to be greater with a larger calibration problem (e.g., more parameters to be calibrated or a larger size of study area).},
doi = {10.5194/gmd-12-3523-2019},
journal = {Geoscientific Model Development (Online)},
number = 8,
volume = 12,
place = {United States},
year = {Tue Aug 13 00:00:00 EDT 2019},
month = {Tue Aug 13 00:00:00 EDT 2019}
}
Web of Science
Works referenced in this record:
Model Evaluation Guidelines for Systematic Quantification of Accuracy in Watershed Simulations
journal, January 2007
- D. N. Moriasi,
- Transactions of the ASABE, Vol. 50, Issue 3
Role of Runoff–Infiltration Partitioning and Resolved Overland Flow on Land–Atmosphere Feedbacks: A Case Study with the WRF-Hydro Coupled Modeling System for West Africa
journal, May 2016
- Arnault, Joel; Wagner, Sven; Rummler, Thomas
- Journal of Hydrometeorology, Vol. 17, Issue 5
Evaluation of precipitation estimates over CONUS derived from satellite, radar, and rain gauge data sets at daily to annual scales (2002–2012)
journal, January 2015
- Prat, O. P.; Nelson, B. R.
- Hydrology and Earth System Sciences, Vol. 19, Issue 4
Calibration and evaluation of a flood forecasting system: Utility of numerical weather prediction model, data assimilation and satellite-based rainfall
journal, April 2015
- Yucel, I.; Onen, A.; Yilmaz, K. K.
- Journal of Hydrology, Vol. 523
Implementation of a vector-based river network routing scheme in the community WRF-Hydro modeling framework for flood discharge simulation
journal, September 2018
- Lin, Peirong; Yang, Zong-Liang; Gochis, David J.
- Environmental Modelling & Software, Vol. 107
Modeling of the effect of land use changes on the runoff generation of a river basin through parameter regionalization of a watershed model
journal, June 2004
- Hundecha, Yeshewatesfa; Bárdossy, András
- Journal of Hydrology, Vol. 292, Issue 1-4
Parameter estimation and regionalization for continuous rainfall-runoff models including uncertainty
journal, March 2006
- Wagener, Thorsten; Wheater, Howard S.
- Journal of Hydrology, Vol. 320, Issue 1-2
Numerical simulation and analysis of the April 2013 Chicago Floods
journal, December 2015
- Campos, Edwin; Wang, Jiali
- Journal of Hydrology, Vol. 531
River flow forecasting through conceptual models part I — A discussion of principles
journal, April 1970
- Nash, J. E.; Sutcliffe, J. V.
- Journal of Hydrology, Vol. 10, Issue 3
Joint atmospheric-terrestrial water balances for East Africa: a WRF-Hydro case study for the upper Tana River basin
journal, February 2017
- Kerandi, Noah; Arnault, Joel; Laux, Patrick
- Theoretical and Applied Climatology, Vol. 131, Issue 3-4
Multi-objective parameter conditioning of a three-source wheat canopy model
journal, March 2004
- Mo, Xingguo; Beven, Keith
- Agricultural and Forest Meteorology, Vol. 122, Issue 1-2
Automatic calibration of a conceptual rainfall–runoff model using multiple objectives
journal, August 2000
- Madsen, H.
- Journal of Hydrology, Vol. 235, Issue 3-4
Equifinality, a blessing in disguise?
journal, January 2001
- Savenije, Hubert H. G.
- Hydrological Processes, Vol. 15, Issue 14
Integrating spatial altimetry data into the automatic calibration of hydrological models
journal, June 2010
- Getirana, Augusto C. V.
- Journal of Hydrology, Vol. 387, Issue 3-4
Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology
journal, August 2001
- Beven, Keith; Freer, Jim
- Journal of Hydrology, Vol. 249, Issue 1-4
Continental-scale water and energy flux analysis and validation for North American Land Data Assimilation System project phase 2 (NLDAS-2): 2. Validation of model-simulated streamflow: VALIDATION OF MODEL-SIMULATED STREAMFLOW
journal, February 2012
- Xia, Youlong; Mitchell, Kenneth; Ek, Michael
- Journal of Geophysical Research: Atmospheres, Vol. 117, Issue D3
An Intercomparison and Validation of High-Resolution Satellite Precipitation Estimates with 3-Hourly Gauge Data
journal, February 2009
- Sapiano, M. R. P.; Arkin, P. A.
- Journal of Hydrometeorology, Vol. 10, Issue 1
Fully coupled atmosphere-hydrology simulations for the central Mediterranean: Impact of enhanced hydrological parameterization for short and long time scales: FULLY COUPLED ATMOSPHERE-HYDROLOGY MODEL
journal, November 2015
- Senatore, Alfonso; Mendicino, Giuseppe; Gochis, David J.
- Journal of Advances in Modeling Earth Systems, Vol. 7, Issue 4
Coupling an Advanced Land Surface–Hydrology Model with the Penn State–NCAR MM5 Modeling System. Part I: Model Implementation and Sensitivity
journal, April 2001
- Chen, Fei; Dudhia, Jimy
- Monthly Weather Review, Vol. 129, Issue 4
Continental-scale water and energy flux analysis and validation for the North American Land Data Assimilation System project phase 2 (NLDAS-2): 1. Intercomparison and application of model products: WATER AND ENERGY FLUX ANALYSIS
journal, February 2012
- Xia, Youlong; Mitchell, Kenneth; Ek, Michael
- Journal of Geophysical Research: Atmospheres, Vol. 117, Issue D3
The community Noah land surface model with multiparameterization options (Noah-MP): 1. Model description and evaluation with local-scale measurements
journal, January 2011
- Niu, Guo-Yue; Yang, Zong-Liang; Mitchell, Kenneth E.
- Journal of Geophysical Research, Vol. 116, Issue D12
Using internal catchment information to reduce the uncertainty of discharge and baseflow predictions
journal, April 2007
- Gallart, Francesc; Latron, Jérôme; Llorens, Pilar
- Advances in Water Resources, Vol. 30, Issue 4
The impact of standard and hard-coded parameters on the hydrologic fluxes in the Noah-MP land surface model: HARD-CODED PARAMETERS IN NOAH-MP
journal, September 2016
- Cuntz, Matthias; Mai, Juliane; Samaniego, Luis
- Journal of Geophysical Research: Atmospheres, Vol. 121, Issue 18
Ground Water Model Calibration Using Pilot Points and Regularization
journal, March 2003
- Doherty, John
- Ground Water, Vol. 41, Issue 2
Joint atmospheric-terrestrial water balances for East Africa: A WRF-Hydro case study for the upper Tana River basin
text, January 2017
- Kerandi, Noah; Arnault, Joel; Laux, Patrick
- Unpublished
Joint atmospheric-terrestrial water balances for East Africa : a WRF-Hydro case study for the upper Tana River basin
text, January 2017
- Kerandi, Noah; Arnault, Joel; Laux, Patrick
- Karlsruhe
Role of Runoff–Infiltration Partitioning and Resolved Overland Flow on Land–Atmosphere Feedbacks: A Case Study with the WRF-Hydro Coupled Modeling System for West Africa
text, January 2016
- Arnault, Joel; Wagner, Sven; Rummler, Thomas
- American Meteorological Society
Using internal catchment information to reduce the uncertainty of discharge and baseflow predictions
journal, April 2007
- Gallart, Francesc; Latron, Jérôme; Llorens, Pilar
- Advances in Water Resources, Vol. 30, Issue 4
Modeling of the effect of land use changes on the runoff generation of a river basin through parameter regionalization of a watershed model
journal, June 2004
- Hundecha, Yeshewatesfa; Bárdossy, András
- Journal of Hydrology, Vol. 292, Issue 1-4
Integrating spatial altimetry data into the automatic calibration of hydrological models
journal, June 2010
- Getirana, Augusto C. V.
- Journal of Hydrology, Vol. 387, Issue 3-4
A hybrid regularized inversion methodology for highly parameterized environmental models: HYBRID REGULARIZATION METHODOLOGY
journal, October 2005
- Tonkin, Matthew James; Doherty, John
- Water Resources Research, Vol. 41, Issue 10