DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States

Abstract

The Weather Research and Forecasting Hydrological (WRF-Hydro) system is a state-of-the-art numerical model that models the entire hydrological cycle based on physical principles. As with other hydrological models, WRF-Hydro parameterizes many physical processes. Hence, WRF-Hydro needs to be calibrated to optimize its output with respect to observations for the application region. When applied to a relatively large domain, both WRF-Hydro simulations and calibrations require intensive computing resources and are best performed on multimode, multicore high-performance computing (HPC) systems. Typically, each physics-based model requires a calibration process that works specifically with that model and is not transferrable to a different process or model. The parameter estimation tool (PEST) is a flexible and generic calibration tool that can be used in principle to calibrate any of these models. In its existing configuration, however, PEST is not designed to work on the current generation of massively parallel HPC clusters. To address this issue, we ported the parallel PEST to HPCs and adapted it to work with WRF-Hydro. The porting involved writing scripts to modify the workflow for different workload managers and job schedulers, as well as to connect the parallel PEST to WRF-Hydro. To test the operational feasibility and the computational benefits ofmore » this first-of-its-kind HPC-enabled parallel PEST, we developed a case study using a flood in the midwestern United States in 2013. Results on a problem involving the calibration of 22 parameters show that on the same computing resources used for parallel WRF-Hydro, the HPC-enabled parallel PEST can speed up the calibration process by a factor of up to 15 compared with commonly used PEST in sequential mode. The speedup factor is expected to be greater with a larger calibration problem (e.g., more parameters to be calibrated or a larger size of study area).« less

Authors:
 [1]; ORCiD logo [1];  [1];  [1];  [1]; ORCiD logo [1]
  1. Argonne National Lab. (ANL), Lemont, IL (United States)
Publication Date:
Research Org.:
Argonne National Laboratory (ANL), Argonne, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
OSTI Identifier:
1557999
Grant/Contract Number:  
AC02-06CH11357
Resource Type:
Accepted Manuscript
Journal Name:
Geoscientific Model Development (Online)
Additional Journal Information:
Journal Name: Geoscientific Model Development (Online); Journal Volume: 12; Journal Issue: 8; Journal ID: ISSN 1991-9603
Publisher:
European Geosciences Union
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES

Citation Formats

Wang, Jiali, Wang, Cheng, Rao, Vishwas, Orr, Andrew, Yan, Eugene, and Kotamarthi, Rao. A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States. United States: N. p., 2019. Web. doi:10.5194/gmd-12-3523-2019.
Wang, Jiali, Wang, Cheng, Rao, Vishwas, Orr, Andrew, Yan, Eugene, & Kotamarthi, Rao. A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States. United States. https://doi.org/10.5194/gmd-12-3523-2019
Wang, Jiali, Wang, Cheng, Rao, Vishwas, Orr, Andrew, Yan, Eugene, and Kotamarthi, Rao. Tue . "A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States". United States. https://doi.org/10.5194/gmd-12-3523-2019. https://www.osti.gov/servlets/purl/1557999.
@article{osti_1557999,
title = {A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States},
author = {Wang, Jiali and Wang, Cheng and Rao, Vishwas and Orr, Andrew and Yan, Eugene and Kotamarthi, Rao},
abstractNote = {The Weather Research and Forecasting Hydrological (WRF-Hydro) system is a state-of-the-art numerical model that models the entire hydrological cycle based on physical principles. As with other hydrological models, WRF-Hydro parameterizes many physical processes. Hence, WRF-Hydro needs to be calibrated to optimize its output with respect to observations for the application region. When applied to a relatively large domain, both WRF-Hydro simulations and calibrations require intensive computing resources and are best performed on multimode, multicore high-performance computing (HPC) systems. Typically, each physics-based model requires a calibration process that works specifically with that model and is not transferrable to a different process or model. The parameter estimation tool (PEST) is a flexible and generic calibration tool that can be used in principle to calibrate any of these models. In its existing configuration, however, PEST is not designed to work on the current generation of massively parallel HPC clusters. To address this issue, we ported the parallel PEST to HPCs and adapted it to work with WRF-Hydro. The porting involved writing scripts to modify the workflow for different workload managers and job schedulers, as well as to connect the parallel PEST to WRF-Hydro. To test the operational feasibility and the computational benefits of this first-of-its-kind HPC-enabled parallel PEST, we developed a case study using a flood in the midwestern United States in 2013. Results on a problem involving the calibration of 22 parameters show that on the same computing resources used for parallel WRF-Hydro, the HPC-enabled parallel PEST can speed up the calibration process by a factor of up to 15 compared with commonly used PEST in sequential mode. The speedup factor is expected to be greater with a larger calibration problem (e.g., more parameters to be calibrated or a larger size of study area).},
doi = {10.5194/gmd-12-3523-2019},
journal = {Geoscientific Model Development (Online)},
number = 8,
volume = 12,
place = {United States},
year = {Tue Aug 13 00:00:00 EDT 2019},
month = {Tue Aug 13 00:00:00 EDT 2019}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 12 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Model Evaluation Guidelines for Systematic Quantification of Accuracy in Watershed Simulations
journal, January 2007


Implementation of a vector-based river network routing scheme in the community WRF-Hydro modeling framework for flood discharge simulation
journal, September 2018


Modeling of the effect of land use changes on the runoff generation of a river basin through parameter regionalization of a watershed model
journal, June 2004


Parameter estimation and regionalization for continuous rainfall-runoff models including uncertainty
journal, March 2006


Numerical simulation and analysis of the April 2013 Chicago Floods
journal, December 2015


River flow forecasting through conceptual models part I — A discussion of principles
journal, April 1970


Joint atmospheric-terrestrial water balances for East Africa: a WRF-Hydro case study for the upper Tana River basin
journal, February 2017

  • Kerandi, Noah; Arnault, Joel; Laux, Patrick
  • Theoretical and Applied Climatology, Vol. 131, Issue 3-4
  • DOI: 10.1007/s00704-017-2050-8

Multi-objective parameter conditioning of a three-source wheat canopy model
journal, March 2004


Automatic calibration of a conceptual rainfall–runoff model using multiple objectives
journal, August 2000


Equifinality, a blessing in disguise?
journal, January 2001

  • Savenije, Hubert H. G.
  • Hydrological Processes, Vol. 15, Issue 14
  • DOI: 10.1002/hyp.494

Integrating spatial altimetry data into the automatic calibration of hydrological models
journal, June 2010


An Intercomparison and Validation of High-Resolution Satellite Precipitation Estimates with 3-Hourly Gauge Data
journal, February 2009

  • Sapiano, M. R. P.; Arkin, P. A.
  • Journal of Hydrometeorology, Vol. 10, Issue 1
  • DOI: 10.1175/2008JHM1052.1

Fully coupled atmosphere-hydrology simulations for the central Mediterranean: Impact of enhanced hydrological parameterization for short and long time scales: FULLY COUPLED ATMOSPHERE-HYDROLOGY MODEL
journal, November 2015

  • Senatore, Alfonso; Mendicino, Giuseppe; Gochis, David J.
  • Journal of Advances in Modeling Earth Systems, Vol. 7, Issue 4
  • DOI: 10.1002/2015MS000510

The community Noah land surface model with multiparameterization options (Noah-MP): 1. Model description and evaluation with local-scale measurements
journal, January 2011

  • Niu, Guo-Yue; Yang, Zong-Liang; Mitchell, Kenneth E.
  • Journal of Geophysical Research, Vol. 116, Issue D12
  • DOI: 10.1029/2010JD015139

Using internal catchment information to reduce the uncertainty of discharge and baseflow predictions
journal, April 2007


The impact of standard and hard-coded parameters on the hydrologic fluxes in the Noah-MP land surface model: HARD-CODED PARAMETERS IN NOAH-MP
journal, September 2016

  • Cuntz, Matthias; Mai, Juliane; Samaniego, Luis
  • Journal of Geophysical Research: Atmospheres, Vol. 121, Issue 18
  • DOI: 10.1002/2016JD025097

Ground Water Model Calibration Using Pilot Points and Regularization
journal, March 2003


Using internal catchment information to reduce the uncertainty of discharge and baseflow predictions
journal, April 2007


Modeling of the effect of land use changes on the runoff generation of a river basin through parameter regionalization of a watershed model
journal, June 2004


Integrating spatial altimetry data into the automatic calibration of hydrological models
journal, June 2010


A hybrid regularized inversion methodology for highly parameterized environmental models: HYBRID REGULARIZATION METHODOLOGY
journal, October 2005

  • Tonkin, Matthew James; Doherty, John
  • Water Resources Research, Vol. 41, Issue 10
  • DOI: 10.1029/2005wr003995