Application of a hybrid MPI/OpenMP approach for parallel groundwater model calibration using multi-core computers

Journal Article · Fri Jan 01 00:00:00 EST 2010 · Computers and Geosciences

DOI:https://doi.org/10.1016/j.cageo.2010.04.013· OSTI ID:1185314

Tang, Guoping ^[1]; D'Azevedo, Ed F ^[1]; Zhang, Fan ^[1]; Parker, Jack C. ^[2]; Watson, David B ^[1]; Jardine, Philip ^[2]

---

1. ORNL
2. University of Tennessee, Knoxville (UTK)

Calibration of groundwater models involves hundreds to thousands of forward solutions, each of which may solve many transient coupled nonlinear partial differential equations, resulting in a computationally intensive problem. We describe a hybrid MPI/OpenMP approach to exploit two levels of parallelisms in software and hardware to reduce calibration time on multi-core computers. HydroGeoChem 5.0 (HGC5) is parallelized using OpenMP for direct solutions for a reactive transport model application, and a field-scale coupled flow and transport model application. In the reactive transport model, a single parallelizable loop is identified to account for over 97% of the total computational time using GPROF. Addition of a few lines of OpenMP compiler directives to the loop yields a speedup of about 10 on a 16-core compute node. For the field-scale model, parallelizable loops in 14 of 174 HGC5 subroutines that require 99% of the execution time are identified. As these loops are parallelized incrementally, the scalability is found to be limited by a loop where Cray PAT detects over 90% cache missing rates. With this loop rewritten, similar speedup as the first application is achieved. The OpenMP-parallelized code can be run efficiently on multiple workstations in a network or multiple compute nodes on a cluster as slaves using parallel PEST to speedup model calibration. To run calibration on clusters as a single task, the Levenberg Marquardt algorithm is added to HGC5 with the Jacobian calculation and lambda search parallelized using MPI. With this hybrid approach, 100 200 compute cores are used to reduce the calibration time from weeks to a few hours for these two applications. This approach is applicable to most of the existing groundwater model codes for many applications.

• Research Organization: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
• Sponsoring Organization: USDOE Office of Science (SC)
• DOE Contract Number: DE-AC05-00OR22725
• OSTI ID: 1185314
• Journal Information: Computers and Geosciences, Vol. 36, Issue 11; ISSN 0098-3004
• Country of Publication: United States
• Language: English