skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN

Abstract

To better inform the subsurface scientist on the expected performance of parallel simulators, this work investigates performance of the reactive multiphase flow and multicomponent biogeochemical transport code PFLOTRAN as it is applied to several realistic modeling scenarios run on the Jaguar supercomputer. After a brief introduction to the code's parallel layout and code design, PFLOTRAN's parallel performance (measured through strong and weak scalability analyses) is evaluated in the context of conceptual model layout, software and algorithmic design, and known hardware limitations. PFLOTRAN scales well (with regard to strong scaling) for three realistic problem scenarios: (1) in situ leaching of copper from a mineral ore deposit within a 5-spot flow regime, (2) transient flow and solute transport within a regional doublet, and (3) a real-world problem involving uranium surface complexation within a heterogeneous and extremely dynamic variably saturated flow field. Weak scalability is discussed in detail for the regional doublet problem, and several difficulties with its interpretation are noted.

Authors:
 [1];  [2];  [3]
  1. Sandia National Lab., Albuquerque, NM (United States). Applied Systems Analysis and Research
  2. OFM Research, Santa Fe, NM (United States)
  3. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Environmental Sciences Division; Univ. of Tennessee, Knoxville, TN (United States).Electrical Engineering and Computer Science and Earth and Planetary Sciences Dept.
Publication Date:
Research Org.:
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Org.:
USDOE Office of Science (SC), Biological and Environmental Research (BER) (SC-23)
OSTI Identifier:
1565099
Grant/Contract Number:  
AC05-76RL01830
Resource Type:
Accepted Manuscript
Journal Name:
Water Resources Research
Additional Journal Information:
Journal Volume: 50; Journal Issue: 1; Journal ID: ISSN 0043-1397
Publisher:
American Geophysical Union (AGU)
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES; environmental sciences & ecology; marine & freshwater biology; water resources; high performance computing; biogeochemical transport; groundwater flow

Citation Formats

Hammond, G. E., Lichtner, P. C., and Mills, R. T. Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN. United States: N. p., 2013. Web. doi:10.1002/2012wr013483.
Hammond, G. E., Lichtner, P. C., & Mills, R. T. Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN. United States. doi:10.1002/2012wr013483.
Hammond, G. E., Lichtner, P. C., and Mills, R. T. Thu . "Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN". United States. doi:10.1002/2012wr013483. https://www.osti.gov/servlets/purl/1565099.
@article{osti_1565099,
title = {Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN},
author = {Hammond, G. E. and Lichtner, P. C. and Mills, R. T.},
abstractNote = {To better inform the subsurface scientist on the expected performance of parallel simulators, this work investigates performance of the reactive multiphase flow and multicomponent biogeochemical transport code PFLOTRAN as it is applied to several realistic modeling scenarios run on the Jaguar supercomputer. After a brief introduction to the code's parallel layout and code design, PFLOTRAN's parallel performance (measured through strong and weak scalability analyses) is evaluated in the context of conceptual model layout, software and algorithmic design, and known hardware limitations. PFLOTRAN scales well (with regard to strong scaling) for three realistic problem scenarios: (1) in situ leaching of copper from a mineral ore deposit within a 5-spot flow regime, (2) transient flow and solute transport within a regional doublet, and (3) a real-world problem involving uranium surface complexation within a heterogeneous and extremely dynamic variably saturated flow field. Weak scalability is discussed in detail for the regional doublet problem, and several difficulties with its interpretation are noted.},
doi = {10.1002/2012wr013483},
journal = {Water Resources Research},
number = 1,
volume = 50,
place = {United States},
year = {2013},
month = {12}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 66 works
Citation information provided by
Web of Science

Figures / Tables:

Figure 1 Figure 1: Flowchart of key PFLOTRAN objects and operations.

Save / Share:

Works referenced in this record:

Coupled 3-D thermo-hydro-mechanical analysis of geotechnological in situ tests
journal, January 2011

  • Nowak, Thomas; Kunz, Herbert; Dixon, David
  • International Journal of Rock Mechanics and Mining Sciences, Vol. 48, Issue 1
  • DOI: 10.1016/j.ijrmms.2010.11.002

Integrated surface–groundwater flow modeling: A free-surface overland flow boundary condition in a parallel groundwater flow model
journal, July 2006


Parallel computing simulation of fluid flow in the unsaturated zone of Yucca Mountain, Nevada
journal, April 2003


Modeling the dewatering and depressurization of the Lihir open-pit gold mine, Papua New Guinea
journal, August 2004


Iterative coupling reservoir simulation on high performance computers
journal, February 2009


OpenGeoSys: an open-source initiative for numerical simulation of thermo-hydro-mechanical/chemical (THM/C) processes in porous media
journal, February 2012


Numerical investigation concerning the impact of CO2 geologic storage on regional groundwater flow
journal, September 2009

  • Yamamoto, Hajime; Zhang, Keni; Karasaki, Kenzi
  • International Journal of Greenhouse Gas Control, Vol. 3, Issue 5
  • DOI: 10.1016/j.ijggc.2009.04.007

Proof of concept of regional scale hydrologic simulations at hydrologic resolution utilizing massively parallel computer resources: RAPID COMMUNICATION
journal, April 2010

  • Kollet, Stefan J.; Maxwell, Reed M.; Woodward, Carol S.
  • Water Resources Research, Vol. 46, Issue 4
  • DOI: 10.1029/2009WR008730

Analysis of subsurface contaminant migration and remediation using high performance computing
journal, November 1998


Scalable Modeling of Carbon Tetrachloride Migration at the Hanford Site Using the STOMP Simulator
journal, January 2008

  • White, M. D.; Oostrom, M.; Rockhold, M. L.
  • Vadose Zone Journal, Vol. 7, Issue 2
  • DOI: 10.2136/vzj2007.0070

HBGC123D: a high-performance computer model of coupled hydrogeological and biogeochemical processes
journal, December 2001


Expressing object-oriented concepts in Fortran 90
journal, April 1997

  • Decyk, Viktor K.; Norton, Charles D.; Szymanski, Boleslaw K.
  • ACM SIGPLAN Fortran Forum, Vol. 16, Issue 1
  • DOI: 10.1145/263877.263880

Supercomputers and their use in modeling subsurface solute transport
journal, January 1990


Three-dimensional Bayesian geostatistical aquifer characterization at the Hanford 300 Area using tracer test data: INVERSION OF TRACER DATA USING MAD
journal, June 2012

  • Chen, Xingyuan; Murakami, Haruko; Hahn, Melanie S.
  • Water Resources Research, Vol. 48, Issue 6
  • DOI: 10.1029/2011WR010675

Elucidating geochemical response of shallow heterogeneous aquifers to CO2 leakage using high-performance computing: Implications for monitoring of CO2 sequestration
journal, March 2013


Variably saturated flow and multicomponent biogeochemical reactive transport modeling of a uranium bioremediation field experiment
journal, November 2011

  • Yabusaki, Steven B.; Fang, Yilin; Williams, Kenneth H.
  • Journal of Contaminant Hydrology, Vol. 126, Issue 3-4
  • DOI: 10.1016/j.jconhyd.2011.09.002

Application of Jacobian-free Newton–Krylov with physics-based preconditioning to biogeochemical transport
journal, April 2005


Newton–Krylov-multigrid solvers for large-scale, highly heterogeneous, variably saturated flow problems
journal, July 2001


Guidance for Choosing Multigrid Preconditioners for Systems of Elliptic Partial Differential Equations
journal, January 2009

  • Lee, B.
  • SIAM Journal on Scientific Computing, Vol. 31, Issue 4
  • DOI: 10.1137/070703636

Stochastic simulation of uranium migration at the Hanford 300 Area
journal, March 2011


Modeling subsurface reactive flows using leadership-class computing
journal, July 2009


    Works referencing / citing this record:

    Modeling the spatiotemporal variability in subsurface thermal regimes across a low-relief polygonal tundra landscape
    journal, January 2016