skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN

Abstract

To better inform the subsurface scientist on the expected performance of parallel simulators, this work investigates performance of the reactive multiphase flow and multicomponent biogeochemical transport code PFLOTRAN as it is applied to several realistic modeling scenarios run on the Jaguar supercomputer. After a brief introduction to the code's parallel layout and code design, PFLOTRAN's parallel performance (measured through strong and weak scalability analyses) is evaluated in the context of conceptual model layout, software and algorithmic design, and known hardware limitations. PFLOTRAN scales well (with regard to strong scaling) for three realistic problem scenarios: (1) in situ leaching of copper from a mineral ore deposit within a 5-spot flow regime, (2) transient flow and solute transport within a regional doublet, and (3) a real-world problem involving uranium surface complexation within a heterogeneous and extremely dynamic variably saturated flow field. Weak scalability is discussed in detail for the regional doublet problem, and several difficulties with its interpretation are noted.

Authors:
 [1];  [2];  [3]
  1. Sandia National Lab., Albuquerque, NM (United States). Applied Systems Analysis and Research
  2. OFM Research, Santa Fe, NM (United States)
  3. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Environmental Sciences Division; Univ. of Tennessee, Knoxville, TN (United States).Electrical Engineering and Computer Science and Earth and Planetary Sciences Dept.
Publication Date:
Research Org.:
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Org.:
USDOE Office of Science (SC), Biological and Environmental Research (BER)
OSTI Identifier:
1565099
Grant/Contract Number:  
AC05-76RL01830
Resource Type:
Accepted Manuscript
Journal Name:
Water Resources Research
Additional Journal Information:
Journal Volume: 50; Journal Issue: 1; Journal ID: ISSN 0043-1397
Publisher:
American Geophysical Union (AGU)
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES; environmental sciences & ecology; marine & freshwater biology; water resources; high performance computing; biogeochemical transport; groundwater flow

Citation Formats

Hammond, G. E., Lichtner, P. C., and Mills, R. T. Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN. United States: N. p., 2013. Web. doi:10.1002/2012wr013483.
Hammond, G. E., Lichtner, P. C., & Mills, R. T. Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN. United States. doi:https://doi.org/10.1002/2012wr013483
Hammond, G. E., Lichtner, P. C., and Mills, R. T. Thu . "Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN". United States. doi:https://doi.org/10.1002/2012wr013483. https://www.osti.gov/servlets/purl/1565099.
@article{osti_1565099,
title = {Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN},
author = {Hammond, G. E. and Lichtner, P. C. and Mills, R. T.},
abstractNote = {To better inform the subsurface scientist on the expected performance of parallel simulators, this work investigates performance of the reactive multiphase flow and multicomponent biogeochemical transport code PFLOTRAN as it is applied to several realistic modeling scenarios run on the Jaguar supercomputer. After a brief introduction to the code's parallel layout and code design, PFLOTRAN's parallel performance (measured through strong and weak scalability analyses) is evaluated in the context of conceptual model layout, software and algorithmic design, and known hardware limitations. PFLOTRAN scales well (with regard to strong scaling) for three realistic problem scenarios: (1) in situ leaching of copper from a mineral ore deposit within a 5-spot flow regime, (2) transient flow and solute transport within a regional doublet, and (3) a real-world problem involving uranium surface complexation within a heterogeneous and extremely dynamic variably saturated flow field. Weak scalability is discussed in detail for the regional doublet problem, and several difficulties with its interpretation are noted.},
doi = {10.1002/2012wr013483},
journal = {Water Resources Research},
number = 1,
volume = 50,
place = {United States},
year = {2013},
month = {12}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 86 works
Citation information provided by
Web of Science

Figures / Tables:

Figure 1 Figure 1: Flowchart of key PFLOTRAN objects and operations.

Save / Share:

Works referenced in this record:

Coupled 3-D thermo-hydro-mechanical analysis of geotechnological in situ tests
journal, January 2011

  • Nowak, Thomas; Kunz, Herbert; Dixon, David
  • International Journal of Rock Mechanics and Mining Sciences, Vol. 48, Issue 1
  • DOI: 10.1016/j.ijrmms.2010.11.002

Integrated surface–groundwater flow modeling: A free-surface overland flow boundary condition in a parallel groundwater flow model
journal, July 2006


Parallel computing simulation of fluid flow in the unsaturated zone of Yucca Mountain, Nevada
journal, April 2003


Modeling the dewatering and depressurization of the Lihir open-pit gold mine, Papua New Guinea
journal, August 2004


Iterative coupling reservoir simulation on high performance computers
journal, February 2009


OpenGeoSys: an open-source initiative for numerical simulation of thermo-hydro-mechanical/chemical (THM/C) processes in porous media
journal, February 2012


Numerical investigation concerning the impact of CO2 geologic storage on regional groundwater flow
journal, September 2009

  • Yamamoto, Hajime; Zhang, Keni; Karasaki, Kenzi
  • International Journal of Greenhouse Gas Control, Vol. 3, Issue 5
  • DOI: 10.1016/j.ijggc.2009.04.007

Proof of concept of regional scale hydrologic simulations at hydrologic resolution utilizing massively parallel computer resources: RAPID COMMUNICATION
journal, April 2010

  • Kollet, Stefan J.; Maxwell, Reed M.; Woodward, Carol S.
  • Water Resources Research, Vol. 46, Issue 4
  • DOI: 10.1029/2009WR008730

Analysis of subsurface contaminant migration and remediation using high performance computing
journal, November 1998


Scalable Modeling of Carbon Tetrachloride Migration at the Hanford Site Using the STOMP Simulator
journal, January 2008

  • White, M. D.; Oostrom, M.; Rockhold, M. L.
  • Vadose Zone Journal, Vol. 7, Issue 2
  • DOI: 10.2136/vzj2007.0070

HBGC123D: a high-performance computer model of coupled hydrogeological and biogeochemical processes
journal, December 2001


Expressing object-oriented concepts in Fortran 90
journal, April 1997

  • Decyk, Viktor K.; Norton, Charles D.; Szymanski, Boleslaw K.
  • ACM SIGPLAN Fortran Forum, Vol. 16, Issue 1
  • DOI: 10.1145/263877.263880

Supercomputers and their use in modeling subsurface solute transport
journal, January 1990


Three-dimensional Bayesian geostatistical aquifer characterization at the Hanford 300 Area using tracer test data: INVERSION OF TRACER DATA USING MAD
journal, June 2012

  • Chen, Xingyuan; Murakami, Haruko; Hahn, Melanie S.
  • Water Resources Research, Vol. 48, Issue 6
  • DOI: 10.1029/2011WR010675

Elucidating geochemical response of shallow heterogeneous aquifers to CO2 leakage using high-performance computing: Implications for monitoring of CO2 sequestration
journal, March 2013


Variably saturated flow and multicomponent biogeochemical reactive transport modeling of a uranium bioremediation field experiment
journal, November 2011

  • Yabusaki, Steven B.; Fang, Yilin; Williams, Kenneth H.
  • Journal of Contaminant Hydrology, Vol. 126, Issue 3-4
  • DOI: 10.1016/j.jconhyd.2011.09.002

Application of Jacobian-free Newton–Krylov with physics-based preconditioning to biogeochemical transport
journal, April 2005


Newton–Krylov-multigrid solvers for large-scale, highly heterogeneous, variably saturated flow problems
journal, July 2001


The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q
conference, January 2003

  • Petrini, Fabrizio; Kerbyson, Darren J.; Pakin, Scott
  • Proceedings of the 2003 ACM/IEEE conference on Supercomputing - SC '03
  • DOI: 10.1145/1048935.1050204

Achieving high sustained performance in an unstructured mesh CFD application
conference, January 1999

  • Anderson, W. K.; Gropp, W. D.; Kaushik, D. K.
  • Proceedings of the 1999 ACM/IEEE conference on Supercomputing (CDROM) - Supercomputing '99
  • DOI: 10.1145/331532.331600

High Resolution Forward And Inverse Earthquake Modeling on Terascale Computers
conference, January 2003

  • Akcelik, Vokan; Tu, Tiankai; Urbanic, John
  • Proceedings of the 2003 ACM/IEEE conference on Supercomputing - SC '03
  • DOI: 10.1145/1048935.1050202

Guidance for Choosing Multigrid Preconditioners for Systems of Elliptic Partial Differential Equations
journal, January 2009

  • Lee, B.
  • SIAM Journal on Scientific Computing, Vol. 31, Issue 4
  • DOI: 10.1137/070703636

Stochastic simulation of uranium migration at the Hanford 300 Area
journal, March 2011


Modeling subsurface reactive flows using leadership-class computing
journal, July 2009


    Works referencing / citing this record:

    Modeling the spatiotemporal variability in subsurface thermal regimes across a low-relief polygonal tundra landscape
    journal, January 2016


    Modeling the spatiotemporal variability in subsurface thermal regimes across a low-relief polygonal tundra landscape
    journal, January 2016


    Detection of carbon dioxide leakage during injection in deep saline formations by pressure tomography: DETECT LEAKY CO 2 BY PRESSURE TOMOGRAPHY
    journal, July 2016

    • Hu, Linwei; Bayer, Peter; Brauchler, Ralf
    • Water Resources Research, Vol. 52, Issue 7
    • DOI: 10.1002/2015wr018420

    Understanding watershed hydrogeochemistry: 1. Development of RT-Flux-PIHM: DEVELOPMENT OF RT-FLUX-PIHM
    journal, March 2017

    • Bao, Chen; Li, Li; Shi, Yuning
    • Water Resources Research, Vol. 53, Issue 3
    • DOI: 10.1002/2016wr018934

    Riverbed Hydrologic Exchange Dynamics in a Large Regulated River Reach
    journal, April 2018

    • Zhou, Tian; Bao, Jie; Huang, Maoyi
    • Water Resources Research, Vol. 54, Issue 4
    • DOI: 10.1002/2017wr020508

    Numerical modelling of stream-aquifer interaction: Quantifying the impact of transient streambed permeability and aquifer heterogeneity
    journal, June 2018

    • Zhou, Dan; Zhang, Ye; Gianni, Guillaume
    • Hydrological Processes, Vol. 32, Issue 14
    • DOI: 10.1002/hyp.13169

    Review of numerical solution of Richardson–Richards equation for variably saturated flow in soils
    journal, June 2019

    • Zha, Yuanyuan; Yang, Jinzhong; Zeng, Jicai
    • Wiley Interdisciplinary Reviews: Water, Vol. 6, Issue 5
    • DOI: 10.1002/wat2.1364

    ParCrunchFlow: an efficient, parallel reactive transport simulation tool for physically and chemically heterogeneous saturated subsurface environments
    journal, April 2015

    • Beisman, James J.; Maxwell, Reed M.; Navarre-Sitchler, Alexis K.
    • Computational Geosciences, Vol. 19, Issue 2
    • DOI: 10.1007/s10596-015-9475-x

    Enhancing speed and scalability of the ParFlow simulation code
    journal, September 2017

    • Burstedde, Carsten; Fonseca, Jose A.; Kollet, Stefan
    • Computational Geosciences, Vol. 22, Issue 1
    • DOI: 10.1007/s10596-017-9696-2

    GEYSER: 3D thermo-hydrodynamic reactive transport numerical simulator including porosity and permeability evolution using GPU clusters
    journal, September 2019


    Simulating Oxygen Intrusion into Highly Heterogeneous Fractured Media Using High Performance Computing
    journal, January 2018

    • Trinchero, Paolo; Molinero, Jorge; Ebrahimi, Hedieh
    • Mathematical Geosciences, Vol. 50, Issue 5
    • DOI: 10.1007/s11004-017-9718-6

    Efficient Monte Carlo With Graph‐Based Subsurface Flow and Transport Models
    journal, May 2018

    • O'Malley, D.; Karra, S.; Hyman, J. D.
    • Water Resources Research, Vol. 54, Issue 5
    • DOI: 10.1029/2017wr022073

    Drought Conditions Maximize the Impact of High‐Frequency Flow Variations on Thermal Regimes and Biogeochemical Function in the Hyporheic Zone
    journal, October 2018

    • Song, Xuehang; Chen, Xingyuan; Stegen, James
    • Water Resources Research, Vol. 54, Issue 10
    • DOI: 10.1029/2018wr022586

    Delineating Facies Spatial Distribution by Integrating Ensemble Data Assimilationand Indicator Geostatistics With Level‐Set Transformation
    journal, April 2019

    • Song, Xuehang; Chen, Xingyuan; Ye, Ming
    • Water Resources Research, Vol. 55, Issue 4
    • DOI: 10.1029/2018wr023262

    Geochemical Exports to River From the Intrameander Hyporheic Zone Under Transient Hydrologic Conditions: East River Mountainous Watershed, Colorado
    journal, October 2018

    • Dwivedi, Dipankar; Steefel, Carl I.; Arora, Bhavna
    • Water Resources Research, Vol. 54, Issue 10
    • DOI: 10.1029/2018wr023377

    Using Bayesian Networks for Sensitivity Analysis of Complex Biogeochemical Models
    journal, April 2019

    • Dai, Heng; Chen, Xingyuan; Ye, Ming
    • Water Resources Research, Vol. 55, Issue 4
    • DOI: 10.1029/2018wr023589

    Can Convolution and Deconvolution Be Used as Tools for Modeling Multicomponent, Mixing‐Limited Reaction Networks?
    journal, July 2019

    • Engdahl, Nicholas B.; McCallum, James L.; Ginn, Timothy R.
    • Water Resources Research, Vol. 55, Issue 7
    • DOI: 10.1029/2018wr024103

    Dam Operations and Subsurface Hydrogeology Control Dynamics of Hydrologic Exchange Flows in a Regulated River Reach
    journal, April 2019

    • Shuai, Pin; Chen, Xingyuan; Song, Xuehang
    • Water Resources Research, Vol. 55, Issue 4
    • DOI: 10.1029/2018wr024193

    On the Field Estimation of Moisture Content Using Electrical Geophysics: The Impact of Petrophysical Model Uncertainty
    journal, August 2019

    • Tso, Chak‐Hau Michael; Kuras, Oliver; Binley, Andrew
    • Water Resources Research, Vol. 55, Issue 8
    • DOI: 10.1029/2019wr024964

    Sensitivity of constrained joint inversions to geological and petrophysical input data uncertainties with posterior geological analysis
    journal, March 2019

    • Giraud, Jérémie; Ogarko, Vitaliy; Lindsay, Mark
    • Geophysical Journal International, Vol. 218, Issue 1
    • DOI: 10.1093/gji/ggz152

    Extreme-scaling applications en route to exascale
    conference, January 2016

    • Brömmel, Dirk; Frings, Wolfgang; Wylie, Brian J. N.
    • Proceedings of the Exascale Applications and Software Conference 2016 on - EASC '16
    • DOI: 10.1145/2938615.2938616

    Representative paths analysis
    conference, November 2017

    • Tallent, Nathan R.; Kerbyson, Darren J.; Hoisie, Adolfy
    • SC '17: The International Conference for High Performance Computing, Networking, Storage and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
    • DOI: 10.1145/3126908.3126962

    Improvement of numerical approximation of coupled multiphase multicomponent flow with reactive geochemical transport in porous media
    journal, January 2018

    • Ahusborde, Etienne; Amaziane, Brahim; El Ossmani, Mustapha
    • Oil & Gas Science and Technology – Revue d’IFP Energies nouvelles, Vol. 73
    • DOI: 10.2516/ogst/2018033

    Coupling a three-dimensional subsurface flow and transport model with a land surface model to simulate stream–aquifer–land interactions (CP v1.0)
    journal, January 2017

    • Bisht, Gautam; Huang, Maoyi; Zhou, Tian
    • Geoscientific Model Development, Vol. 10, Issue 12
    • DOI: 10.5194/gmd-10-4539-2017

    A run control framework to streamline profiling, porting, and tuning simulation runs and provenance tracking of geoscientific applications
    journal, January 2018

    • Sharples, Wendy; Zhukov, Ilya; Geimer, Markus
    • Geoscientific Model Development, Vol. 11, Issue 7
    • DOI: 10.5194/gmd-11-2875-2018