Accelerating the Lagrangian Particle Tracking in Hydrologic Modeling to Continental-Scale

Yang, Chen; Ponder, Carl; Wang, Bei; Tran, Hoang V.; Zhang, Jun; Swilley, Jackson; Condon, Laura E.; Maxwell, Reed M.

doi:10.1029/2022ms003507

Title: Accelerating the Lagrangian Particle Tracking in Hydrologic Modeling to Continental-Scale

Journal Article · 10 May 2023 · Journal of Advances in Modeling Earth Systems

DOI: https://doi.org/10.1029/2022ms003507 · OSTI ID:1985896

^[1]; Ponder, Carl ^[2]; Wang, Bei ^[1]; Tran, Hoang V. ^[3]; Zhang, Jun ^[4]; Swilley, Jackson ^[1]; Condon, Laura E. ^[5];

^[1]

Princeton Univ., NJ (United States)
NVIDIA Developer Technology, Austin, TX (United States)
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
Nanjing Normal University (China)
Univ. of Arizona, Tucson, AZ (United States)

Unprecedented climate change and anthropogenic activities have induced increasing ecohydrological problems, which have motivated the development of large-scale modeling for solutions. Water age/quality is as important as water quantity for understanding the water cycle. However, current scientific progress in tracking water parcels at large-scale with high spatiotemporal resolutions is far behind that in water balance/quantity owing to the lack of powerful tools. EcoSLIM is a particle tracking model that works with the hydrologic model ParFlow-CLM, which couples surface-subsurface hydrology with land surface processes. Here, we demonstrate a parallel framework to accelerate EcoSLIM to continental-scale on a distributed, multi-GPU platform with CUDA-Aware MPI. In tests from catchment-, to regional-, and then to continental-scale using 25-million to 1.6-billion particles, EcoSLIM shows significant speedup and excellent parallel performance. The parallel framework is portable to atmospheric and oceanic particle tracking models, where parallelization is inadequate and a standard parallel framework is absent. Parallelized EcoSLIM is a promising tool to accelerate our understanding of the terrestrial water cycle and the upscaling of subsurface hydrology to Earth system models.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: National Natural Science Foundation of China (NSFC); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC05-76RL01830; AC02-05CH11231

OSTI ID:: 1985896

Report Number(s):: PNNL-SA-175029

Journal Information:: Journal of Advances in Modeling Earth Systems, Vol. 15, Issue 5; ISSN 1942-2466

Publisher:: American Geophysical Union (AGU)Copyright Statement

Country of Publication:: United States

Language:: English

References (45)

Towards European-scale convection-resolving climate simulations with GPUs: a study with COSMO 4.19 Leutwyler, David; Fuhrer, Oliver; Lapillonne, Xavier Geoscientific Model Development, Vol. 9, Issue 9 https://doi.org/10.5194/gmd-9-3393-2016	journal	January 2016
Solute Transport in Heterogeneous Soil with Time-Dependent Boundary Conditions Cremer, Clemens J. M.; Neuweiler, Insa; Bechtold, Michel Vadose Zone Journal, Vol. 15, Issue 6 https://doi.org/10.2136/vzj2015.11.0144	journal	June 2016
Parallel SPH modeling using dynamic domain decomposition and load balancing displacement of Voronoi subdomains Egorova, M. S.; Dyachkov, S. A.; Parshikov, A. N. Computer Physics Communications, Vol. 234 https://doi.org/10.1016/j.cpc.2018.07.019	journal	January 2019
Evaluation of oceanic and atmospheric trajectory schemes in the TRACMASS trajectory model v6.0 Döös, Kristofer; Jönsson, Bror; Kjellsson, Joakim Geoscientific Model Development, Vol. 10, Issue 4 https://doi.org/10.5194/gmd-10-1733-2017	journal	January 2017
Risk of groundwater contamination widely underestimated because of fast flow into aquifers Hartmann, Andreas; Jasechko, Scott; Gleeson, Tom Proceedings of the National Academy of Sciences, Vol. 118, Issue 20 https://doi.org/10.1073/pnas.2024492118	journal	May 2021
Iterative load-balancing method with multigrid level relaxation for particle simulation with short-range interactions Furuichi, Mikito; Nishiura, Daisuke Computer Physics Communications, Vol. 219 https://doi.org/10.1016/j.cpc.2017.05.015	journal	October 2017
The Lagrangian particle dispersion model FLEXPART version 10.4 Pisso, Ignacio; Sollum, Espen; Grythe, Henrik Geoscientific Model Development, Vol. 12, Issue 12 https://doi.org/10.5194/gmd-12-4955-2019	journal	January 2019
Leveraging HPC accelerator architectures with modern techniques — hydrologic modeling on GPUs with ParFlow Hokkanen, Jaro; Kollet, Stefan; Kraus, Jiri Computational Geosciences, Vol. 25, Issue 5 https://doi.org/10.1007/s10596-021-10051-4	journal	May 2021
A High-Resolution Land Model With Groundwater Lateral Flow, Water Use, and Soil Freeze-Thaw Front Dynamics and its Applications in an Endorheic Basin Xie, Zhenghui; Liu, Shuang; Zeng, Yujin Journal of Geophysical Research: Atmospheres https://doi.org/10.1029/2018jd028369	journal	July 2018
Accelerating the Lagrangian simulation of water ages on distributed, multi-GPU platforms: The importance of dynamic load balancing Yang, Chen; Maxwell, Reed M.; Valent, Richard Computers & Geosciences, Vol. 166 https://doi.org/10.1016/j.cageo.2022.105189	journal	September 2022
Effect of Root Water and Solute Uptake on Apparent Soil Dispersivity: A Simulation Study Schröder, Natalie; Javaux, Mathieu; Vanderborght, Jan Vadose Zone Journal, Vol. 11, Issue 3 https://doi.org/10.2136/vzj2012.0009	journal	August 2012
Fresh groundwater discharge insignificant for the world’s oceans but important for coastal ecosystems Luijendijk, Elco; Gleeson, Tom; Moosdorf, Nils Nature Communications, Vol. 11, Issue 1 https://doi.org/10.1038/s41467-020-15064-8	journal	March 2020
User guide for MODPATH Version 7—A particle-tracking model for MODFLOW Pollock, David W. https://doi.org/10.3133/ofr20161086	report	January 2016
ParCrunchFlow: an efficient, parallel reactive transport simulation tool for physically and chemically heterogeneous saturated subsurface environments Beisman, James J.; Maxwell, Reed M.; Navarre-Sitchler, Alexis K. Computational Geosciences, Vol. 19, Issue 2 https://doi.org/10.1007/s10596-015-9475-x	journal	April 2015
TRACMASS—A Lagrangian Trajectory Model Döös, Kristofer; Kjellsson, Joakim; Jönsson, Bror Preventive Methods for Coastal Protection https://doi.org/10.1007/978-3-319-00440-2_7	book	January 2013
Accelerating Streamline Tracking in Groundwater Flow Modeling on GPUs Ji, Xiaohui; Luo, Mulan; Wang, Xu‐Sheng Groundwater, Vol. 58, Issue 4 https://doi.org/10.1111/gwat.12959	journal	November 2019
The Demographics of Water: A Review of Water Ages in the Critical Zone Sprenger, Matthias; Stumpp, Christine; Weiler, Markus Reviews of Geophysics, Vol. 57, Issue 3 https://doi.org/10.1029/2018RG000633	journal	July 2019
Global Patterns of Groundwater Table Depth Fan, Y.; Li, H.; Miguez-Macho, G. Science, Vol. 339, Issue 6122 https://doi.org/10.1126/science.1229881	journal	February 2013
An GPU-accelerated particle tracking method for Eulerian–Lagrangian simulations using hardware ray tracing cores Wang, Bin; Wald, Ingo; Morrical, Nate Computer Physics Communications, Vol. 271 https://doi.org/10.1016/j.cpc.2021.108221	journal	February 2022
Quality matters for water scarcity van Vliet, Michelle T. H.; Flörke, Martina; Wada, Yoshihide Nature Geoscience, Vol. 10, Issue 11 https://doi.org/10.1038/ngeo3047	journal	October 2017
TRITON: A Multi-GPU open source 2D hydrodynamic flood model Morales-Hernández, M.; Sharif, Md B.; Kalyanapu, A. Environmental Modelling & Software, Vol. 141 https://doi.org/10.1016/j.envsoft.2021.105034	journal	July 2021
Fractal stream chemistry and its implications for contaminant transport in catchments Kirchner, James W.; Feng, Xiahong; Neal, Colin Nature, Vol. 403, Issue 6769 https://doi.org/10.1038/35000537	journal	February 2000
PAR 2 : Parallel Random Walk Particle Tracking Method for solute transport in porous media Rizzo, Calogero B.; Nakano, Aiichiro; de Barros, Felipe P. J. Computer Physics Communications, Vol. 239 https://doi.org/10.1016/j.cpc.2019.01.013	journal	June 2019
A high-resolution simulation of groundwater and surface water over most of the continental US with the integrated hydrologic model ParFlow v3 Maxwell, R. M.; Condon, L. E.; Kollet, S. J. Geoscientific Model Development, Vol. 8, Issue 3 https://doi.org/10.5194/gmd-8-923-2015	journal	January 2015
Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 GPUs with COSMO 5.0 Fuhrer, Oliver; Chadha, Tarun; Hoefler, Torsten Geoscientific Model Development, Vol. 11, Issue 4 https://doi.org/10.5194/gmd-11-1665-2018	journal	January 2018
A model comparison assessing the importance of lateral groundwater flows at the global scale de Graaf, Inge E. M.; Stahl, Kerstin Environmental Research Letters, Vol. 17, Issue 4 https://doi.org/10.1088/1748-9326/ac50d2	journal	March 2022
How much and how old? Fan, Ying Nature Geoscience, Vol. 9, Issue 2 https://doi.org/10.1038/ngeo2609	journal	November 2015
Using Particle Tracking to Understand Flow Paths, Age Distributions, and the Paradoxical Origins of the Inverse Storage Effect in an Experimental Catchment Wilusz, D. C.; Harman, C. J.; Ball, W. P. Water Resources Research, Vol. 56, Issue 4 https://doi.org/10.1029/2019wr025140	journal	April 2020
Sensitivity of Simulated Mountain Block Hydrology to Subsurface Conceptualization Rapp, G. A.; Condon, L. E.; Markovich, K. H. Water Resources Research, Vol. 56, Issue 10 https://doi.org/10.1029/2020WR027714	journal	October 2020
Beyond the water balance McDonnell, Jeffrey J. Nature Geoscience, Vol. 10, Issue 6 https://doi.org/10.1038/ngeo2964	journal	May 2017
Fractal topography and subsurface water flows from fluvial bedforms to the continental shield Wörman, Anders; Packman, Aaron I.; Marklund, Lars Geophysical Research Letters, Vol. 34, Issue 7 https://doi.org/10.1029/2007GL029426	journal	January 2007
Groundwater influence on soil moisture memory and land–atmosphere fluxes in the Iberian Peninsula Martínez-de la Torre, Alberto; Miguez-Macho, Gonzalo Hydrology and Earth System Sciences, Vol. 23, Issue 12 https://doi.org/10.5194/hess-23-4909-2019	journal	December 2019
Quantifying the effects of land use and model scale on water partitioning and water ages using tracer-aided ecohydrological models Smith, Aaron; Tetzlaff, Doerthe; Kleine, Lukas Hydrology and Earth System Sciences, Vol. 25, Issue 4 https://doi.org/10.5194/hess-25-2239-2021	journal	April 2021
Catchment Functioning Under Prolonged Drought Stress: Tracer‐Aided Ecohydrological Modeling in an Intensively Managed Agricultural Catchment Yang, Xiaoqiang; Tetzlaff, Doerthe; Soulsby, Chris Water Resources Research, Vol. 57, Issue 3 https://doi.org/10.1029/2020WR029094	journal	March 2021
Connections between groundwater flow and transpiration partitioning Maxwell, R. M.; Condon, L. E. Science, Vol. 353, Issue 6297 https://doi.org/10.1126/science.aaf7891	journal	July 2016
EcH ₂ O-iso 1.0: water isotopes and age tracking in a process-based, distributed ecohydrological model Kuppel, Sylvain; Tetzlaff, Doerthe; Maneta, Marco P. Geoscientific Model Development, Vol. 11, Issue 7 https://doi.org/10.5194/gmd-11-3045-2018	journal	January 2018
Critical Zone Storage Controls on the Water Ages of Ecohydrological Outputs Kuppel, S.; Tetzlaff, D.; Maneta, M. P. Geophysical Research Letters, Vol. 47, Issue 16 https://doi.org/10.1029/2020gl088897	journal	August 2020
The subsurface–land surface–atmosphere connection under convective conditions Rahman, M.; Sulis, M.; Kollet, S. J. Advances in Water Resources, Vol. 83 https://doi.org/10.1016/j.advwatres.2015.06.003	journal	September 2015
Exploring source water mixing and transient residence time distributions of outflow and evapotranspiration with an integrated hydrologic model and Lagrangian particle tracking approach: Source water mixing and transient residence time distributions ET and Q. Maxwell, Reed M.; Condon, Laura E.; Danesh-Yazdi, Mohammad Ecohydrology, Vol. 12, Issue 1 https://doi.org/10.1002/eco.2042	journal	October 2018
Numerical simulation of solute transport in three-dimensional, randomly heterogeneous porous media Tompson, Andrew F. B.; Gelhar, Lynn W. Water Resources Research, Vol. 26, Issue 10 https://doi.org/10.1029/WR026i010p02541	journal	October 1990
Twenty-three unsolved problems in hydrology (UPH) – a community perspective Blöschl, Günter; Bierkens, Marc F. P.; Chambel, Antonio Hydrological Sciences Journal, Vol. 64, Issue 10 https://doi.org/10.1080/02626667.2019.1620507	journal	July 2019
Lagrangian ocean analysis: Fundamentals and practices van Sebille, Erik; Griffies, Stephen M.; Abernathey, Ryan Ocean Modelling, Vol. 121 https://doi.org/10.1016/j.ocemod.2017.11.008	journal	January 2018
Parallelization of MODFLOW Using a GPU Library Ji, Xiaohui; Li, Dandan; Cheng, Tangpei Groundwater, Vol. 52, Issue 4 https://doi.org/10.1111/gwat.12104	journal	August 2013
Prevalence and magnitude of groundwater use by vegetation: a global stable isotope meta-analysis Evaristo, Jaivime; McDonnell, Jeffrey J. Scientific Reports, Vol. 7, Issue 1 https://doi.org/10.1038/srep44110	journal	March 2017
Accelerating the Lagrangian particle tracking of residence time distributions and source water mixing towards large scales Yang, Chen; Zhang, You-Kuan; Liang, Xiuyu Computers & Geosciences, Vol. 151 https://doi.org/10.1016/j.cageo.2021.104760	journal	June 2021

Similar Records

ParFlow Sand Tank: A tool for groundwater exploration

Journal Article · 2023 · The Journal of Open Source Education (JOSE) · OSTI ID:2375799

Condon, Laura E.; Farley, Abram; Jourdain, Sebastien; +5 more

Matrix Diffusion Controls Mountain Hillslope Groundwater Ages and Inferred Storage Dynamics

Journal Article · 2025 · Ground Water · OSTI ID:2543161

Thiros, Nicholas E.; Woodburn, Erica R.; Gardner, W. Payton; +2 more

Old-Aged groundwater contributes to mountain hillslope hydrologic dynamics

Journal Article · 2024 · Journal of Hydrology · OSTI ID:2336814

Thiros, Nicholas E.; Siirila-Woodburn, Erica R.; Sprenger, Matthias; +4 more

Related Subjects

54 ENVIRONMENTAL SCIENCES

Title: Accelerating the Lagrangian Particle Tracking in Hydrologic Modeling to Continental-Scale

Citation Formats

References (45)

Similar Records

Related Subjects