A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics

Xia, Yidong; Blumers, Ansel; Li, Zhen; Luo, Lixiang; Tang, Yu-Hang; Kane, Joshua; Goral, Jan; Huang, Hai; Deo, Milind; Andrew, Matthew

doi:10.1016/j.cpc.2019.106874

Title: A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics

Journal Article · Thu Aug 29 00:00:00 EDT 2019 · Computer Physics Communications

DOI:https://doi.org/10.1016/j.cpc.2019.106874· OSTI ID:1597088

Xia, Yidong ^[1]; Blumers, Ansel ^[2]; Li, Zhen ^[3]; Luo, Lixiang ^[4]; Tang, Yu-Hang ^[5];

^[1]; Goral, Jan ^[6]; Huang, Hai ^[1]; Deo, Milind ^[6]; Andrew, Matthew ^[7]

Idaho National Lab. (INL), Idaho Falls, ID (United States)
Idaho National Lab. (INL), Idaho Falls, ID (United States); Brown Univ., Providence, RI (United States)
Clemson Univ., SC (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Univ. of Utah, Salt Lake City, UT (United States)
Carl Zeiss X-ray Microscopy, Inc., Pleasanton, CA (United States)

Mesoscopic simulations of hydrocarbon flow in source shales are concerning, in part due to the heterogeneous shale pores with sizes ranging from a few nanometers to a few micrometers. Additionally, the sub-continuum fluid–fluid and fluid–solid interactions in nano- to micro-scale shale pores, which are physically and chemically sophisticated, must be captured. To address those challenges, we present a GPU-accelerated package for simulation of flow in nano- to micro-pore networks with a many-body dissipative particle dynamics (mDPD) mesoscale model. Based on a fully distributed parallel paradigm, the code offloads all intensive workloads on GPUs. Other advancements, such as smart particle packing and no-slip boundary condition in complex pore geometries, are also implemented for the construction and the simulation of the realistic shale pores from 3D nanometer-resolution stack images. Our code is validated for accuracy and compared against the CPU counterpart for speedup. In our benchmark tests, the code delivers nearly perfect strong scaling and weak scaling (with up to 512 million particles) on up to 512 K20X GPUs on Oak Ridge National Laboratory’s (ORNL) Titan supercomputer. Moreover, a single-GPU benchmark on ORNL’s SummitDev and IBM’s AC922 suggests that the host-to-device NVLink can boost performance over PCIe by a remarkable 40%. Lastly, we demonstrate, through a flow simulation in realistic shale pores, that the CPU counterpart requires 840 Power9 cores to rival the performance delivered by our package with four V100 GPUs on ORNL’s Summit architecture. This simulation package enables quick-turnaround and high-throughput mesoscopic numerical simulations for exploring complex flow phenomena in nano- to micro-porous rocks with realistic pore geometries.

View Accepted Manuscript (DOE)

View Accepted Manuscript (Publisher)

Cite

Export

Save

Research Organization:: Energy Frontier Research Centers (EFRC) (United States). Multi-Scale Fluid-Solid Interactions in Architected and Natural Materials (MUSE); Univ. of Utah, Salt Lake City, UT (United States); Idaho National Laboratory (INL), Idaho Falls, ID (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Organization:: USDOE Office of Science (SC), Basic Energy Sciences (BES); USDOE Office of Nuclear Energy (NE)

Grant/Contract Number:: SC0019285; AC07-05ID14517; AC05-00OR22725

OSTI ID:: 1597088

Alternate ID(s):: OSTI ID: 1575938

Report Number(s):: INL/JOU-19-52933; TRN: US2103024

Journal Information:: Computer Physics Communications, Vol. 247, Issue C; ISSN 0010-4655

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 17 works

Citation information provided by
Web of Science

References (46)

Simulating Microscopic Hydrodynamic Phenomena with Dissipative Particle Dynamics Hoogerbrugge, P. J.; Koelman, J. M. V. A. Europhysics Letters (EPL), Vol. 19, Issue 3 https://doi.org/10.1209/0295-5075/19/3/001	journal	June 1992
Energy-conserving dissipative particle dynamics with temperature-dependent properties Li, Zhen; Tang, Yu-Hang; Lei, Huan Journal of Computational Physics, Vol. 265 https://doi.org/10.1016/j.jcp.2014.02.003	journal	May 2014
A dissipative particle dynamics method for arbitrarily complex geometries Li, Zhen; Bian, Xin; Tang, Yu-Hang Journal of Computational Physics, Vol. 355 https://doi.org/10.1016/j.jcp.2017.11.014	journal	February 2018
An adhesive DPD wall model for dynamic wetting Henrich, B.; Cupelli, C.; Moseler, M. Europhysics Letters (EPL), Vol. 80, Issue 6 https://doi.org/10.1209/0295-5075/80/60004	journal	November 2007
Effective Fluid Front of the Moving Meniscus in Capillary Chen, Chen; Lu, Kangjie; Zhuang, Lin Langmuir, Vol. 29, Issue 10 https://doi.org/10.1021/la304598h	journal	February 2013
Interannual co-variability of tropical temperature and humidity: A comparison of model, reanalysis data and satellite observation: TROPICAL TEMPERATURE-HUMIDITY VARIATIONS Huang, Xianglei; Soden, Brian J.; Jackson, Darren L. Geophysical Research Letters, Vol. 32, Issue 17 https://doi.org/10.1029/2005GL023375	journal	September 2005
New multi-GPU implementation for smoothed particle hydrodynamics on heterogeneous clusters Domínguez, J. M.; Crespo, A. J. C.; Valdez-Balderas, D. Computer Physics Communications, Vol. 184, Issue 8 https://doi.org/10.1016/j.cpc.2013.03.008	journal	August 2013
Accelerating dissipative particle dynamics simulations on GPUs: Algorithms, numerics and applications Tang, Yu-Hang; Karniadakis, George Em Computer Physics Communications, Vol. 185, Issue 11 https://doi.org/10.1016/j.cpc.2014.06.015	journal	November 2014
Fast Parallel Algorithms for Short-Range Molecular Dynamics Plimpton, Steve Journal of Computational Physics, Vol. 117, Issue 1 https://doi.org/10.1006/jcph.1995.1039	journal	March 1995
Dissipative particle dynamics: Bridging the gap between atomistic and mesoscopic simulation Groot, Robert D.; Warren, Patrick B. The Journal of Chemical Physics, Vol. 107, Issue 11 https://doi.org/10.1063/1.474784	journal	September 1997
Microfracturing during primary migration in shales Teixeira, Marcello Goulart; Donzé, Frédéric; Renard, François Tectonophysics, Vol. 694 https://doi.org/10.1016/j.tecto.2016.11.010	journal	January 2017
Many-body dissipative particle dynamics modeling of fluid flow in fine-grained nanoporous shales Xia, Yidong; Goral, Jan; Huang, Hai Physics of Fluids, Vol. 29, Issue 5 https://doi.org/10.1063/1.4981136	journal	May 2017
Pore scale modeling of immiscible and miscible fluid flows using smoothed particle hydrodynamics Tartakovsky, Alexandre M.; Meakin, Paul Advances in Water Resources, Vol. 29, Issue 10 https://doi.org/10.1016/j.advwatres.2005.11.014	journal	October 2006
Pore-scale modeling of saturated permeabilities in random sphere packings Pan, Chongxun; Hilpert, Markus; Miller, Cass T. Physical Review E, Vol. 64, Issue 6 https://doi.org/10.1103/PhysRevE.64.066702	journal	November 2001
GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers Abraham, Mark James; Murtola, Teemu; Schulz, Roland SoftwareX, Vol. 1-2 https://doi.org/10.1016/j.softx.2015.06.001	journal	September 2015
Modelling multi-viscosity systems with dissipative particle dynamics Visser, D. C.; Hoefsloot, H. C. J.; Iedema, P. D. Journal of Computational Physics, Vol. 214, Issue 2 https://doi.org/10.1016/j.jcp.2005.09.022	journal	May 2006
Sailfish: A flexible multi-GPU implementation of the lattice Boltzmann method Januszewski, M.; Kostur, M. Computer Physics Communications, Vol. 185, Issue 9 https://doi.org/10.1016/j.cpc.2014.04.018	journal	September 2014
Dissipative particle dynamics simulation of fluid motion through an unsaturated fracture and fracture junction Liu, Moubin; Meakin, Paul; Huang, Hai Journal of Computational Physics, Vol. 222, Issue 1 https://doi.org/10.1016/j.jcp.2006.07.017	journal	March 2007
Fluid expulsion and microfracturing during the pyrolysis of an organic rich shale Panahi, Hamed; Kobchenko, Maya; Meakin, Paul Fuel, Vol. 235 https://doi.org/10.1016/j.fuel.2018.07.069	journal	January 2019
Toward a Coarse Graining/All Atoms Force Field (CG/AA) from a Multiscale Optimization Method: An Application to the MCM-41 Mesoporous Silicates Ghoufi, A.; Morineau, D.; Lefort, R. Journal of Chemical Theory and Computation, Vol. 6, Issue 10 https://doi.org/10.1021/ct100169r	journal	September 2010
Dynamic and transport properties of dissipative particle dynamics with energy conservation Avalos, J. Bonet; Mackie, A. D. The Journal of Chemical Physics, Vol. 111, Issue 11 https://doi.org/10.1063/1.479780	journal	September 1999
Boundary conditions in dissipative particle dynamics Revenga, M.; Zúñiga, I.; Español, P. Computer Physics Communications, Vol. 121-122 https://doi.org/10.1016/S0010-4655(99)00341-0	journal	September 1999
Dissipative Particle Dynamics: Introduction, Methodology and Complex Fluid Applications — a Review Moeendarbary, E.; Ng, T. Y.; Zangeneh, M. International Journal of Applied Mechanics, Vol. 01, Issue 04 https://doi.org/10.1142/S1758825109000381	journal	December 2009
Dissipative Particle Dynamics (DPD): An Overview and Recent Developments Liu, M. B.; Liu, G. R.; Zhou, L. W. Archives of Computational Methods in Engineering, Vol. 22, Issue 4 https://doi.org/10.1007/s11831-014-9124-x	journal	July 2014
Optimization strategies for CPU and GPU implementations of a smoothed particle hydrodynamics method Domínguez, Jose M.; Crespo, Alejandro J. C.; Gómez-Gesteira, Moncho Computer Physics Communications, Vol. 184, Issue 3 https://doi.org/10.1016/j.cpc.2012.10.015	journal	March 2013
Massively parallel lattice–Boltzmann codes on large GPU clusters Calore, E.; Gabbana, A.; Kraus, J. Parallel Computing, Vol. 58 https://doi.org/10.1016/j.parco.2016.08.005	journal	October 2016
Implementing molecular dynamics on hybrid high performance computers – short range forces Brown, W. Michael; Wang, Peng; Plimpton, Steven J. Computer Physics Communications, Vol. 182, Issue 4 https://doi.org/10.1016/j.cpc.2010.12.021	journal	April 2011
Statistical Mechanics of Dissipative Particle Dynamics Español, P.; Warren, P. Europhysics Letters (EPL), Vol. 30, Issue 4 https://doi.org/10.1209/0295-5075/30/4/001	journal	May 1995
Particle methods for simulation of subsurface multiphase fluid flow and biogeochemical processes Meakin, Paul; Tartakovsky, Alexandre; Scheibe, Tim Journal of Physics: Conference Series, Vol. 78 https://doi.org/10.1088/1742-6596/78/1/012047	journal	July 2007
Dissipative particle dynamics with energy conservation Español, P. Europhysics Letters (EPL), Vol. 40, Issue 6 https://doi.org/10.1209/epl/i1997-00515-8	journal	December 1997
Mesoscale modeling of the water liquid-vapor interface: A surface tension calculation Ghoufi, A.; Malfreyt, P. Physical Review E, Vol. 83, Issue 5 https://doi.org/10.1103/PhysRevE.83.051601	journal	May 2011
A new method to impose no-slip boundary conditions in dissipative particle dynamics Pivkin, Igor V.; Karniadakis, George Em Journal of Computational Physics, Vol. 207, Issue 1 https://doi.org/10.1016/j.jcp.2005.01.006	journal	July 2005
A smoothed particle hydrodynamics model for miscible flow in three-dimensional fractures and the two-dimensional Rayleigh–Taylor instability Tartakovsky, Alexandre M.; Meakin, Paul Journal of Computational Physics, Vol. 207, Issue 2 https://doi.org/10.1016/j.jcp.2005.02.001	journal	August 2005
Dissipative Particle Dynamics with Energy Conservation: Heat Conduction Ripoll, M.; Español, P.; Ernst, M. H. International Journal of Modern Physics C, Vol. 09, Issue 08 https://doi.org/10.1142/S0129183198001205	journal	December 1998
Development of organic porosity in the Woodford Shale with increasing thermal maturity Curtis, Mark E.; Cardott, Brian J.; Sondergeld, Carl H. International Journal of Coal Geology, Vol. 103 https://doi.org/10.1016/j.coal.2012.08.004	journal	December 2012
Strong scaling of general-purpose molecular dynamics simulations on GPUs Glaser, Jens; Nguyen, Trung Dac; Anderson, Joshua A. Computer Physics Communications, Vol. 192 https://doi.org/10.1016/j.cpc.2015.02.028	journal	July 2015
Dissipative-particle-dynamics model for two-phase flows Tiwari, Anupam; Abraham, John Physical Review E, Vol. 74, Issue 5 https://doi.org/10.1103/PhysRevE.74.056701	journal	November 2006
A Many-Body Dissipative Particle Dynamics Study of Forced Water–Oil Displacement in Capillary Chen, Chen; Zhuang, Lin; Li, Xuefeng Langmuir, Vol. 28, Issue 2 https://doi.org/10.1021/la204207s	journal	December 2011
GPU-accelerated red blood cells simulations with transport dissipative particle dynamics Blumers, Ansel L.; Tang, Yu-Hang; Li, Zhen Computer Physics Communications, Vol. 217 https://doi.org/10.1016/j.cpc.2017.03.016	journal	August 2017
Dissipative particle dynamics with attractive and repulsive particle-particle interactions Liu, Moubin; Meakin, Paul; Huang, Hai Physics of Fluids, Vol. 18, Issue 1 https://doi.org/10.1063/1.2163366	journal	January 2006
Three-dimensional pore networks and transport properties of a shale gas formation determined from focused ion beam serial imaging Dewers, Thomas A.; Heath, Jason; Ewy, Russ International Journal of Oil, Gas and Coal Technology, Vol. 5, Issue 2/3 https://doi.org/10.1504/IJOGCT.2012.046322	journal	January 2012
Microstructural investigation of gas shales in two and three dimensions using nanometer-scale resolution imaging Curtis, Mark E.; Sondergeld, Carl H.; Ambrose, Raymond J. AAPG Bulletin, Vol. 96, Issue 4 https://doi.org/10.1306/08151110188	journal	April 2012
A many-body dissipative particle dynamics study of fluid–fluid spontaneous capillary displacement Chen, Chen; Lu, Kangjie; Li, Xuefeng RSC Advances, Vol. 4, Issue 13 https://doi.org/10.1039/c3ra47275b	journal	January 2014
A new GPU implementation for lattice-Boltzmann simulations on sparse geometries Tomczak, Tadeusz; Szafran, Roman G. Computer Physics Communications, Vol. 235 https://doi.org/10.1016/j.cpc.2018.04.031	journal	February 2019
GPU-accelerated adaptive particle splitting and merging in SPH Xiong, Qingang; Li, Bo; Xu, Ji Computer Physics Communications, Vol. 184, Issue 7 https://doi.org/10.1016/j.cpc.2013.02.021	journal	July 2013
Lattice-Boltzmann simulation of two-phase flow in porous media: LB SIMULATION OF TWO-PHASE FLOW Pan, C.; Hilpert, M.; Miller, C. T. Water Resources Research, Vol. 40, Issue 1 https://doi.org/10.1029/2003wr002120	journal	January 2004

Cited By (2)

Confinement Effect on Porosity and Permeability of Shales Goral, Jan; Panja, Palash; Deo, Milind Scientific Reports, Vol. 10, Issue 1 https://doi.org/10.1038/s41598-019-56885-y	journal	January 2020
HPC, Cloud and Big-Data Convergent Architectures: The LEXIS Approach Scionti, Alberto; Martinovic, Jan; Terzo, Olivier Complex, Intelligent, and Software Intensive Systems: Proceedings of the 13th International Conference on Complex, Intelligent, and Software Intensive Systems (CISIS-2019), p. 200-212 https://doi.org/10.1007/978-3-030-22354-0_19	book	June 2019

Linked Research (2)

Similar Records

Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

Journal Article · Wed Jan 01 00:00:00 EST 2020 · IEEE Transactions on Parallel and Distributed Systems · OSTI ID:1597088

Li, Ang; Song, Shuaiwen; Chen, Jieyang; +4 more

GPU-Centric Communication on NVIDIA GPU Clusters with InfiniBand: A Case Study with OpenSHMEM

Conference · Fri Dec 01 00:00:00 EST 2017 · OSTI ID:1597088

Potluri, Sreeram; Goswami, Anshuman; Rossetti, Davide; +3 more

A prospect for computing in porous materials research: Very large fluid flow simulations

Journal Article · Fri Jan 01 00:00:00 EST 2016 · Journal of Computational Science · OSTI ID:1597088

Mattila, Keijo; Puurtinen, Tuomas; Hyväluoma, Jari; +6 more

Related Subjects

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS
97 MATHEMATICS AND COMPUTING
Digital rock physics
Shale
GPU
Dissipative particle dynamics
Multiphase flow

A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics Xia, Yidong https://doi.org/10.17632/zzpv74bz9m.1 The current record is supplemented by this dataset	dataset	January 2019
A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics Xia, Yidong https://doi.org/10.17632/zzpv74bz9m The current record is supplemented by this dataset	dataset	January 2019