DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics

Abstract

Mesoscopic simulations of hydrocarbon flow in source shales are concerning, in part due to the heterogeneous shale pores with sizes ranging from a few nanometers to a few micrometers. Additionally, the sub-continuum fluid–fluid and fluid–solid interactions in nano- to micro-scale shale pores, which are physically and chemically sophisticated, must be captured. To address those challenges, we present a GPU-accelerated package for simulation of flow in nano- to micro-pore networks with a many-body dissipative particle dynamics (mDPD) mesoscale model. Based on a fully distributed parallel paradigm, the code offloads all intensive workloads on GPUs. Other advancements, such as smart particle packing and no-slip boundary condition in complex pore geometries, are also implemented for the construction and the simulation of the realistic shale pores from 3D nanometer-resolution stack images. Our code is validated for accuracy and compared against the CPU counterpart for speedup. In our benchmark tests, the code delivers nearly perfect strong scaling and weak scaling (with up to 512 million particles) on up to 512 K20X GPUs on Oak Ridge National Laboratory’s (ORNL) Titan supercomputer. Moreover, a single-GPU benchmark on ORNL’s SummitDev and IBM’s AC922 suggests that the host-to-device NVLink can boost performance over PCIe by a remarkable 40%. Lastly,more » we demonstrate, through a flow simulation in realistic shale pores, that the CPU counterpart requires 840 Power9 cores to rival the performance delivered by our package with four V100 GPUs on ORNL’s Summit architecture. This simulation package enables quick-turnaround and high-throughput mesoscopic numerical simulations for exploring complex flow phenomena in nano- to micro-porous rocks with realistic pore geometries.« less

Authors:
 [1];  [2];  [3];  [4];  [5]; ORCiD logo [1];  [6];  [1];  [6];  [7]
  1. Idaho National Lab. (INL), Idaho Falls, ID (United States)
  2. Idaho National Lab. (INL), Idaho Falls, ID (United States); Brown Univ., Providence, RI (United States)
  3. Clemson Univ., SC (United States)
  4. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
  5. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  6. Univ. of Utah, Salt Lake City, UT (United States)
  7. Carl Zeiss X-ray Microscopy, Inc., Pleasanton, CA (United States)
Publication Date:
Research Org.:
Energy Frontier Research Centers (EFRC) (United States). Multi-Scale Fluid-Solid Interactions in Architected and Natural Materials (MUSE); Univ. of Utah, Salt Lake City, UT (United States); Idaho National Laboratory (INL), Idaho Falls, ID (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Org.:
USDOE Office of Science (SC), Basic Energy Sciences (BES); USDOE Office of Nuclear Energy (NE)
OSTI Identifier:
1597088
Alternate Identifier(s):
OSTI ID: 1575938
Report Number(s):
INL/JOU-19-52933
Journal ID: ISSN 0010-4655; TRN: US2103024
Grant/Contract Number:  
SC0019285; AC07-05ID14517; AC05-00OR22725
Resource Type:
Accepted Manuscript
Journal Name:
Computer Physics Communications
Additional Journal Information:
Journal Volume: 247; Journal Issue: C; Journal ID: ISSN 0010-4655
Publisher:
Elsevier
Country of Publication:
United States
Language:
English
Subject:
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS; 97 MATHEMATICS AND COMPUTING; Digital rock physics; Shale; GPU; Dissipative particle dynamics; Multiphase flow

Citation Formats

Xia, Yidong, Blumers, Ansel, Li, Zhen, Luo, Lixiang, Tang, Yu-Hang, Kane, Joshua, Goral, Jan, Huang, Hai, Deo, Milind, and Andrew, Matthew. A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics. United States: N. p., 2019. Web. doi:10.1016/j.cpc.2019.106874.
Xia, Yidong, Blumers, Ansel, Li, Zhen, Luo, Lixiang, Tang, Yu-Hang, Kane, Joshua, Goral, Jan, Huang, Hai, Deo, Milind, & Andrew, Matthew. A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics. United States. https://doi.org/10.1016/j.cpc.2019.106874
Xia, Yidong, Blumers, Ansel, Li, Zhen, Luo, Lixiang, Tang, Yu-Hang, Kane, Joshua, Goral, Jan, Huang, Hai, Deo, Milind, and Andrew, Matthew. Thu . "A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics". United States. https://doi.org/10.1016/j.cpc.2019.106874. https://www.osti.gov/servlets/purl/1597088.
@article{osti_1597088,
title = {A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics},
author = {Xia, Yidong and Blumers, Ansel and Li, Zhen and Luo, Lixiang and Tang, Yu-Hang and Kane, Joshua and Goral, Jan and Huang, Hai and Deo, Milind and Andrew, Matthew},
abstractNote = {Mesoscopic simulations of hydrocarbon flow in source shales are concerning, in part due to the heterogeneous shale pores with sizes ranging from a few nanometers to a few micrometers. Additionally, the sub-continuum fluid–fluid and fluid–solid interactions in nano- to micro-scale shale pores, which are physically and chemically sophisticated, must be captured. To address those challenges, we present a GPU-accelerated package for simulation of flow in nano- to micro-pore networks with a many-body dissipative particle dynamics (mDPD) mesoscale model. Based on a fully distributed parallel paradigm, the code offloads all intensive workloads on GPUs. Other advancements, such as smart particle packing and no-slip boundary condition in complex pore geometries, are also implemented for the construction and the simulation of the realistic shale pores from 3D nanometer-resolution stack images. Our code is validated for accuracy and compared against the CPU counterpart for speedup. In our benchmark tests, the code delivers nearly perfect strong scaling and weak scaling (with up to 512 million particles) on up to 512 K20X GPUs on Oak Ridge National Laboratory’s (ORNL) Titan supercomputer. Moreover, a single-GPU benchmark on ORNL’s SummitDev and IBM’s AC922 suggests that the host-to-device NVLink can boost performance over PCIe by a remarkable 40%. Lastly, we demonstrate, through a flow simulation in realistic shale pores, that the CPU counterpart requires 840 Power9 cores to rival the performance delivered by our package with four V100 GPUs on ORNL’s Summit architecture. This simulation package enables quick-turnaround and high-throughput mesoscopic numerical simulations for exploring complex flow phenomena in nano- to micro-porous rocks with realistic pore geometries.},
doi = {10.1016/j.cpc.2019.106874},
journal = {Computer Physics Communications},
number = C,
volume = 247,
place = {United States},
year = {Thu Aug 29 00:00:00 EDT 2019},
month = {Thu Aug 29 00:00:00 EDT 2019}
}

Journal Article:

Citation Metrics:
Cited by: 17 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Simulating Microscopic Hydrodynamic Phenomena with Dissipative Particle Dynamics
journal, June 1992


Energy-conserving dissipative particle dynamics with temperature-dependent properties
journal, May 2014


A dissipative particle dynamics method for arbitrarily complex geometries
journal, February 2018


An adhesive DPD wall model for dynamic wetting
journal, November 2007


Effective Fluid Front of the Moving Meniscus in Capillary
journal, February 2013

  • Chen, Chen; Lu, Kangjie; Zhuang, Lin
  • Langmuir, Vol. 29, Issue 10
  • DOI: 10.1021/la304598h

Interannual co-variability of tropical temperature and humidity: A comparison of model, reanalysis data and satellite observation: TROPICAL TEMPERATURE-HUMIDITY VARIATIONS
journal, September 2005

  • Huang, Xianglei; Soden, Brian J.; Jackson, Darren L.
  • Geophysical Research Letters, Vol. 32, Issue 17
  • DOI: 10.1029/2005GL023375

New multi-GPU implementation for smoothed particle hydrodynamics on heterogeneous clusters
journal, August 2013

  • Domínguez, J. M.; Crespo, A. J. C.; Valdez-Balderas, D.
  • Computer Physics Communications, Vol. 184, Issue 8
  • DOI: 10.1016/j.cpc.2013.03.008

Accelerating dissipative particle dynamics simulations on GPUs: Algorithms, numerics and applications
journal, November 2014


Fast Parallel Algorithms for Short-Range Molecular Dynamics
journal, March 1995


Dissipative particle dynamics: Bridging the gap between atomistic and mesoscopic simulation
journal, September 1997

  • Groot, Robert D.; Warren, Patrick B.
  • The Journal of Chemical Physics, Vol. 107, Issue 11
  • DOI: 10.1063/1.474784

Microfracturing during primary migration in shales
journal, January 2017


Many-body dissipative particle dynamics modeling of fluid flow in fine-grained nanoporous shales
journal, May 2017

  • Xia, Yidong; Goral, Jan; Huang, Hai
  • Physics of Fluids, Vol. 29, Issue 5
  • DOI: 10.1063/1.4981136

Pore scale modeling of immiscible and miscible fluid flows using smoothed particle hydrodynamics
journal, October 2006


Pore-scale modeling of saturated permeabilities in random sphere packings
journal, November 2001


GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers
journal, September 2015


Modelling multi-viscosity systems with dissipative particle dynamics
journal, May 2006

  • Visser, D. C.; Hoefsloot, H. C. J.; Iedema, P. D.
  • Journal of Computational Physics, Vol. 214, Issue 2
  • DOI: 10.1016/j.jcp.2005.09.022

Sailfish: A flexible multi-GPU implementation of the lattice Boltzmann method
journal, September 2014


Dissipative particle dynamics simulation of fluid motion through an unsaturated fracture and fracture junction
journal, March 2007


Fluid expulsion and microfracturing during the pyrolysis of an organic rich shale
journal, January 2019


Toward a Coarse Graining/All Atoms Force Field (CG/AA) from a Multiscale Optimization Method: An Application to the MCM-41 Mesoporous Silicates
journal, September 2010

  • Ghoufi, A.; Morineau, D.; Lefort, R.
  • Journal of Chemical Theory and Computation, Vol. 6, Issue 10
  • DOI: 10.1021/ct100169r

Dynamic and transport properties of dissipative particle dynamics with energy conservation
journal, September 1999

  • Avalos, J. Bonet; Mackie, A. D.
  • The Journal of Chemical Physics, Vol. 111, Issue 11
  • DOI: 10.1063/1.479780

Boundary conditions in dissipative particle dynamics
journal, September 1999


Dissipative Particle Dynamics: Introduction, Methodology and Complex Fluid Applications — a Review
journal, December 2009

  • Moeendarbary, E.; Ng, T. Y.; Zangeneh, M.
  • International Journal of Applied Mechanics, Vol. 01, Issue 04
  • DOI: 10.1142/S1758825109000381

Dissipative Particle Dynamics (DPD): An Overview and Recent Developments
journal, July 2014

  • Liu, M. B.; Liu, G. R.; Zhou, L. W.
  • Archives of Computational Methods in Engineering, Vol. 22, Issue 4
  • DOI: 10.1007/s11831-014-9124-x

Optimization strategies for CPU and GPU implementations of a smoothed particle hydrodynamics method
journal, March 2013

  • Domínguez, Jose M.; Crespo, Alejandro J. C.; Gómez-Gesteira, Moncho
  • Computer Physics Communications, Vol. 184, Issue 3
  • DOI: 10.1016/j.cpc.2012.10.015

Massively parallel lattice–Boltzmann codes on large GPU clusters
journal, October 2016


Implementing molecular dynamics on hybrid high performance computers – short range forces
journal, April 2011

  • Brown, W. Michael; Wang, Peng; Plimpton, Steven J.
  • Computer Physics Communications, Vol. 182, Issue 4
  • DOI: 10.1016/j.cpc.2010.12.021

Statistical Mechanics of Dissipative Particle Dynamics
journal, May 1995


Particle methods for simulation of subsurface multiphase fluid flow and biogeochemical processes
journal, July 2007


Dissipative particle dynamics with energy conservation
journal, December 1997


Mesoscale modeling of the water liquid-vapor interface: A surface tension calculation
journal, May 2011


A new method to impose no-slip boundary conditions in dissipative particle dynamics
journal, July 2005


A smoothed particle hydrodynamics model for miscible flow in three-dimensional fractures and the two-dimensional Rayleigh–Taylor instability
journal, August 2005


Dissipative Particle Dynamics with Energy Conservation: Heat Conduction
journal, December 1998

  • Ripoll, M.; Español, P.; Ernst, M. H.
  • International Journal of Modern Physics C, Vol. 09, Issue 08
  • DOI: 10.1142/S0129183198001205

Development of organic porosity in the Woodford Shale with increasing thermal maturity
journal, December 2012

  • Curtis, Mark E.; Cardott, Brian J.; Sondergeld, Carl H.
  • International Journal of Coal Geology, Vol. 103
  • DOI: 10.1016/j.coal.2012.08.004

Strong scaling of general-purpose molecular dynamics simulations on GPUs
journal, July 2015

  • Glaser, Jens; Nguyen, Trung Dac; Anderson, Joshua A.
  • Computer Physics Communications, Vol. 192
  • DOI: 10.1016/j.cpc.2015.02.028

Dissipative-particle-dynamics model for two-phase flows
journal, November 2006


A Many-Body Dissipative Particle Dynamics Study of Forced Water–Oil Displacement in Capillary
journal, December 2011

  • Chen, Chen; Zhuang, Lin; Li, Xuefeng
  • Langmuir, Vol. 28, Issue 2
  • DOI: 10.1021/la204207s

GPU-accelerated red blood cells simulations with transport dissipative particle dynamics
journal, August 2017


Dissipative particle dynamics with attractive and repulsive particle-particle interactions
journal, January 2006

  • Liu, Moubin; Meakin, Paul; Huang, Hai
  • Physics of Fluids, Vol. 18, Issue 1
  • DOI: 10.1063/1.2163366

Three-dimensional pore networks and transport properties of a shale gas formation determined from focused ion beam serial imaging
journal, January 2012

  • Dewers, Thomas A.; Heath, Jason; Ewy, Russ
  • International Journal of Oil, Gas and Coal Technology, Vol. 5, Issue 2/3
  • DOI: 10.1504/IJOGCT.2012.046322

Microstructural investigation of gas shales in two and three dimensions using nanometer-scale resolution imaging
journal, April 2012

  • Curtis, Mark E.; Sondergeld, Carl H.; Ambrose, Raymond J.
  • AAPG Bulletin, Vol. 96, Issue 4
  • DOI: 10.1306/08151110188

A many-body dissipative particle dynamics study of fluid–fluid spontaneous capillary displacement
journal, January 2014

  • Chen, Chen; Lu, Kangjie; Li, Xuefeng
  • RSC Advances, Vol. 4, Issue 13
  • DOI: 10.1039/c3ra47275b

A new GPU implementation for lattice-Boltzmann simulations on sparse geometries
journal, February 2019


GPU-accelerated adaptive particle splitting and merging in SPH
journal, July 2013


A new GPU implementation for lattice-Boltzmann simulations on sparse geometries
journal, February 2019


Toward a Coarse Graining/All Atoms Force Field (CG/AA) from a Multiscale Optimization Method: An Application to the MCM-41 Mesoporous Silicates
journal, September 2010

  • Ghoufi, A.; Morineau, D.; Lefort, R.
  • Journal of Chemical Theory and Computation, Vol. 6, Issue 10
  • DOI: 10.1021/ct100169r

Lattice-Boltzmann simulation of two-phase flow in porous media: LB SIMULATION OF TWO-PHASE FLOW
journal, January 2004

  • Pan, C.; Hilpert, M.; Miller, C. T.
  • Water Resources Research, Vol. 40, Issue 1
  • DOI: 10.1029/2003wr002120

Simulating Microscopic Hydrodynamic Phenomena with Dissipative Particle Dynamics
journal, June 1992


Works referencing / citing this record:

Confinement Effect on Porosity and Permeability of Shales
journal, January 2020


HPC, Cloud and Big-Data Convergent Architectures: The LEXIS Approach
book, June 2019

  • Scionti, Alberto; Martinovic, Jan; Terzo, Olivier
  • Complex, Intelligent, and Software Intensive Systems: Proceedings of the 13th International Conference on Complex, Intelligent, and Software Intensive Systems (CISIS-2019), p. 200-212
  • DOI: 10.1007/978-3-030-22354-0_19