Multi-GPU immersed boundary method hemodynamics simulations

Ames, Jeff; Puleri, Daniel F.; Balogh, Peter; Gounley, John; Draeger, Erik W.; Randles, Amanda

doi:10.1016/j.jocs.2020.101153

Multi-GPU immersed boundary method hemodynamics simulations

Journal Article · 14 June 2020 · Journal of Computational Science

DOI:https://doi.org/10.1016/j.jocs.2020.101153· OSTI ID:1649415

Ames, Jeff ^[1]; ^[2]; Balogh, Peter ^[2]; Gounley, John ^[3]; Draeger, Erik W. ^[4]; Randles, Amanda ^[2]

Duke Univ., Durham, NC (United States). Dept. of Computer Science
Duke Univ., Durham, NC (United States). Dept. of Biomedical Engineering
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Computational Sciences and Engineering Division
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States). Center for Applied Scientific Computing

Large-scale simulations of blood flow that resolve the 3D deformation of each comprising cell are increasingly popular owing to algorithmic developments in conjunction with advances in compute capability. Among different approaches for modeling cell-resolved hemodynamics, fluid structure interaction (FSI) algorithms based on the immersed boundary method are frequently employed for coupling separate solvers for the background fluid and the cells within one framework. GPUs can accelerate these simulations; however, both current pre-exascale and future exascale CPU-GPU heterogeneous systems face communication challenges critical to performance and scalability. In this paper, we describe, to our knowledge, the largest distributed GPU-accelerated FSI simulations of high hematocrit cell-resolved flows with over 17 million red blood cells. We compare scaling on a fat node system with six GPUs per node and on a system with a single GPU per node. Through comparison between the CPU- and GPU-based implementations, we identify the costs of data movement in multiscale multi-grid FSI simulations on heterogeneous systems and show it to be the greatest performance bottleneck on the GPU.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: National Institutes of Health (NIH); USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC)

Grant/Contract Number:: AC05-00OR22725; AC52-07NA27344

OSTI ID:: 1649415

Alternate ID(s):: OSTI ID: 1738914

Report Number(s):: LLNL-JRNL--817821

Journal Information:: Journal of Computational Science, Journal Name: Journal of Computational Science Vol. 44; ISSN 1877-7503

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (37)

Subdivision surfaces: a new paradigm for thin-shell finite-element analysis Cirak, Fehmi; Ortiz, Michael; Schr�der, Peter International Journal for Numerical Methods in Engineering, Vol. 47, Issue 12 https://doi.org/10.1002/(SICI)1097-0207(20000430)47:12<2039::AID-NME872>3.0.CO;2-1	journal	April 2000
Coupling of finite element and boundary integral methods for a capsule in a Stokes flow: NUMERICAL METHODS FOR A CAPSULE IN A STOKES FLOW Walter, J.; Salsac, A. -V.; Barthès-Biesel, D. International Journal for Numerical Methods in Engineering, Vol. 83, Issue 7 https://doi.org/10.1002/nme.2859	journal	March 2010
Study on the packed volume-to-void ratio of idealized human red blood cells using a finite-discrete element method Xu, Dong; Ji, Chunning; Munjiza, A. Applied Mathematics and Mechanics, Vol. 40, Issue 5 https://doi.org/10.1007/s10483-019-2473-6	journal	April 2019
A nonlinear programming model with implicit variables for packing ellipsoids Birgin, E. G.; Lobato, R. D.; Martínez, J. M. Journal of Global Optimization, Vol. 68, Issue 3 https://doi.org/10.1007/s10898-016-0483-8	journal	December 2016
Transparent partial page migration between CPU and GPU Zhang, Shiqing; Qin, Zheng; Yang, Yaohua Frontiers of Computer Science, Vol. 14, Issue 3 https://doi.org/10.1007/s11704-018-7386-4	journal	December 2019
Strain Energy Function of Red Blood Cell Membranes Skalak, R.; Tozeren, A.; Zarda, R. P. Biophysical Journal, Vol. 13, Issue 3 https://doi.org/10.1016/S0006-3495(73)85983-1	journal	March 1973
Efficient and accurate simulations of deformable particles immersed in a fluid using a combined immersed boundary lattice Boltzmann finite element method Krüger, T.; Varnik, F.; Raabe, D. Computers & Mathematics with Applications, Vol. 61, Issue 12 https://doi.org/10.1016/j.camwa.2010.03.057	journal	June 2011
Multi-GPU implementation of the lattice Boltzmann method Obrecht, Christian; Kuznik, Frédéric; Tourancheau, Bernard Computers & Mathematics with Applications, Vol. 65, Issue 2 https://doi.org/10.1016/j.camwa.2011.02.020	journal	January 2013
GPU acceleration of FSI simulations by the immersed boundary-lattice Boltzmann coupling scheme Wu, Jiayang; Cheng, Yongguang; Zhou, Wei Computers & Mathematics with Applications, Vol. 78, Issue 4 https://doi.org/10.1016/j.camwa.2016.10.005	journal	August 2019
On optimal node spacing for immersed boundary–lattice Boltzmann method in 2D and 3D Fučík, Radek; Eichler, Pavel; Straka, Robert Computers & Mathematics with Applications, Vol. 77, Issue 4 https://doi.org/10.1016/j.camwa.2018.10.045	journal	February 2019
On the single processor performance of simple lattice Boltzmann kernels Wellein, G.; Zeiser, T.; Hager, G. Computers & Fluids, Vol. 35, Issue 8-9 https://doi.org/10.1016/j.compfluid.2005.02.008	journal	September 2006
Performance engineering for the lattice Boltzmann method on GPGPUs: Architectural requirements and performance results Habich, J.; Feichtinger, C.; Köstler, H. Computers & Fluids, Vol. 80 https://doi.org/10.1016/j.compfluid.2012.02.013	journal	July 2013
A new GPU implementation for lattice-Boltzmann simulations on sparse geometries Tomczak, Tadeusz; Szafran, Roman G. Computer Physics Communications, Vol. 235 https://doi.org/10.1016/j.cpc.2018.04.031	journal	February 2019
Numerical methods for simulating blood flow at macro, micro, and multi scales Imai, Yohsuke; Omori, Toshihiro; Shimogonya, Yuji Journal of Biomechanics, Vol. 49, Issue 11 https://doi.org/10.1016/j.jbiomech.2015.11.047	journal	July 2016
Isogeometric FEM-BEM simulations of drop, capsule and vesicle dynamics in Stokes flow Boedec, Gwenn; Leonetti, Marc; Jaeger, Marc Journal of Computational Physics, Vol. 342 https://doi.org/10.1016/j.jcp.2017.04.024	journal	August 2017
Parallel performance of an IB-LBM suspension simulation framework Mountrakis, Lampros; Lorenz, Eric; Malaspinas, Orestis Journal of Computational Science, Vol. 9 https://doi.org/10.1016/j.jocs.2015.04.006	journal	July 2015
Accelerating fluid–solid simulations (Lattice-Boltzmann & Immersed-Boundary) on heterogeneous architectures Valero-Lara, Pedro; Igual, Francisco D.; Prieto-Matías, Manuel Journal of Computational Science, Vol. 10 https://doi.org/10.1016/j.jocs.2015.07.002	journal	September 2015
Load balancing of parallel cell-based blood flow simulations Alowayyed, S.; Závodszky, G.; Azizi, V. Journal of Computational Science, Vol. 24 https://doi.org/10.1016/j.jocs.2017.11.008	journal	January 2018
Massively parallel lattice–Boltzmann codes on large GPU clusters Calore, E.; Gabbana, A.; Kraus, J. Parallel Computing, Vol. 58 https://doi.org/10.1016/j.parco.2016.08.005	journal	October 2016
Free-Surface Lattice-Boltzmann Simulation on Many-Core Architectures Schreiber, Martin; Neumann, Philipp; Zimmer, Stefan Procedia Computer Science, Vol. 4 https://doi.org/10.1016/j.procs.2011.04.104	journal	January 2011
An Accelerated Iterative Linear Solver with GPUs for CFD Calculations of Unstructured Grids Williams, Justin; Sarofeen, Christian; Shan, Hau Procedia Computer Science, Vol. 80 https://doi.org/10.1016/j.procs.2016.05.504	journal	January 2016
Hemocell: a high-performance microscopic cellular library Zavodszky, Gábor; van Rooij, Britt; Azizi, Victor Procedia Computer Science, Vol. 108 https://doi.org/10.1016/j.procs.2017.05.084	journal	January 2017
Numerical simulation of a compound capsule in a constricted microchannel Gounley, John; Draeger, Erik W.; Randles, Amanda Procedia Computer Science, Vol. 108 https://doi.org/10.1016/j.procs.2017.05.209	journal	January 2017
The immersed boundary method Peskin, Charles S. Acta Numerica, Vol. 11 https://doi.org/10.1017/S0962492902000077	journal	January 2002
Shared-Memory Parallel Vector Implementation of the Immersed Boundary Method for the Computation of Blood Flow in the Beating Mammalian Heart McQueen, David; Peskin, Charles The Journal of Supercomputing, Vol. 11, Issue 3, p. 213-236 https://doi.org/10.1023/A:1007951707260	journal	November 1997
Quantifying the biophysical characteristics of Plasmodium-falciparum -parasitized red blood cells in microcirculation Fedosov, D. A.; Caswell, B.; Suresh, S. Proceedings of the National Academy of Sciences, Vol. 108, Issue 1 https://doi.org/10.1073/pnas.1009492108	journal	December 2010
Patient-specific blood rheology in sickle-cell anaemia Li, Xuejin; Du, E.; Lei, Huan Interface Focus, Vol. 6, Issue 1 https://doi.org/10.1098/rsfs.2015.0065	journal	February 2016
Three-dimensional multi-scale model of deformable platelets adhesion to vessel wall in blood flow Wu, Ziheng; Xu, Zhiliang; Kim, Oleg Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 372, Issue 2021 https://doi.org/10.1098/rsta.2013.0380	journal	August 2014
Discrete lattice effects on the forcing term in the lattice Boltzmann method Guo, Zhaoli; Zheng, Chuguang; Shi, Baochang Physical Review E, Vol. 65, Issue 4 https://doi.org/10.1103/PhysRevE.65.046308	journal	April 2002
Effect of bending stiffness on the deformation of liquid capsules enclosed by thin shells in shear flow Le, Duc Vinh Physical Review E, Vol. 82, Issue 1 https://doi.org/10.1103/PhysRevE.82.016318	journal	July 2010
Toward Full GPU Implementation of Fluid-Structure Interaction Beny, Joel; Kotsalos, Christos; Latt, Jonas 2019 18th International Symposium on Parallel and Distributed Computing (ISPDC) https://doi.org/10.1109/ISPDC.2019.000-2	conference	June 2019
3.5-D Blocking Optimization for Stencil Computations on Modern CPUs and GPUs Nguyen, Anthony; Satish, Nadathur; Chhugani, Jatin 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2010.2	conference	November 2010
Massively Parallel Lattice-Boltzmann Simulation of Turbulent Channel Flow Amati, Giorgio; Succi, Sauro; Piva, Renzo International Journal of Modern Physics C, Vol. 08, Issue 04 https://doi.org/10.1142/S0129183197000746	journal	August 1997
Lattice Boltzmann Method for Fluid Flows Chen, Shiyi; Doolen, Gary D. Annual Review of Fluid Mechanics, Vol. 30, Issue 1 https://doi.org/10.1146/annurev.fluid.30.1.329	journal	January 1998
Immersed Boundary Methods Mittal, Rajat; Iaccarino, Gianluca Annual Review of Fluid Mechanics, Vol. 37, Issue 1 https://doi.org/10.1146/annurev.fluid.37.061903.175743	journal	January 2005
Erythrocyte Membrane Elasticity and Viscosity Hochmuth, R. M.; Waugh, R. E. Annual Review of Physiology, Vol. 49, Issue 1 https://doi.org/10.1146/annurev.ph.49.030187.001233	journal	March 1987
Large deformation finite element analysis of non-linear viscoelastic membranes with reference to thermoforming Shrivastava, S.; Tang, J. The Journal of Strain Analysis for Engineering Design, Vol. 28, Issue 1 https://doi.org/10.1243/03093247V281031	journal	January 1993

Similar Records

Immersed Boundary Method Halo Exchange in a Hemodynamics Application

Conference · 2019 · OSTI ID:1526390

Development and performance of a HemeLB GPU code for human-scale blood flow simulation

Journal Article · 2022 · Computer Physics Communications · OSTI ID:2424007

Performance Evaluation of Heterogeneous GPU Programming Frameworks for Hemodynamic Simulations

Conference · 2023 · OSTI ID:2224199

Related Subjects

97 MATHEMATICS AND COMPUTING
GPU
distributed parallelization
fluid structure interaction
immersed boundary method
lattice boltzmann method

Multi-GPU immersed boundary method hemodynamics simulations

Citation Formats

References (37)

Similar Records

Related Subjects