Development and performance of a HemeLB GPU code for human-scale blood flow simulation

Zacharoudiou, I.; McCullough, J. W.S.; Coveney, P. V.

doi:10.1016/j.cpc.2022.108548

Development and performance of a HemeLB GPU code for human-scale blood flow simulation

Journal Article · Thu Sep 22 00:00:00 EDT 2022 · Computer Physics Communications

DOI:https://doi.org/10.1016/j.cpc.2022.108548· OSTI ID:2424007

Zacharoudiou, I. ^[1]; ^[1]; Coveney, P. V. ^[2]

Univ. College London (United Kingdom)
Univ. College London (United Kingdom); Univ. of Amsterdam (Netherlands)

In recent years, it has become increasingly common for high performance computers (HPC) to possess some level of heterogeneous architecture - typically in the form of GPU accelerators. In some machines these are isolated within a dedicated partition, whilst in others they are integral to all compute nodes - often with multiple GPUs per node - and provide the majority of a machine's compute performance. In light of this trend, it is becoming essential that codes deployed on HPC are updated to execute on accelerator hardware. Here, in this paper, we introduce a GPU implementation of the 3D blood flow simulation code HemeLB that has been developed using CUDA C++. We demonstrate how taking advantage of NVIDIA GPU hardware can achieve significant performance improvements compared to the equivalent CPU only code on which it has been built whilst retaining the excellent strong scaling characteristics that have been repeatedly demonstrated by the CPU version. With HPC positioned on the brink of the exascale era, we use HemeLB as a motivation to provide a discussion on some of the challenges that many users will face when deploying their own applications on upcoming exascale machines.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 2424007

Journal Information:: Computer Physics Communications, Journal Name: Computer Physics Communications Vol. 282; ISSN 0010-4655

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (37)

A global multiscale mathematical model for the human circulation with emphasis on the venous system Müller, Lucas O.; Toro, Eleuterio F. International Journal for Numerical Methods in Biomedical Engineering, Vol. 30, Issue 7 https://doi.org/10.1002/cnm.2622	journal	January 2014
A flexible high-performance Lattice Boltzmann GPU code for the simulations of fluid flows in complex geometries Bernaschi, Massimo; Fatica, Massimiliano; Melchionna, Simone Concurrency and Computation: Practice and Experience, Vol. 22, Issue 1 https://doi.org/10.1002/cpe.1466	journal	January 2010
Heterogeneous CPU+GPU approaches for mesh refinement over Lattice‐Boltzmann simulations Valero‐Lara, Pedro; Jansson, Johan Concurrency and Computation: Practice and Experience, Vol. 29, Issue 7 https://doi.org/10.1002/cpe.3919	journal	August 2016
Computational simulation of blood flow in human systemic circulation incorporating an external force field Sheng, C.; Sarwal, S. N.; Watts, K. C. Medical & Biological Engineering & Computing, Vol. 33, Issue 1 https://doi.org/10.1007/BF02522938	journal	January 1995
Numerical simulation of blood flow and pressure drop in the pulmonary arterial and venous circulation Qureshi, M. Umar; Vaughan, Gareth D. A.; Sainsbury, Christopher Biomechanics and Modeling in Mechanobiology, Vol. 13, Issue 5 https://doi.org/10.1007/s10237-014-0563-y	journal	March 2014
One-Dimensional Haemodynamic Modeling and Wave Dynamics in the Entire Adult Circulation Mynard, Jonathan P.; Smolich, Joseph J. Annals of Biomedical Engineering, Vol. 43, Issue 6 https://doi.org/10.1007/s10439-015-1313-8	journal	April 2015
Multi-GPU implementation of the lattice Boltzmann method Obrecht, Christian; Kuznik, Frédéric; Tourancheau, Bernard Computers & Mathematics with Applications, Vol. 65, Issue 2 https://doi.org/10.1016/j.camwa.2011.02.020	journal	January 2013
Multi-thread implementations of the lattice Boltzmann method on non-uniform grids for CPUs and GPUs Schönherr, M.; Kucher, K.; Geier, M. Computers & Mathematics with Applications, Vol. 61, Issue 12 https://doi.org/10.1016/j.camwa.2011.04.012	journal	June 2011
waLBerla: A block-structured high-performance framework for multiphysics simulations Bauer, Martin; Eibl, Sebastian; Godenschwager, Christian Computers & Mathematics with Applications, Vol. 81 https://doi.org/10.1016/j.camwa.2020.01.007	journal	January 2021
Palabos: Parallel Lattice Boltzmann Solver Latt, Jonas; Malaspinas, Orestis; Kontaxakis, Dimitrios Computers & Mathematics with Applications, Vol. 81 https://doi.org/10.1016/j.camwa.2020.03.022	journal	January 2021
An evaluation of lattice Boltzmann schemes for porous medium flow simulation Pan, Chongxun; Luo, Li-Shi; Miller, Cass T. Computers & Fluids, Vol. 35, Issue 8-9 https://doi.org/10.1016/j.compfluid.2005.03.008	journal	September 2006
Performance engineering for the lattice Boltzmann method on GPGPUs: Architectural requirements and performance results Habich, J.; Feichtinger, C.; Köstler, H. Computers & Fluids, Vol. 80 https://doi.org/10.1016/j.compfluid.2012.02.013	journal	July 2013
HemeLB: A high performance parallel lattice-Boltzmann code for large scale fluid flow in complex geometries Mazzeo, M. D.; Coveney, P. V. Computer Physics Communications, Vol. 178, Issue 12 https://doi.org/10.1016/j.cpc.2008.02.013	journal	June 2008
Hydrokinetic approach to large-scale cardiovascular blood flow Melchionna, Simone; Bernaschi, Massimo; Succi, Sauro Computer Physics Communications, Vol. 181, Issue 3 https://doi.org/10.1016/j.cpc.2009.10.017	journal	March 2010
Memory transfer optimization for a lattice Boltzmann solver on Kepler architecture nVidia GPUs Mawson, Mark J.; Revell, Alistair J. Computer Physics Communications, Vol. 185, Issue 10 https://doi.org/10.1016/j.cpc.2014.06.003	journal	October 2014
The influence of vessel wall elasticity and peripheral resistance on the carotid artery flow wave form: A CFD model compared to in vivo ultrasound measurements Maurits, N. M.; Loots, G. E.; Veldman, A. E. P. Journal of Biomechanics, Vol. 40, Issue 2 https://doi.org/10.1016/j.jbiomech.2005.12.008	journal	January 2007
Multi-scale computational model of three-dimensional hemodynamics within a deformable full-body arterial network Xiao, Nan; Humphrey, Jay D.; Figueroa, C. Alberto Journal of Computational Physics, Vol. 244 https://doi.org/10.1016/j.jcp.2012.09.016	journal	July 2013
WaLBerla: HPC software design for computational engineering simulations Feichtinger, C.; Donath, S.; Köstler, H. Journal of Computational Science, Vol. 2, Issue 2 https://doi.org/10.1016/j.jocs.2011.01.004	journal	May 2011
Analysing and modelling the performance of the HemeLB lattice-Boltzmann simulation environment Groen, Derek; Hetherington, James; Carver, Hywel B. Journal of Computational Science, Vol. 4, Issue 5 https://doi.org/10.1016/j.jocs.2013.03.002	journal	September 2013
Massively parallel simulations of hemodynamics in the primary large arteries of the human vasculature Randles, Amanda; Draeger, Erik W.; Bailey, Peter E. Journal of Computational Science, Vol. 9 https://doi.org/10.1016/j.jocs.2015.04.003	journal	July 2015
A prospect for computing in porous materials research: Very large fluid flow simulations Mattila, Keijo; Puurtinen, Tuomas; Hyväluoma, Jari Journal of Computational Science, Vol. 12 https://doi.org/10.1016/j.jocs.2015.11.013	journal	January 2016
Massively parallel lattice–Boltzmann codes on large GPU clusters Calore, E.; Gabbana, A.; Kraus, J. Parallel Computing, Vol. 58 https://doi.org/10.1016/j.parco.2016.08.005	journal	October 2016
A Lattice-Boltzmann solver for 3D fluid simulation on GPU Rinaldi, P. R.; Dari, E. A.; Vénere, M. J. Simulation Modelling Practice and Theory, Vol. 25 https://doi.org/10.1016/j.simpat.2012.03.004	journal	June 2012
Numerical simulations of particulate suspensions via a discretized Boltzmann equation. Part 1. Theoretical foundation Ladd, Anthony J. C. Journal of Fluid Mechanics, Vol. 271 https://doi.org/10.1017/S0022112094001771	journal	July 1994
Lattice-Boltzmann Simulations of Particle-Fluid Suspensions Ladd, A. J. C.; Verberg, R. Journal of Statistical Physics, Vol. 104, Issue 5/6, p. 1191-1251 https://doi.org/10.1023/A:1010414013942	journal	January 2001
Accelerating massively parallel hemodynamic models of coarctation of the aorta using neural networks Feiger, Bradley; Gounley, John; Adler, Dale Scientific Reports, Vol. 10, Issue 1 https://doi.org/10.1038/s41598-020-66225-0	journal	June 2020
Plasma Hsp90 levels in patients with systemic sclerosis and relation to lung and skin involvement: a cross-sectional and longitudinal study Štorkánová, Hana; Oreská, Sabína; Špiritović, Maja Scientific Reports, Vol. 11, Issue 1 https://doi.org/10.1038/s41598-020-79139-8	journal	January 2021
Towards blood flow in the virtual human: efficient self-coupling of HemeLB McCullough, J. W. S.; Richardson, R. A.; Patronis, A. Interface Focus, Vol. 11, Issue 1 https://doi.org/10.1098/rsfs.2019.0119	journal	December 2020
A Model for Collision Processes in Gases. I. Small Amplitude Processes in Charged and Neutral One-Component Systems Bhatnagar, P. L.; Gross, E. P.; Krook, M. Physical Review, Vol. 94, Issue 3 https://doi.org/10.1103/PhysRev.94.511	journal	May 1954
Theory of the lattice Boltzmann method: Lattice Boltzmann models for nonideal gases Luo, Li-Shi Physical Review E, Vol. 62, Issue 4 https://doi.org/10.1103/PhysRevE.62.4982	journal	October 2000
Choice of boundary condition for lattice-Boltzmann simulation of moderate-Reynolds-number flow in complex domains Nash, Rupert W.; Carver, Hywel B.; Bernabeu, Miguel O. Physical Review E, Vol. 89, Issue 2 https://doi.org/10.1103/PhysRevE.89.023303	journal	February 2014
Analysis of GPU Data Access Patterns on Complex Geometries for the D3Q19 Lattice Boltzmann Algorithm Herschlag, Gregory; Lee, Seyong; Vetter, Jeffrey S. IEEE Transactions on Parallel and Distributed Systems, Vol. 32, Issue 10 https://doi.org/10.1109/TPDS.2021.3061895	journal	October 2021
Optimization of lattice Boltzmann simulations on heterogeneous computers Calore, E.; Gabbana, A.; Schifano, Sf The International Journal of High Performance Computing Applications, Vol. 33, Issue 1 https://doi.org/10.1177/1094342017703771	journal	April 2017
Review of Zero-D and 1-D Models of Blood Flow in the Cardiovascular System Shi, Yubing; Lawford, Patricia; Hose, Rodney BioMedical Engineering OnLine, Vol. 10, Issue 1 https://doi.org/10.1186/1475-925X-10-33	journal	January 2011
Cross-platform programming model for many-core lattice Boltzmann simulations Latt, Jonas; Coreixas, Christophe; Beny, Joël PLOS ONE, Vol. 16, Issue 4 https://doi.org/10.1371/journal.pone.0250306	journal	April 2021
Validation of Patient-Specific Cerebral Blood Flow Simulation Using Transcranial Doppler Measurements Groen, Derek; Richardson, Robin A.; Coy, Rachel Frontiers in Physiology, Vol. 9 https://doi.org/10.3389/fphys.2018.00721	journal	June 2018
Multi-GPU Based Lattice Boltzmann Method for Hemodynamic Simulation in Patient-Specific Cerebral Aneurysm Huang, Changsheng; Shi, Baochang; Guo, Zhaoli Communications in Computational Physics, Vol. 17, Issue 4 https://doi.org/10.4208/cicp.2014.m342	journal	April 2015