Memory access optimization for particle operations in computational fluid dynamics-discrete element method simulations

Vaidhynathan, Deepthi; Sitaraman, Hariswaran; Grout, Ray; Hauser, Thomas; Hrenya, Christine M.; Musser, Jordan

doi:10.1016/j.partic.2022.10.004

Title: Memory access optimization for particle operations in computational fluid dynamics-discrete element method simulations

Journal Article · Tue Nov 08 00:00:00 EST 2022 · Particuology

DOI:https://doi.org/10.1016/j.partic.2022.10.004· OSTI ID:1906320

^[1];

^[1]; Grout, Ray ^[1]; Hauser, Thomas ^[2]; Hrenya, Christine M. ^[2]; Musser, Jordan ^[3]

National Renewable Energy Laboratory (NREL), Golden, CO (United States)
University of Colorado, Boulder, CO (United States)
National Energy Technology Laboratory (NETL), Pittsburgh, PA, Morgantown, WV, and Albany, OR (United States)

Computational Fluid Dynamics - Discrete Element Method is used to model gas-solid systems in several applications in energy, pharmaceutical and petrochemical industries. Computational performance bottlenecks often limit the problem sizes that can be simulated at industrial scale. The data structures used to store several millions of particles in such large-scale simulations have a large memory footprint that does not fit into the processor cache hierarchies on current high-performance-computing platforms, leading to reduced computational performance. This paper specifically addresses this aspect of memory access bottlenecks in industrial scale simulations. The use of space-filling curves to improve memory access patterns is described and their impact on computational performance is quantified in both shared and distributed memory parallelization paradigms. The Morton space filling curve applied to uniform grids and k-dimensional tree partitions are used to reorder the particle data-structure thus improving spatial and temporal locality in memory. The performance impact of these techniques when applied to two benchmark problems, namely the homogeneous-cooling-system and a fluidized-bed, are presented. We report these optimization techniques lead to approximately two-fold performance improvement in particle focused operations such as neighbor-list creation and data-exchange, with ~ 1.5 times overall improvement in a fluidization simulation with 1.27 million particles.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: National Renewable Energy Laboratory (NREL), Golden, CO (United States)

Sponsoring Organization:: USDOE Office of Energy Efficiency and Renewable Energy (EERE); USDOE Office of Fossil Energy (FE)

Grant/Contract Number:: AC36-08GO28308; FE0026298

OSTI ID:: 1906320

Report Number(s):: NREL/JA-2C00-79825; MainId:39043; UUID:d578def6-b8e9-40fc-938e-d42037e66dcc; MainAdminID:68281

Journal Information:: Particuology, Vol. 78; ISSN 1674-2001

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (25)

Drag force of intermediate Reynolds number flow past mono- and bidisperse arrays of spheres Beetstra, R.; van der Hoef, M. A.; Kuipers, J. A. M. AIChE Journal, Vol. 53, Issue 2 https://doi.org/10.1002/aic.11065	journal	January 2007
Multidimensional binary search trees used for associative searching Bentley, Jon Louis Communications of the ACM, Vol. 18, Issue 9 https://doi.org/10.1145/361002.361007	journal	September 1975
Renumbering unstructured grids to improve the performance of codes on hierarchical memory machines Burgess, D. A.; Giles, M. B. Advances in Engineering Software, Vol. 28, Issue 3 https://doi.org/10.1016/S0965-9978(96)00039-7	journal	April 1997
A bubbling fluidization model using kinetic theory of granular flow Ding, Jianmin; Gidaspow, Dimitri AIChE Journal, Vol. 36, Issue 4 https://doi.org/10.1002/aic.690360404	journal	April 1990
Performance Evaluation of Parallel Count Sort using GPU Computing with CUDA Faujdar, Neetu; Ghrera, SatyaPrakash Indian Journal of Science and Technology, Vol. 9, Issue 15 https://doi.org/10.17485/ijst/2016/v9i15/80080	journal	May 2016
A GPU-based DEM approach for modelling of particulate systems Gan, J. Q.; Zhou, Z. Y.; Yu, A. B. Powder Technology, Vol. 301 https://doi.org/10.1016/j.powtec.2016.07.072	journal	November 2016
An Algorithm for Reducing the Bandwidth and Profile of a Sparse Matrix Gibbs, Norman E.; Poole, Jr., William G.; Stockmeyer, Paul K. SIAM Journal on Numerical Analysis, Vol. 13, Issue 2 https://doi.org/10.1137/0713023	journal	April 1976
Grain flow as a fluid-mechanical phenomenon Haff, P. K. Journal of Fluid Mechanics, Vol. 134, Issue -1 https://doi.org/10.1017/S0022112083003419	journal	September 1983
Advanced parallelization strategies using hybrid MPI-CUDA octree DSMC method for modeling flow through porous media Jambunathan, Revathi; Levin, Deborah A. Computers & Fluids, Vol. 149 https://doi.org/10.1016/j.compfluid.2017.02.020	journal	June 2017
The NETL MFiX Suite of multiphase flow models: A brief review and recent applications of MFiX-TFM to fossil energy Technologies Li, Tingwen; Rogers, William A.; Syamlal, Madhava Chemical Engineering Science, Vol. 169 https://doi.org/10.1016/j.ces.2016.07.043	journal	September 2017
Some useful renumbering strategies for unstructured grids Löhner, Rainald International Journal for Numerical Methods in Engineering, Vol. 36, Issue 19 https://doi.org/10.1002/nme.1620361904	journal	October 1993
The Homogeneous Cooling State as a Verification Test for Kinetic Theory-Based Continuum Models of Gas–Solid Flows Fullmer, William D.; Hrenya, Christine M. Journal of Verification, Validation and Uncertainty Quantification, Vol. 2, Issue 4 https://doi.org/10.1115/1.4038916	journal	December 2017
An error-controlled adaptive time-stepping method for particle advancement in coupled CFD-DEM simulations Sitaraman, Hariswaran; Vaidhynathan, Deepthi; Grout, Ray Powder Technology, Vol. 379 https://doi.org/10.1016/j.powtec.2020.10.051	journal	February 2021
Granular temperature: Comparison of Magnetic Resonance measurements with Discrete Element Model simulations Müller, C. R.; Holland, D. J.; Sederman, A. J. Powder Technology, Vol. 184, Issue 2 https://doi.org/10.1016/j.powtec.2007.11.046	journal	May 2008
Validation of a discrete element model using magnetic resonance measurements Müller, Christoph R.; Scott, Stuart A.; Holland, Daniel J. Particuology, Vol. 7, Issue 4 https://doi.org/10.1016/j.partic.2009.04.002	journal	August 2009
Parallel-vector algorithms for particle simulations on shared-memory multiprocessors Nishiura, Daisuke; Sakaguchi, Hide Journal of Computational Physics, Vol. 230, Issue 5 https://doi.org/10.1016/j.jcp.2010.11.040	journal	March 2011
Computational fluid dynamics study of full-scale aerobic bioreactors: Evaluation of gas–liquid mass transfer, oxygen uptake, and dynamic oxygen distribution Rahimi, Mohammad J.; Sitaraman, Hariswaran; Humbird, David Chemical Engineering Research and Design, Vol. 139 https://doi.org/10.1016/j.cherd.2018.08.033	journal	November 2018
Balancing conflicting requirements for grid and particle decomposition in continuum-Lagrangian solvers Sitaraman, Hariswaran; Grout, Ray Parallel Computing, Vol. 52 https://doi.org/10.1016/j.parco.2015.10.010	journal	February 2016
A matrix free implicit scheme for solution of resistive magneto-hydrodynamics equations on unstructured grids Sitaraman, H.; Raja, L. L. Journal of Computational Physics, Vol. 251 https://doi.org/10.1016/j.jcp.2013.06.003	journal	October 2013
A Primer on Memory Consistency and Cache Coherence Nagarajan, Vijay; Sorin, Daniel J.; Hill, Mark D. Synthesis Lectures on Computer Architecture https://doi.org/10.1007/978-3-031-01764-3	book	January 2020
Cluster patterns in circulating fluidized beds predicted by numerical simulation (discrete particle model versus two-fluid model) Tsuji, Y.; Tanaka, T.; Yonemura, S. Powder Technology, Vol. 95, Issue 3 https://doi.org/10.1016/S0032-5910(97)03349-4	journal	March 1998
On a quasi-static discrete element model of granular materials Tzaferopoulos, M. Ap. Computers and Geotechnics, Vol. 18, Issue 2 https://doi.org/10.1016/0266-352X(95)00020-B	journal	January 1996
Morton ordering of 2D arrays for efficient access to hierarchical memory Walker, David W. The International Journal of High Performance Computing Applications, Vol. 32, Issue 1 https://doi.org/10.1177/1094342017725568	journal	August 2017
Roofline: an insightful visual performance model for multicore architectures Williams, Samuel; Waterman, Andrew; Patterson, David Communications of the ACM, Vol. 52, Issue 4 https://doi.org/10.1145/1498765.1498785	journal	April 2009
AMReX: a framework for block-structured adaptive mesh refinement Zhang, Weiqun; Almgren, Ann; Beckner, Vince Journal of Open Source Software, Vol. 4, Issue 37 https://doi.org/10.21105/joss.01370	journal	May 2019

Similar Records

Library API for Z-Order Memory Layout

Software · Sun Feb 01 00:00:00 EST 2015 · OSTI ID:1906320

Bethel, E. Wes

Roofline Analysis in the Intel® Advisor to Deliver Optimized Performance for applications on Intel® Xeon Phi™ Processor

Conference · Tue May 23 00:00:00 EDT 2017 · OSTI ID:1906320

Koskela, Tuomas S.; Lobet, Mathieu; Deslippe, Jack; +1 more

Access ordering and coherence in shared-memory multi-processors

Book · Sun Jan 01 00:00:00 EST 1989 · OSTI ID:1906320

Scheurich, C E

Related Subjects

97 MATHEMATICS AND COMPUTING
CFD-DEM
memory access optimization
performance optimization
spatial reordering

Title: Memory access optimization for particle operations in computational fluid dynamics-discrete element method simulations

Citation Formats

References (25)

Similar Records

Related Subjects