ExaCA: A performance portable exascale cellular automata application for alloy solidification modeling

Rolchigo, Matt; Reeve, Samuel Temple; Stump, Benjamin; Knapp, Gerald L.; Coleman, John; Plotkowski, Alex; Belak, James

doi:10.1016/j.commatsci.2022.111692

ExaCA: A performance portable exascale cellular automata application for alloy solidification modeling

Journal Article · Mon Aug 08 00:00:00 EDT 2022 · Computational Materials Science

DOI:https://doi.org/10.1016/j.commatsci.2022.111692· OSTI ID:1881144

^[1]; ^[1]; Stump, Benjamin ^[1]; ^[1]; ^[1]; ^[1]; Belak, James ^[2]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

Modeling the as-solidified grain structures that form during alloy processing is a critical component in understanding process-property relationships, particularly for additive manufacturing (AM) where grain structure is very sensitive to processing conditions. While cellular automata (CA)-based models have proven able to predict aspects of microstructure for several alloys and AM process conditions, long run times and large resource sets required limit the utility and the problem size to which existing CA models can be applied. As part of the ExaAM project, an initiative within the Exascale Computing Project (ECP) to develop, test, and optimize an exascale-capable coupled and self-consistent model of AM parts, we developed ExaCA (https://github.com/LLNL/ExaCA) for the liquid–solid phase transformation in the wake of AM melt pools. The CA-based code is parallelized using MPI and the Kokkos programming model, the latter enabling simulation on both CPUs and GPUs within a single-source implementation. Here, we detail the steps taken to transform a baseline, MPI-based CA code into one that is performant on CPUs and GPUs. Performance testing of ExaCA on Summit (a pre-exascale machine at Oak Ridge National Laboratory) was used to quantify CPU–GPU speedup comparing with equal numbers of nodes. Testing showed comparable CPU performance to the MPI-only CA code and a 5-20x speedup when running AM-based test problems using GPUs. The improved performance of CA through GPU utilization and the performance portable nature of ExaCA will enable accurate part-scale modeling by harnessing the power of current and future generations of high performance computing resources. Future work will include improving the strong scaling of ExaCA on GPUs by reducing load imbalance associated with the locality of the problem, and continuing performance optimization across exascale hardware.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: Lawrence Livermore National Laboratory (LLNL); USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC)

Grant/Contract Number:: AC05-00OR22725; AC52-07NA27344

OSTI ID:: 1881144

Journal Information:: Computational Materials Science, Journal Name: Computational Materials Science Vol. 214; ISSN 0927-0256

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (46)

A parallelized three-dimensional cellular automaton model for grain growth during additive manufacturing Lian, Yanping; Lin, Stephen; Yan, Wentao Computational Mechanics, Vol. 61, Issue 5 https://doi.org/10.1007/s00466-017-1535-8	journal	January 2018
A quantitative dendrite growth model and analysis of stability concepts Beltran-Sanchez, Lazaro; Stefanescu, Doru M. Metallurgical and Materials Transactions A, Vol. 35, Issue 8 https://doi.org/10.1007/s11661-006-0227-3	journal	August 2004
Modeling of Ti-W Solidification Microstructures Under Additive Manufacturing Conditions Rolchigo, Matthew R.; Mendoza, Michael Y.; Samimi, Peyman Metallurgical and Materials Transactions A, Vol. 48, Issue 7 https://doi.org/10.1007/s11661-017-4120-z	journal	May 2017
A three-dimensional cellular automation-finite element model for the prediction of solidification grain structures Gandin, Ch. -A.; Desbiolles, J. -L.; Rappaz, M. Metallurgical and Materials Transactions A, Vol. 30, Issue 12 https://doi.org/10.1007/s11661-999-0226-2	journal	December 1999
A Novel Approach to Predict the Process-Induced Mechanical Behavior of Additively Manufactured Materials Kergaßner, Andreas; Koepf, Johannes A.; Markl, Matthias Journal of Materials Engineering and Performance, Vol. 30, Issue 7 https://doi.org/10.1007/s11665-021-05725-0	journal	April 2021
Outcomes and Conclusions from the 2018 AM-Bench Measurements, Challenge Problems, Modeling Submissions, and Conference Levine, Lyle; Lane, Brandon; Heigel, Jarred Integrating Materials and Manufacturing Innovation, Vol. 9, Issue 1 https://doi.org/10.1007/s40192-019-00164-1	journal	February 2020
Location-Specific Microstructure Characterization Within IN625 Additive Manufacturing Benchmark Test Artifacts Stoudt, M. R.; Williams, M. E.; Levine, L. E. Integrating Materials and Manufacturing Innovation, Vol. 9, Issue 1 https://doi.org/10.1007/s40192-020-00172-6	journal	March 2020
Theory of microstructural development during rapid solidification Kurz, W.; Giovanola, B.; Trivedi, R. Acta Metallurgica, Vol. 34, Issue 5 https://doi.org/10.1016/0001-6160(86)90056-8	journal	May 1986
Probabilistic modelling of microstructure formation in solidification processes Rappaz, M.; Gandin, Ch. -A. Acta Metallurgica et Materialia, Vol. 41, Issue 2 https://doi.org/10.1016/0956-7151(93)90065-Z	journal	February 1993
Prediction of dendritic growth and microsegregation patterns in a binary alloy using the phase-field method Warren, J. A.; Boettinger, W. J. Acta Metallurgica et Materialia, Vol. 43, Issue 2 https://doi.org/10.1016/0956-7151(94)00285-P	journal	February 1995
A coupled finite element-cellular automaton model for the prediction of dendritic grain structures in solidification processes Gandin, Ch. -A.; Rappaz, M. Acta Metallurgica et Materialia, Vol. 42, Issue 7 https://doi.org/10.1016/0956-7151(94)90302-6	journal	July 1994
A model of solidification microstructures in nickel-based superalloys: predicting primary dendrite spacing selection Wang, W.; Lee, P. D.; McLean, M. Acta Materialia, Vol. 51, Issue 10 https://doi.org/10.1016/S1359-6454(03)00110-1	journal	June 2003
A 3D Cellular Automaton algorithm for the prediction of dendritic grain growth Gandin, Ch. -A; Rappaz, M. Acta Materialia, Vol. 45, Issue 5 https://doi.org/10.1016/S1359-6454(96)00303-5	journal	May 1997
A three-dimensional sharp interface model for the quantitative simulation of solutal dendritic growth Pan, Shiyan; Zhu, Mingfang Acta Materialia, Vol. 58, Issue 1 https://doi.org/10.1016/j.actamat.2009.09.012	journal	January 2010
Dendrite growth simulation during solidification in the LENS process Yin, H.; Felicelli, S. D. Acta Materialia, Vol. 58, Issue 4 https://doi.org/10.1016/j.actamat.2009.10.053	journal	February 2010
3D multi-layer grain structure simulation of powder bed fusion additive manufacturing Koepf, Johannes A.; Gotterbarm, Martin R.; Markl, Matthias Acta Materialia, Vol. 152 https://doi.org/10.1016/j.actamat.2018.04.030	journal	June 2018
Optimizing the cellular automata finite element model for additive manufacturing to simulate large microstructures Teferra, Kirubel; Rowenhorst, David J. Acta Materialia, Vol. 213 https://doi.org/10.1016/j.actamat.2021.116930	journal	July 2021
Multi-scale modeling of solidification and microstructure development in laser keyhole welding process for austenitic stainless steel Tan, Wenda; Shin, Yung C. Computational Materials Science, Vol. 98 https://doi.org/10.1016/j.commatsci.2014.10.063	journal	February 2015
A coupled Cellular Automaton–Lattice Boltzmann model for grain structure simulation during additive manufacturing Rai, Abha; Markl, Matthias; Körner, Carolin Computational Materials Science, Vol. 124 https://doi.org/10.1016/j.commatsci.2016.07.005	journal	November 2016
Simulation of metal additive manufacturing microstructures using kinetic Monte Carlo Rodgers, Theron M.; Madison, Jonathan D.; Tikare, Veena Computational Materials Science, Vol. 135 https://doi.org/10.1016/j.commatsci.2017.03.053	journal	July 2017
Three-dimensional modeling of the microstructure evolution during metal additive manufacturing Zinovieva, O.; Zinoviev, A.; Ploshikhin, V. Computational Materials Science, Vol. 141 https://doi.org/10.1016/j.commatsci.2017.09.018	journal	January 2018
Numerical investigation of effects of nucleation mechanisms on grain structure in metal additive manufacturing Li, Xuxiao; Tan, Wenda Computational Materials Science, Vol. 153 https://doi.org/10.1016/j.commatsci.2018.06.019	journal	October 2018
Integrated 2D cellular automata-phase field modeling of solidification and microstructure evolution during additive manufacturing of Ti6Al4V Liu, Shunyu; Shin, Yung C. Computational Materials Science, Vol. 183 https://doi.org/10.1016/j.commatsci.2020.109889	journal	October 2020
Sensitivity of cellular automata grain structure predictions for high solidification rates Rolchigo, Matthew; Plotkowski, Alex; Belak, James Computational Materials Science, Vol. 196 https://doi.org/10.1016/j.commatsci.2021.110498	journal	August 2021
A physics-informed machine learning method for predicting grain structure characteristics in directed energy deposition Kats, Dmitriy; Wang, Zhidong; Gan, Zhengtao Computational Materials Science, Vol. 202 https://doi.org/10.1016/j.commatsci.2021.110958	journal	February 2022
Large-scale parallel lattice Boltzmann–cellular automaton model of two-dimensional dendritic growth Jelinek, Bohumir; Eshraghi, Mohsen; Felicelli, Sergio Computer Physics Communications, Vol. 185, Issue 3 https://doi.org/10.1016/j.cpc.2013.09.013	journal	March 2014
Numerical simulation of dendritic growth in directional solidification of binary alloys using a lattice Boltzmann scheme Sun, Dongke; Pan, Shiyan; Han, Qinyou International Journal of Heat and Mass Transfer, Vol. 103 https://doi.org/10.1016/j.ijheatmasstransfer.2016.07.055	journal	December 2016
Investigation on evolution mechanisms of site-specific grain structures during metal additive manufacturing Liu, P. W.; Ji, Y. Z.; Wang, Z. Journal of Materials Processing Technology, Vol. 257 https://doi.org/10.1016/j.jmatprotec.2018.02.042	journal	July 2018
A fast method based on GPU for solidification structure simulation of continuous casting billets Wang, Jing Jing; Meng, Hong Ji; Yang, Jian Journal of Computational Science, Vol. 48 https://doi.org/10.1016/j.jocs.2020.101265	journal	January 2021
Kokkos: Enabling manycore performance portability through polymorphic memory access patterns Carter Edwards, H.; Trott, Christian R.; Sunderland, Daniel Journal of Parallel and Distributed Computing, Vol. 74, Issue 12 https://doi.org/10.1016/j.jpdc.2014.07.003	journal	December 2014
Evolution of grain structure during laser additive manufacturing. Simulation by a cellular automata method Zinoviev, A.; Zinovieva, O.; Ploshikhin, V. Materials & Design, Vol. 106 https://doi.org/10.1016/j.matdes.2016.05.125	journal	September 2016
Grain growth prediction in selective electron beam melting of Ti-6Al-4V with a cellular automaton method Xiong, Feiyu; Huang, Chenyang; Kafka, Orion L. Materials & Design, Vol. 199 https://doi.org/10.1016/j.matdes.2020.109410	journal	February 2021
Effects of scanning pattern on the grain structure and elastic properties of additively manufactured 316L austenitic stainless steel Zinovieva, O.; Romanova, V.; Balokhonov, R. Materials Science and Engineering: A, Vol. 832 https://doi.org/10.1016/j.msea.2021.142447	journal	January 2022
Phase-field modeling of grain evolutions in additive manufacturing from nucleation, growth, to coarsening Yang, Min; Wang, Lu; Yan, Wentao npj Computational Materials, Vol. 7, Issue 1 https://doi.org/10.1038/s41524-021-00524-6	journal	April 2021
Sparse thermal data for cellular automata modeling of grain structure in additive manufacturing Rolchigo, Matthew; Stump, Benjamin; Belak, James Modelling and Simulation in Materials Science and Engineering, Vol. 28, Issue 6 https://doi.org/10.1088/1361-651X/ab9734	journal	June 2020
Simulation of solidified β grain for Ti–6Al–4V during wire laser additive manufacturing by three-dimensional cellular automaton method Sun, Weizhao; Shan, Feihu; Zong, Nanfu Modelling and Simulation in Materials Science and Engineering, Vol. 29, Issue 6 https://doi.org/10.1088/1361-651X/ac0c23	journal	July 2021
Phase-field and sharp-interface alloy models Caginalp, G.; Xie, W. Physical Review E, Vol. 48, Issue 3 https://doi.org/10.1103/PhysRevE.48.1897	journal	September 1993
Phase-field model for binary alloys Kim, Seong Gyoon; Kim, Won Tae; Suzuki, Toshio Physical Review E, Vol. 60, Issue 6 https://doi.org/10.1103/PhysRevE.60.7186	journal	December 1999
Kokkos 3: Programming Model Extensions for the Exascale Era Trott, Christian; Lebrun-Grandie, Damien; Arndt, Daniel IEEE Transactions on Parallel and Distributed Systems https://doi.org/10.1109/TPDS.2021.3097283	journal	January 2021
Sensitivity of Thermal Predictions to Uncertain Surface Tension Data in Laser Additive Manufacturing Coleman, J.; Plotkowski, A.; Stump, B. Journal of Heat Transfer, Vol. 142, Issue 12 https://doi.org/10.1115/1.4047916	journal	October 2020
Epitaxy and Microstructure Evolution in Metal Additive Manufacturing Basak, Amrita; Das, Suman Annual Review of Materials Research, Vol. 46, Issue 1 https://doi.org/10.1146/annurev-matsci-070115-031728	journal	July 2016
Enabling particle applications for exascale computing platforms Mniszewski, Susan M.; Belak, James; Fattebert, Jean-Luc The International Journal of High Performance Computing Applications, Vol. 35, Issue 6 https://doi.org/10.1177/10943420211022829	journal	July 2021
ExaAM: Metal additive manufacturing simulation at the fidelity of the microstructure Turner, John A.; Belak, James; Barton, Nathan The International Journal of High Performance Computing Applications, Vol. 36, Issue 1 https://doi.org/10.1177/10943420211042558	journal	January 2022
AMReX: a framework for block-structured adaptive mesh refinement Zhang, Weiqun; Almgren, Ann; Beckner, Vince Journal of Open Source Software, Vol. 4, Issue 37 https://doi.org/10.21105/joss.01370	journal	May 2019
Understanding Uncertainty in Microstructure Evolution and Constitutive Properties in Additive Process Modeling Rolchigo, Matthew; Carson, Robert; Belak, James Metals, Vol. 12, Issue 2 https://doi.org/10.3390/met12020324	journal	February 2022
Texture Analysis with MTEX – Free and Open Source Software Toolbox Bachmann, F.; Hielscher, Ralf; Schaeben, Helmut Solid State Phenomena, Vol. 160 https://doi.org/10.4028/www.scientific.net/SSP.160.63	journal	February 2010

Similar Records

An OpenMP GPU-offload implementation of a non-equilibrium solidification cellular automata model for additive manufacturing

Journal Article · Wed Nov 23 19:00:00 EST 2022 · Computer Physics Communications · OSTI ID:1908088

Toucan: A performance portable, scalable implementation of the DECA algorithm

Journal Article · Wed Feb 05 19:00:00 EST 2025 · Computational Materials Science · OSTI ID:2538254

Sparse thermal data for cellular automata modeling of grain structure in additive manufacturing

Journal Article · Thu Jun 18 20:00:00 EDT 2020 · Modelling and Simulation in Materials Science and Engineering · OSTI ID:1642368

Related Subjects

36 MATERIALS SCIENCE
Additive manufacturing
Alloy solidification
Cellular automata
High performance computing

ExaCA: A performance portable exascale cellular automata application for alloy solidification modeling

Citation Formats

References (46)

Similar Records

Related Subjects