OpenMP and MPI implementations of an elasto-viscoplastic fast Fourier transform-based micromechanical solver for fast crystal plasticity modeling

Eghtesad, Adnan; Barrett, Timothy J.; Germaschewski, Kai; Lebensohn, Ricardo A.; McCabe, Rodney J.; Knezevic, Marko

doi:10.1016/j.advengsoft.2018.09.010

Title: OpenMP and MPI implementations of an elasto-viscoplastic fast Fourier transform-based micromechanical solver for fast crystal plasticity modeling

Journal Article · Sun Oct 14 00:00:00 EDT 2018 · Advances in Engineering Software

DOI:https://doi.org/10.1016/j.advengsoft.2018.09.010· OSTI ID:1734712

Eghtesad, Adnan ^[1]; Barrett, Timothy J. ^[1];

^[1];

^[2];

^[2]; Knezevic, Marko ^[1]

Univ. of New Hampshire, Durham, NH (United States)
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Here, we explore several parallel implementations of an elasto-viscoplastic fast Fourier transform (EVPFFT) model using Message Passing Interface (MPI), OpenMP, and a hybrid of MPI and OpenMP to efficiently predict micromechanical response of polycrystals. Performance studies using EVPFFT are performed based on domain decomposition over voxels of a periodic cell, which is a representative volume element (RVE) of polycrystalline copper. We begin by parallelizing the computationally intensive Newton–Raphson (NR) single crystal solver within EVPFFT. Next, we compare the performance of the serial and parallel FFTW (Fastest Fourier Transform in the West) using OpenMP (OpenMP-FFTW) and MPI (MPI-FFTW) with the original Numerical Recipes-based FOURN routine within EVPFFT. In the parallel environment, we find that the FFT calculations are best performed using the MPI version of FFTW. Finally, the remainder of the code, except read/write subroutines, is parallelized. Significant speedups of the original EVPFFT model are achieved using MPI on shared memory multicore workstations. Furthermore, results achieved on a distributed memory Cray supercomputer show promising strong and weak scalability and in some cases even super scalability for the single crystal NR solver in EVPFFT. MPI-FFTW also scales perfectly for microstructure RVEs larger than 64³ FFT voxels. For example, the MPI-EVPFFT parallel version of the code accelerates the simulations for approximately two orders of magnitude using 64 cores over the old serial code for an RVE size of 128³. Finally, the parallel EVPFFT code developed in this work can run massive voxel-based microstructural RVEs taking the advantages of thousands of logical cores provided by more advanced clusters.

View Accepted Manuscript (DOE)

View Accepted Manuscript (Publisher)

Cite

Export

Save

Research Organization:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Organization:: USDOE; National Science Foundation (NSF)

Grant/Contract Number:: 89233218CNA000001; 388715; CMMI-1650641; AC52-06NA25396

OSTI ID:: 1734712

Alternate ID(s):: OSTI ID: 1480006; OSTI ID: 1636913

Report Number(s):: LA-UR-18-29864; LA-UR-18-27367

Journal Information:: Advances in Engineering Software, Vol. 126; ISSN 0965-9978

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 35 works

Citation information provided by
Web of Science

References (62)

An analysis of nonuniform and localized deformation in ductile single crystals Peirce, D.; Asaro, R. J.; Needleman, A. Acta Metallurgica, Vol. 30, Issue 6 https://doi.org/10.1016/0001-6160(82)90005-0	journal	June 1982
Distribution of normal stress at grain boundaries in multicrystals: application to an intergranular damage modeling Diard, O.; Leclercq, S.; Rousselier, G. Computational Materials Science, Vol. 25, Issue 1-2 https://doi.org/10.1016/S0927-0256(02)00251-3	journal	September 2002
Enhancement of orientation gradients during simple shear deformation by application of simple compression Jahedi, Mohammad; Ardeljan, Milan; Beyerlein, Irene J. Journal of Applied Physics, Vol. 117, Issue 21 https://doi.org/10.1063/1.4922032	journal	June 2015
Explicit incorporation of deformation twins into crystal plasticity finite element models Ardeljan, Milan; McCabe, Rodney J.; Beyerlein, Irene J. Computer Methods in Applied Mechanics and Engineering, Vol. 295 https://doi.org/10.1016/j.cma.2015.07.003	journal	October 2015
Investigation of three-dimensional aspects of grain-scale plastic surface deformation of an aluminum oligocrystal Zhao, Z.; Ramesh, M.; Raabe, D. International Journal of Plasticity, Vol. 24, Issue 12 https://doi.org/10.1016/j.ijplas.2008.01.002	journal	December 2008
A new implementation of the spectral crystal plasticity framework in implicit finite elements Zecevic, Miroslav; McCabe, Rodney J.; Knezevic, Marko Mechanics of Materials, Vol. 84 https://doi.org/10.1016/j.mechmat.2015.01.018	journal	May 2015
Effect of dislocation density-twin interactions on twin growth in AZ31 as revealed by explicit crystal plasticity finite element modeling Ardeljan, Milan; Beyerlein, Irene J.; Knezevic, Marko International Journal of Plasticity, Vol. 99 https://doi.org/10.1016/j.ijplas.2017.09.002	journal	December 2017
A study of microstructure-driven strain localizations in two-phase polycrystalline HCP/BCC composites using a multi-scale model Ardeljan, Milan; Knezevic, Marko; Nizolek, Thomas International Journal of Plasticity, Vol. 74 https://doi.org/10.1016/j.ijplas.2015.06.003	journal	November 2015
The plasticity of highly oriented nano-layered Zr/Nb composites Ardeljan, Milan; Savage, Daniel J.; Kumar, Anil Acta Materialia, Vol. 115 https://doi.org/10.1016/j.actamat.2016.05.058	journal	August 2016
A multilevel finite element method (FE2) to describe the response of highly non-linear structures using generalized continua Feyel, Frédéric Computer Methods in Applied Mechanics and Engineering, Vol. 192, Issue 28-30 https://doi.org/10.1016/S0045-7825(03)00348-7	journal	July 2003
Parallelization of a finite element Fortran code using OpenMP library Oh, Seong Eun; Hong, Jung-Wuk Advances in Engineering Software, Vol. 104 https://doi.org/10.1016/j.advengsoft.2016.11.004	journal	February 2017
Parallel programming of a peridynamics code coupled with finite element method Lee, Jooeun; Oh, Seong Eun; Hong, Jung-Wuk International Journal of Fracture, Vol. 203, Issue 1-2 https://doi.org/10.1007/s10704-016-0121-y	journal	June 2016
Development of a parallel FE simulator for modeling the whole trans-scale failure process of rock from meso- to engineering-scale Li, Gen; Tang, Chun-An; Liang, Zheng-Zhao Computers & Geosciences, Vol. 98 https://doi.org/10.1016/j.cageo.2016.08.014	journal	January 2017
Higher-Order Finite Element Electromagnetics Code for HPC environments Garcia-Donoro, Daniel; Amor-Martin, Adrian; Garcia-Castillo, Luis E. Procedia Computer Science, Vol. 108 https://doi.org/10.1016/j.procs.2017.05.239	journal	January 2017
Developing a scalable hybrid MPI/OpenMP unstructured finite element model Guo, Xiaohu; Lange, Michael; Gorman, Gerard Computers & Fluids, Vol. 110 https://doi.org/10.1016/j.compfluid.2014.09.007	journal	March 2015
A new approach to fluid–structure interaction within graphics hardware accelerated smooth particle hydrodynamics considering heterogeneous particle size distribution Eghtesad, Adnan; Knezevic, Marko Computational Particle Mechanics, Vol. 5, Issue 3 https://doi.org/10.1007/s40571-017-0176-1	journal	December 2017
A numerical method for computing the overall response of nonlinear composites with complex microstructure Moulinec, H.; Suquet, P. Computer Methods in Applied Mechanics and Engineering, Vol. 157, Issue 1-2 https://doi.org/10.1016/S0045-7825(97)00218-1	journal	April 1998
N-site modeling of a 3D viscoplastic polycrystal using Fast Fourier Transform A. Lebensohn, R. Acta Materialia, Vol. 49, Issue 14 https://doi.org/10.1016/S1359-6454(01)00172-0	journal	August 2001
A call to arms for task parallelism in multi-scale materials modeling Barton, Nathan R.; Bernier, Joel V.; Knap, Jaroslaw International Journal for Numerical Methods in Engineering, Vol. 86, Issue 6 https://doi.org/10.1002/nme.3071	journal	February 2011
Key computational modeling issues in Integrated Computational Materials Engineering Panchal, Jitesh H.; Kalidindi, Surya R.; McDowell, David L. Computer-Aided Design, Vol. 45, Issue 1 https://doi.org/10.1016/j.cad.2012.06.006	journal	January 2013
Accelerating crystal plasticity simulations using GPU multiprocessors: GPU-accelerated crystal plasticity simulations Mellbin, Y.; Hallberg, H.; Ristinmaa, M. International Journal for Numerical Methods in Engineering, Vol. 100, Issue 2 https://doi.org/10.1002/nme.4724	journal	July 2014
Three orders of magnitude improved efficiency with high-performance spectral crystal plasticity on GPU platforms: HIGH-PERFORMANCE IMPLEMENTATION OF CRYSTAL PLASTICITY ON GPU PLATFORMS Mihaila, Bogdan; Knezevic, Marko; Cardenas, Andres International Journal for Numerical Methods in Engineering, Vol. 97, Issue 11 https://doi.org/10.1002/nme.4592	journal	January 2014
Computer implementations of iterative and non-iterative crystal plasticity solvers on high performance graphics hardware Savage, Daniel J.; Knezevic, Marko Computational Mechanics, Vol. 56, Issue 4 https://doi.org/10.1007/s00466-015-1194-6	journal	August 2015
Crystal plasticity with Jacobian-Free Newton–Krylov Chockalingam, K.; Tonks, M. R.; Hales, J. D. Computational Mechanics, Vol. 51, Issue 5 https://doi.org/10.1007/s00466-012-0741-7	journal	July 2012
Building texture evolution networks for deformation processing of polycrystalline fcc metals using spectral approaches: Applications to process design for targeted performance Shaffer, Joshua B.; Knezevic, Marko; Kalidindi, Surya R. International Journal of Plasticity, Vol. 26, Issue 8 https://doi.org/10.1016/j.ijplas.2010.03.010	journal	August 2010
Spectral calibration of crystal plasticity models Kalidindi, Surya R.; Duvvuru, Hari K.; Knezevic, Marko Acta Materialia, Vol. 54, Issue 7 https://doi.org/10.1016/j.actamat.2005.12.018	journal	April 2006
Procedures for reducing large datasets of crystal orientations using generalized spherical harmonics Knezevic, Marko; Landry, Nicholas W. Mechanics of Materials, Vol. 88 https://doi.org/10.1016/j.mechmat.2015.04.014	journal	September 2015
Crystal plasticity simulations using discrete Fourier transforms Knezevic, Marko; Al-Harbi, Hamad F.; Kalidindi, Surya R. Acta Materialia, Vol. 57, Issue 6 https://doi.org/10.1016/j.actamat.2008.12.017	journal	April 2009
Embedded polycrystal plasticity and adaptive sampling Barton, Nathan R.; Knap, Jaroslaw; Arsenlis, Athanasios International Journal of Plasticity, Vol. 24, Issue 2 https://doi.org/10.1016/j.ijplas.2007.03.004	journal	February 2008
The use of discrete harmonics in direct multi-scale embedding of polycrystal plasticity Barton, Nathan R.; Bernier, Joel V.; Lebensohn, Ricardo A. Computer Methods in Applied Mechanics and Engineering, Vol. 283 https://doi.org/10.1016/j.cma.2014.09.017	journal	January 2015
A self-consistent anisotropic approach for the simulation of plastic deformation and texture development of polycrystals: Application to zirconium alloys Lebensohn, R. A.; Tomé, C. N. Acta Metallurgica et Materialia, Vol. 41, Issue 9 https://doi.org/10.1016/0956-7151(93)90130-K	journal	September 1993
Predicting intragranular misorientation distributions in polycrystalline metals using the viscoplastic self-consistent formulation Zecevic, Miroslav; Pantleon, Wolfgang; Lebensohn, Ricardo A. Acta Materialia, Vol. 140 https://doi.org/10.1016/j.actamat.2017.08.056	journal	November 2017
Strain rate and temperature sensitive multi-level crystal plasticity model for large plastic deformation behavior: Application to AZ31 magnesium alloy Ardeljan, Milan; Beyerlein, Irene J.; McWilliams, Brandon A. International Journal of Plasticity, Vol. 83 https://doi.org/10.1016/j.ijplas.2016.04.005	journal	August 2016
Dual-phase steel sheets under cyclic tension–compression to large strains: Experiments and crystal plasticity modeling Zecevic, Milovan; Korkolis, Yannis P.; Kuwabara, Toshihiko Journal of the Mechanics and Physics of Solids, Vol. 96 https://doi.org/10.1016/j.jmps.2016.07.003	journal	November 2016
A polycrystal plasticity model for predicting mechanical response and texture evolution during strain-path changes: Application to beryllium Knezevic, Marko; Beyerlein, Irene J.; Brown, Donald W. International Journal of Plasticity, Vol. 49 https://doi.org/10.1016/j.ijplas.2013.03.008	journal	October 2013
Origin of texture development in orthorhombic uranium Zecevic, Miroslav; Knezevic, Marko; Beyerlein, Irene J. Materials Science and Engineering: A, Vol. 665 https://doi.org/10.1016/j.msea.2016.04.018	journal	May 2016
Three-dimensional deformation process simulation with explicit use of polycrystal plasticity models Beaudoin, A. J.; Mathur, K. K.; Dawson, P. R. International Journal of Plasticity, Vol. 9, Issue 7 https://doi.org/10.1016/0749-6419(93)90054-T	journal	January 1993
Integration of self-consistent polycrystal plasticity with dislocation density based hardening laws within an implicit finite element framework: Application to low-symmetry metals Knezevic, Marko; McCabe, Rodney J.; Lebensohn, Ricardo A. Journal of the Mechanics and Physics of Solids, Vol. 61, Issue 10 https://doi.org/10.1016/j.jmps.2013.05.005	journal	October 2013
Modeling bending of α-titanium with embedded polycrystal plasticity in implicit finite elements Knezevic, Marko; Lebensohn, Ricardo A.; Cazacu, Oana Materials Science and Engineering: A, Vol. 564 https://doi.org/10.1016/j.msea.2012.11.037	journal	March 2013
Transitioning rate sensitivities across multiple length scales: Microstructure-property relationships in the Taylor cylinder impact test on zirconium Zecevic, Miroslav; Beyerlein, Irene J.; McCabe, Rodney J. International Journal of Plasticity, Vol. 84 https://doi.org/10.1016/j.ijplas.2016.05.005	journal	September 2016
Modeling of Sheet Metal Forming Based on Implicit Embedding of the Elasto-Plastic Self-Consistent Formulation in Shell Elements: Application to Cup Drawing of AA6022-T4 Zecevic, Milovan; Knezevic, Marko JOM, Vol. 69, Issue 5 https://doi.org/10.1007/s11837-017-2255-4	journal	February 2017
Texture formation in orthorhombic alpha-uranium under simple compression and rolling to high strains Zecevic, Miroslav; Knezevic, Marko; Beyerlein, Irene J. Journal of Nuclear Materials, Vol. 473 https://doi.org/10.1016/j.jnucmat.2016.02.021	journal	May 2016
Anisotropic modeling of structural components using embedded crystal plasticity constructive laws within finite elements Knezevic, Marko; Crapps, Justin; Beyerlein, Irene J. International Journal of Mechanical Sciences, Vol. 105 https://doi.org/10.1016/j.ijmecsci.2015.11.021	journal	January 2016
A dislocation density based crystal plasticity finite element model: Application to a two-phase polycrystalline HCP/BCC composites Ardeljan, Milan; Beyerlein, Irene J.; Knezevic, Marko Journal of the Mechanics and Physics of Solids, Vol. 66 https://doi.org/10.1016/j.jmps.2014.01.006	journal	May 2014
Deformation twinning in AZ31: Influence on strain hardening and texture evolution Knezevic, Marko; Levinson, Amanda; Harris, Ryan Acta Materialia, Vol. 58, Issue 19 https://doi.org/10.1016/j.actamat.2010.07.041	journal	November 2010
Comparison of finite element and fast Fourier transform crystal plasticity solvers for texture prediction Liu, B.; Raabe, D.; Roters, F. Modelling and Simulation in Materials Science and Engineering, Vol. 18, Issue 8 https://doi.org/10.1088/0965-0393/18/8/085005	journal	October 2010
Spectral database constitutive representation within a spectral micromechanical solver for computationally efficient polycrystal plasticity modelling Eghtesad, Adnan; Zecevic, Miroslav; Lebensohn, Ricardo A. Computational Mechanics, Vol. 61, Issue 1-2 https://doi.org/10.1007/s00466-017-1413-4	journal	May 2017
Simulation of micromechanical behavior of polycrystals: finite elements versus fast Fourier transforms Prakash, A.; Lebensohn, R. A. Modelling and Simulation in Materials Science and Engineering, Vol. 17, Issue 6 https://doi.org/10.1088/0965-0393/17/6/064010	journal	August 2009
An automated procedure for geometry creation and finite element mesh generation: Application to explicit grain structure models and machining distortion Barrett, Timothy J.; Savage, Daniel J.; Ardeljan, Milan Computational Materials Science, Vol. 141 https://doi.org/10.1016/j.commatsci.2017.09.048	journal	January 2018
An elasto-viscoplastic formulation based on fast Fourier transforms for the prediction of micromechanical fields in polycrystalline materials Lebensohn, Ricardo A.; Kanjarla, Anand K.; Eisenlohr, Philip International Journal of Plasticity, Vol. 32-33 https://doi.org/10.1016/j.ijplas.2011.12.005	journal	May 2012
Graphics processing unit accelerated phase field dislocation dynamics: Application to bi-metallic interfaces Eghtesad, Adnan; Germaschewski, Kai; Beyerlein, Irene J. Advances in Engineering Software, Vol. 115 https://doi.org/10.1016/j.advengsoft.2017.09.010	journal	January 2018
Texture evolution and enhanced grain refinement under high-pressure-double-torsion Jahedi, Mohammad; Paydar, Mohammad Hossein; Zheng, Shijian Materials Science and Engineering: A, Vol. 611 https://doi.org/10.1016/j.msea.2014.05.081	journal	August 2014
High-Pressure Double Torsion as a Severe Plastic Deformation Process: Experimental Procedure and Finite Element Modeling Jahedi, Mohammad; Knezevic, Marko; Paydar, Mohammad Hossein Journal of Materials Engineering and Performance, Vol. 24, Issue 4 https://doi.org/10.1007/s11665-015-1426-0	journal	February 2015
Effects of Pressure and Number of Turns on Microstructural Homogeneity Developed in High-Pressure Double Torsion Jahedi, Mohammad; Beyerlein, Irene J.; Paydar, Mohammad Hossein Metallurgical and Materials Transactions A, Vol. 48, Issue 3 https://doi.org/10.1007/s11661-016-3923-7	journal	January 2017
Three dimensional predictions of grain scale plasticity and grain boundaries using crystal plasticity finite element models Knezevic, Marko; Drach, Borys; Ardeljan, Milan Computer Methods in Applied Mechanics and Engineering, Vol. 277 https://doi.org/10.1016/j.cma.2014.05.003	journal	August 2014
Crystallographic texture evolution in bulk deformation processing of FCC metals Kalidindi, S. R.; Bronkhorst, C. A.; Anand, L. Journal of the Mechanics and Physics of Solids, Vol. 40, Issue 3 https://doi.org/10.1016/0022-5096(92)80003-9	journal	January 1992
Macroscopic yield criteria for plastic anisotropic materials containing spheroidal voids Monchiet, Vincent; Cazacu, Oana; Charkaluk, Eric International Journal of Plasticity, Vol. 24, Issue 7 https://doi.org/10.1016/j.ijplas.2007.08.008	journal	July 2008
Amdahl's Law in the Multicore Era Hill, Mark D.; Marty, Michael R. Computer, Vol. 41, Issue 7 https://doi.org/10.1109/MC.2008.209	journal	July 2008
OpenMP: an industry standard API for shared-memory programming Dagum, L.; Menon, R. IEEE Computational Science and Engineering, Vol. 5, Issue 1 https://doi.org/10.1109/99.660313	journal	January 1998
A tale of two laws Heath, Michael T. The International Journal of High Performance Computing Applications, Vol. 29, Issue 3 https://doi.org/10.1177/1094342015572031	journal	February 2015
Using processor affinity in loop scheduling on shared-memory multiprocessors Markatos, E. P.; LeBlanc, T. J. IEEE Transactions on Parallel and Distributed Systems, Vol. 5, Issue 4 https://doi.org/10.1109/71.273046	journal	April 1994
Using processor-cache affinity information in shared-memory multiprocessor scheduling Squillante, M. S.; Lazowska, E. D. IEEE Transactions on Parallel and Distributed Systems, Vol. 4, Issue 2 https://doi.org/10.1109/71.207589	journal	January 1993

Cited By (2)

On Quasi‐Newton methods in fast Fourier transform‐based micromechanics Wicht, Daniel; Schneider, Matti; Böhlke, Thomas International Journal for Numerical Methods in Engineering, Vol. 121, Issue 8 https://doi.org/10.1002/nme.6283	journal	December 2019
On Quasi‐Newton methods in fast Fourier transform‐based micromechanics Wicht, Daniel; Schneider, Matti; Böhlke, Thomas Wiley https://doi.org/10.5445/ir/1000104874	text	January 2020

Similar Records

PETSc/TAO Users Manual (Rev. 3.19)

Technical Report · Thu Mar 30 00:00:00 EDT 2023 · OSTI ID:1734712

Balay, S.; Abhyankar, S.; Adams, Mark F.; +32 more

A fast Fourier transform-based mesoscale field dislocation mechanics study of grain size effects and reversible plasticity in polycrystals

Journal Article · Sat Nov 30 00:00:00 EST 2019 · Journal of the Mechanics and Physics of Solids · OSTI ID:1734712

Berbenni, Stéphane; Taupin, Vincent; Lebensohn, Ricardo A.

Graphics processing unit accelerated phase field dislocation dynamics: Application to bi-metallic interfaces

Journal Article · Sat Oct 14 00:00:00 EDT 2017 · Advances in Engineering Software · OSTI ID:1734712

Eghtesad, Adnan; Germaschewski, Kai; Beyerlein, Irene J.; +2 more

Related Subjects

97 MATHEMATICS AND COMPUTING
crystal plasticity
constitutive equations
cray supercomputer
parallel computing
MPI
42 ENGINEERING

Title: OpenMP and MPI implementations of an elasto-viscoplastic fast Fourier transform-based micromechanical solver for fast crystal plasticity modeling

Citation Formats

References (62)

Cited By (2)

Similar Records

Related Subjects