DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization
Abstract
In this work, we present DFT-FE 1.0, building on DFT-FE 0.6 [Comput. Phys. Commun. 246, 106853 (2020)], to conduct fast and accurate large-scale density functional theory (DFT) calculations (reaching ~ 100,000 electrons) on both many-core CPU and hybrid CPU-GPU computing architectures. This work involves improvements in the real-space formulation—via an improved treatment of the electrostatic interactions that substantially enhances the computational efficiency—as well high-performance computing aspects, including the GPU acceleration of all the key compute kernels in DFT-FE. We demonstrate the accuracy by comparing the ground-state energies, ionic forces and cell stresses on a wide-range of benchmark systems against those obtained from widely used DFT codes. Further, we demonstrate the numerical efficiency of our implementation, which yields ~ 20× CPU-GPU speed-up by using GPU acceleration on hybrid CPU-GPU nodes. Notably, owing to the parallel-scaling of the GPU implementation, we obtain wall-times of 80–140 seconds for full ground-state calculations, with stringent accuracy, on benchmark systems containing ~ 6, 000 – 15,000 electrons.
- Authors:
-
- Univ. of Michigan, Ann Arbor, MI (United States)
- Indian Inst. of Science, Bangalore (India)
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Publication Date:
- Research Org.:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE Office of Science (SC), Basic Energy Sciences (BES); National Science Foundation (NSF); US Air Force Office of Scientific Research (AFOSR); US Army Research Office (ARO)
- OSTI Identifier:
- 1888932
- Alternate Identifier(s):
- OSTI ID: 1884863
- Grant/Contract Number:
- AC05-00OR22725; SC0008637; AC02-05CH11231; FA9550-21-1-0302; ACI-1053575; W911NF1810242
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Computer Physics Communications
- Additional Journal Information:
- Journal Volume: 280; Journal Issue: 1; Journal ID: ISSN 0010-4655
- Publisher:
- Elsevier
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS; electronic structure; real-space; spectral finite-elements; mixed-precision arithmetic; pseudopotential; all-electron; GPU
Citation Formats
Das, Sambit, Motamarri, Phani, Subramanian, Vishal, Rogers, David M., and Gavini, Vikram. DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization. United States: N. p., 2022.
Web. doi:10.1016/j.cpc.2022.108473.
Das, Sambit, Motamarri, Phani, Subramanian, Vishal, Rogers, David M., & Gavini, Vikram. DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization. United States. https://doi.org/10.1016/j.cpc.2022.108473
Das, Sambit, Motamarri, Phani, Subramanian, Vishal, Rogers, David M., and Gavini, Vikram. Wed .
"DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization". United States. https://doi.org/10.1016/j.cpc.2022.108473. https://www.osti.gov/servlets/purl/1888932.
@article{osti_1888932,
title = {DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization},
author = {Das, Sambit and Motamarri, Phani and Subramanian, Vishal and Rogers, David M. and Gavini, Vikram},
abstractNote = {In this work, we present DFT-FE 1.0, building on DFT-FE 0.6 [Comput. Phys. Commun. 246, 106853 (2020)], to conduct fast and accurate large-scale density functional theory (DFT) calculations (reaching ~ 100,000 electrons) on both many-core CPU and hybrid CPU-GPU computing architectures. This work involves improvements in the real-space formulation—via an improved treatment of the electrostatic interactions that substantially enhances the computational efficiency—as well high-performance computing aspects, including the GPU acceleration of all the key compute kernels in DFT-FE. We demonstrate the accuracy by comparing the ground-state energies, ionic forces and cell stresses on a wide-range of benchmark systems against those obtained from widely used DFT codes. Further, we demonstrate the numerical efficiency of our implementation, which yields ~ 20× CPU-GPU speed-up by using GPU acceleration on hybrid CPU-GPU nodes. Notably, owing to the parallel-scaling of the GPU implementation, we obtain wall-times of 80–140 seconds for full ground-state calculations, with stringent accuracy, on benchmark systems containing ~ 6, 000 – 15,000 electrons.},
doi = {10.1016/j.cpc.2022.108473},
journal = {Computer Physics Communications},
number = 1,
volume = 280,
place = {United States},
year = {Wed Aug 03 00:00:00 EDT 2022},
month = {Wed Aug 03 00:00:00 EDT 2022}
}
Works referenced in this record:
Tensor-structured algorithm for reduced-order scaling large-scale Kohn–Sham density functional theory calculations
journal, April 2021
- Lin, Chih-Chuen; Motamarri, Phani; Gavini, Vikram
- npj Computational Materials, Vol. 7, Issue 1
Generalized Gradient Approximation Made Simple
journal, October 1996
- Perdew, John P.; Burke, Kieron; Ernzerhof, Matthias
- Physical Review Letters, Vol. 77, Issue 18, p. 3865-3868
RESCU: A real space electronic structure method
journal, February 2016
- Michaud-Rioux, Vincent; Zhang, Lei; Guo, Hong
- Journal of Computational Physics, Vol. 307
Non-periodic finite-element formulation of orbital-free density functional theory
journal, April 2007
- Gavini, Vikram; Knap, Jaroslaw; Bhattacharya, Kaushik
- Journal of the Mechanics and Physics of Solids, Vol. 55, Issue 4
Beyond Magic Numbers: Atomic Scale Equilibrium Nanoparticle Shapes for Any Size
journal, August 2017
- Rahm, J. Magnus; Erhart, Paul
- Nano Letters, Vol. 17, Issue 9
On the adaptive finite element analysis of the Kohn-Sham equations: methods, algorithms, and implementation: On the adaptive finite element analysis of the Kohn-Sham equations: methods, algorithms, and implementation
journal, November 2015
- Davydov, Denis; Young, Toby D.; Steinmann, Paul
- International Journal for Numerical Methods in Engineering, Vol. 106, Issue 11
Projector augmented-wave method
journal, December 1994
- Blöchl, P. E.
- Physical Review B, Vol. 50, Issue 24, p. 17953-17979
Ab initio theory of the nitrogen-vacancy center in diamond
journal, September 2019
- Gali, Ádám
- Nanophotonics, Vol. 8, Issue 11
Electronic structure study of screw dislocation core energetics in Aluminum and core energetics informed forces in a dislocation aggregate
journal, July 2017
- Das, Sambit; Gavini, Vikram
- Journal of the Mechanics and Physics of Solids, Vol. 104
Real-Space Density Functional Theory on Graphical Processing Units: Computational Approach and Comparison to Gaussian Basis Set Methods
journal, September 2013
- Andrade, Xavier; Aspuru-Guzik, Alán
- Journal of Chemical Theory and Computation, Vol. 9, Issue 10
A Comparison of Exact and Model Exchange–Correlation Potentials for Molecules
journal, December 2021
- Kanungo, Bikash; Zimmerman, Paul M.; Gavini, Vikram
- The Journal of Physical Chemistry Letters, Vol. 12, Issue 50
Spin–spin interactions in defects in solids from mixed all-electron and pseudopotential first-principles calculations
journal, July 2021
- Ghosh, Krishnendu; Ma, He; Onizhuk, Mykyta
- npj Computational Materials, Vol. 7, Issue 1
QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials
journal, September 2009
- Giannozzi, Paolo; Baroni, Stefano; Bonini, Nicola
- Journal of Physics: Condensed Matter, Vol. 21, Issue 39, Article No. 395502
Optimizations of the eigensolvers in the ELPA library
journal, July 2019
- Kůs, P.; Marek, A.; Köcher, S. S.
- Parallel Computing, Vol. 85
Adaptive Finite Element Approximations for Kohn--Sham Models
journal, January 2014
- Chen, Huajie; Dai, Xiaoying; Gong, Xingao
- Multiscale Modeling & Simulation, Vol. 12, Issue 4
SPARC: Accurate and efficient finite-difference formulation and parallel implementation of Density Functional Theory: Isolated clusters
journal, March 2017
- Ghosh, Swarnava; Suryanarayana, Phanish
- Computer Physics Communications, Vol. 212
Applications of large-scale density functional theory in biology
journal, August 2016
- Cole, Daniel J.; Hine, Nicholas D. M.
- Journal of Physics: Condensed Matter, Vol. 28, Issue 39
exciting: a full-potential all-electron package implementing density-functional theory and many-body perturbation theory
journal, August 2014
- Gulans, Andris; Kontur, Stefan; Meisenbichler, Christian
- Journal of Physics: Condensed Matter, Vol. 26, Issue 36
Electronic-structure study of an edge dislocation in Aluminum and the role of macroscopic deformations on its energetics
journal, March 2015
- Iyer, Mrinal; Radhakrishnan, Balachandran; Gavini, Vikram
- Journal of the Mechanics and Physics of Solids, Vol. 76
Application of order-N first-principles DFT calculations with temperature controlled molecular dynamics to biomolecular system
journal, December 2018
- Otsuka, Takao; Taiji, Makoto
- Journal of Physics: Conference Series, Vol. 1136
Electronic structure calculations with GPAW: a real-space implementation of the projector augmented-wave method
journal, June 2010
- Enkovaara, J.; Rostgaard, C.; Mortensen, J. J.
- Journal of Physics: Condensed Matter, Vol. 22, Issue 25
The Chemistry of Deformation: How Solutes Soften Pure Metals
journal, December 2005
- Trinkle, D. R.
- Science, Vol. 310, Issue 5754
\mathcal{O}(N) methods in electronic structure calculations
journal, February 2012
- Bowler, D. R.; Miyazaki, T.
- Reports on Progress in Physics, Vol. 75, Issue 3
Efficacious Form for Model Pseudopotentials
journal, May 1982
- Kleinman, Leonard; Bylander, D. M.
- Physical Review Letters, Vol. 48, Issue 20
Fast and robust all-electron density functional theory calculations in solids using orthogonalized enriched finite elements
journal, August 2021
- Rufus, Nelson D.; Kanungo, Bikash; Gavini, Vikram
- Physical Review B, Vol. 104, Issue 8
Configurational forces in electronic structure calculations using Kohn-Sham density functional theory
journal, April 2018
- Motamarri, Phani; Gavini, Vikram
- Physical Review B, Vol. 97, Issue 16
Introducing ONETEP : Linear-scaling density functional simulations on parallel computers
journal, February 2005
- Skylaris, Chris-Kriton; Haynes, Peter D.; Mostofi, Arash A.
- The Journal of Chemical Physics, Vol. 122, Issue 8
DFT-FE – A massively parallel adaptive finite-element code for large-scale density functional theory calculations
journal, January 2020
- Motamarri, Phani; Das, Sambit; Rudraraju, Shiva
- Computer Physics Communications, Vol. 246
NWChem: A comprehensive and scalable open-source solution for large scale molecular simulations
journal, September 2010
- Valiev, M.; Bylaska, E. J.; Govind, N.
- Computer Physics Communications, Vol. 181, Issue 9, p. 1477-1489
Linear-scaling first-principles molecular dynamics with plane-waves accuracy
journal, March 2006
- Fattebert, J. -L.; Gygi, F.
- Physical Review B, Vol. 73, Issue 11
Orbital-free density functional theory simulations of dislocations in magnesium
journal, November 2011
- Shin, Ilgyou; Carter, Emily A.
- Modelling and Simulation in Materials Science and Engineering, Vol. 20, Issue 1
DFT modelling of explicit solid–solid interfaces in batteries: methods and challenges
journal, January 2020
- Leung, Kevin
- Physical Chemistry Chemical Physics, Vol. 22, Issue 19
Adaptive Finite Element Method for Solving the Exact Kohn−Sham Equation of Density Functional Theory
journal, March 2009
- Bylaska, Eric J.; Holst, Michael; Weare, John H.
- Journal of Chemical Theory and Computation, Vol. 5, Issue 4
A generic interface for parallel cell-based finite element operator application
journal, June 2012
- Kronbichler, Martin; Kormann, Katharina
- Computers & Fluids, Vol. 63
Efficient iteration scheme for self-consistent pseudopotential calculations
journal, March 1981
- Kerker, G. P.
- Physical Review B, Vol. 23, Issue 6
Large-scale DFT calculations in implicit solvent-A case study on the T4 lysozyme L99A/M102Q protein
journal, March 2012
- Dziedzic, Jacek; Fox, Stephen J.; Fox, Thomas
- International Journal of Quantum Chemistry, Vol. 113, Issue 6
All-electron Kohn–Sham density functional theory on hierarchic finite element spaces
journal, October 2013
- Schauer, Volker; Linder, Christian
- Journal of Computational Physics, Vol. 250
Ab initio modeling of dislocation core properties in metals and semiconductors
journal, February 2017
- Rodney, D.; Ventelon, L.; Clouet, E.
- Acta Materialia, Vol. 124
Beyond the local-density approximation in calculations of ground-state electronic properties
journal, August 1983
- Langreth, David C.; Mehl, M. J.
- Physical Review B, Vol. 28, Issue 4
Discrete discontinuous basis projection method for large-scale electronic structure calculations
journal, September 2018
- Xu, Qimen; Suryanarayana, Phanish; Pask, John E.
- The Journal of Chemical Physics, Vol. 149, Issue 9
Backbone charge transport in double-stranded DNA
journal, August 2020
- Zhuravel, Roman; Huang, Haichao; Polycarpou, Georgia
- Nature Nanotechnology, Vol. 15, Issue 10
Self-Consistent Equations Including Exchange and Correlation Effects
journal, November 1965
- Kohn, W.; Sham, L. J.
- Physical Review, Vol. 140, Issue 4A, p. A1133-A1138
Efficient pseudopotentials for plane-wave calculations
journal, January 1991
- Troullier, N.; Martins, José Luriaas
- Physical Review B, Vol. 43, Issue 3
The PseudoDojo: Training and grading a 85 element optimized norm-conserving pseudopotential table
journal, May 2018
- van Setten, M. J.; Giantomassi, M.; Bousquet, E.
- Computer Physics Communications, Vol. 226
Non-periodic finite-element formulation of Kohn–Sham density functional theory
journal, February 2010
- Suryanarayana, Phanish; Gavini, Vikram; Blesgen, Thomas
- Journal of the Mechanics and Physics of Solids, Vol. 58, Issue 2
Review on modeling of the anode solid electrolyte interphase (SEI) for lithium-ion batteries
journal, March 2018
- Wang, Aiping; Kadam, Sanket; Li, Hong
- npj Computational Materials, Vol. 4, Issue 1
Dislocation Core Energies and Core Fields from First Principles
journal, February 2009
- Clouet, Emmanuel; Ventelon, Lisa; Willaime, F.
- Physical Review Letters, Vol. 102, Issue 5
Self‐Consistent Molecular‐Orbital Methods. I. Use of Gaussian Expansions of Slater‐Type Atomic Orbitals
journal, September 1969
- Hehre, W. J.; Stewart, R. F.; Pople, J. A.
- The Journal of Chemical Physics, Vol. 51, Issue 6
DGDFT: A massively parallel method for large scale density functional theory calculations
journal, September 2015
- Hu, Wei; Lin, Lin; Yang, Chao
- The Journal of Chemical Physics, Vol. 143, Issue 12
Ab Initio Study of Screw Dislocations in Mo and Ta: A New Picture of Plasticity in bcc Transition Metals
journal, February 2000
- Ismail-Beigi, Sohrab; Arias, T. A.
- Physical Review Letters, Vol. 84, Issue 7
Orbital-free density functional theory study of the energetics of vacancy clustering and prismatic dislocation loop nucleation in aluminium
journal, July 2016
- Radhakrishnan, Balachandran; Gavini, Vikram
- Philosophical Magazine, Vol. 96, Issue 23
Electronic-structure calculations based on the finite-element method
journal, August 1995
- Tsuchida, Eiji; Tsukada, Masaru
- Physical Review B, Vol. 52, Issue 8
Speeding up plane-wave electronic-structure calculations using graphics-processing units
journal, July 2011
- Maintz, Stefan; Eck, Bernhard; Dronskowski, Richard
- Computer Physics Communications, Vol. 182, Issue 7
First-principles computation of material properties: the ABINIT software project
journal, November 2002
- Gonze, X.; Beuken, J. -M.; Caracas, R.
- Computational Materials Science, Vol. 25, Issue 3
Real time time-dependent density functional theory using higher order finite-element methods
journal, September 2019
- Kanungo, Bikash; Gavini, Vikram
- Physical Review B, Vol. 100, Issue 11
Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set
journal, October 1996
- Kresse, G.; Furthmüller, J.
- Physical Review B, Vol. 54, Issue 16, p. 11169-11186
Real-space local polynomial basis for solid-state electronic-structure calculations: A finite-element approach
journal, May 1999
- Pask, J. E.; Klein, B. M.; Fong, C. Y.
- Physical Review B, Vol. 59, Issue 19
Linear scaling electronic structure methods
journal, July 1999
- Goedecker, Stefan
- Reviews of Modern Physics, Vol. 71, Issue 4
Exact exchange-correlation potentials from ground-state electron densities
journal, October 2019
- Kanungo, Bikash; Zimmerman, Paul M.; Gavini, Vikram
- Nature Communications, Vol. 10, Issue 1
Adaptive finite-element method for electronic-structure calculations
journal, September 1996
- Tsuchida, Eiji; Tsukada, Masaru
- Physical Review B, Vol. 54, Issue 11
The deal.II library, Version 9.0
journal, December 2018
- Alzetta, Giovanni; Arndt, Daniel; Bangerth, Wolfgang
- Journal of Numerical Mathematics, Vol. 26, Issue 4
Two-Level Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of Large-Scale Electronic Structure Calculations
journal, April 2018
- Banerjee, Amartya S.; Lin, Lin; Suryanarayana, Phanish
- Journal of Chemical Theory and Computation, Vol. 14, Issue 6
Real-space grids and the Octopus code as tools for the development of new simulation approaches for electronic systems
journal, January 2015
- Andrade, Xavier; Strubbe, David; De Giovannini, Umberto
- Physical Chemistry Chemical Physics, Vol. 17, Issue 47
Higher-order adaptive finite-element methods for Kohn–Sham density functional theory
journal, November 2013
- Motamarri, P.; Nowak, M. R.; Leiter, K.
- Journal of Computational Physics, Vol. 253
Daubechies wavelets for high performance electronic structure calculations: The BigDFT project
journal, February 2011
- Genovese, Luigi; Videau, Brice; Ospici, Matthieu
- Comptes Rendus Mécanique, Vol. 339, Issue 2-3
PARSEC – the pseudopotential algorithm for real-space electronic structure calculations: recent advances and novel applications to nano-structures
journal, April 2006
- Kronik, Leeor; Makmal, Adi; Tiago, Murilo L.
- physica status solidi (b), Vol. 243, Issue 5
Large-scale all-electron density functional theory calculations using an enriched finite-element basis
journal, January 2017
- Kanungo, Bikash; Gavini, Vikram
- Physical Review B, Vol. 95, Issue 3
Ab initio molecular simulations with numeric atom-centered orbitals
journal, November 2009
- Blum, Volker; Gehrke, Ralf; Hanke, Felix
- Computer Physics Communications, Vol. 180, Issue 11
Spectrum-splitting approach for Fermi-operator expansion in all-electron Kohn-Sham DFT calculations
journal, January 2017
- Motamarri, Phani; Gavini, Vikram; Bhattacharya, Kaushik
- Physical Review B, Vol. 95, Issue 3
Finite element methods in ab initio electronic structure calculations
journal, April 2005
- Pask, J. E.; Sterne, P. A.
- Modelling and Simulation in Materials Science and Engineering, Vol. 13, Issue 3
Pushing the boundaries of lithium battery research with atomistic modelling on different scales
journal, December 2021
- Morgan, Lucy M.; Mercer, Michael P.; Bhandari, Arihant
- Progress in Energy, Vol. 4, Issue 1
Density functional theory calculation on many-cores hybrid central processing unit-graphic processing unit architectures
journal, July 2009
- Genovese, Luigi; Ospici, Matthieu; Deutsch, Thierry
- The Journal of Chemical Physics, Vol. 131, Issue 3
Algorithms and data structures for massively parallel generic adaptive finite element codes
journal, December 2011
- Bangerth, Wolfgang; Burstedde, Carsten; Heister, Timo
- ACM Transactions on Mathematical Software, Vol. 38, Issue 2
First principles methods using CASTEP
journal, January 2005
- Clark, Stewart J.; Segall, Matthew D.; Pickard, Chris J.
- Zeitschrift für Kristallographie - Crystalline Materials, Vol. 220, Issue 5/6
Optimization algorithm for the generation of ONCV pseudopotentials
journal, November 2015
- Schlipf, Martin; Gygi, François
- Computer Physics Communications, Vol. 196
The ELPA library: scalable parallel eigenvalue solutions for electronic structure theory and computational science
journal, May 2014
- Marek, A.; Blum, V.; Johanni, R.
- Journal of Physics: Condensed Matter, Vol. 26, Issue 21
Polarization consistent basis sets. II. Estimating the Kohn–Sham basis set limit
journal, May 2002
- Jensen, Frank
- The Journal of Chemical Physics, Vol. 116, Issue 17
Advanced capabilities for materials modelling with Quantum ESPRESSO
journal, October 2017
- Giannozzi, P.; Andreussi, O.; Brumme, T.
- Journal of Physics: Condensed Matter, Vol. 29, Issue 46
Subquadratic-scaling subspace projection method for large-scale Kohn-Sham density functional theory calculations using spectral finite-element discretization
journal, September 2014
- Motamarri, Phani; Gavini, Vikram
- Physical Review B, Vol. 90, Issue 11
cp2k: atomistic simulations of condensed matter systems
journal, June 2013
- Hutter, Jürg; Iannuzzi, Marcella; Schiffmann, Florian
- Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 4, Issue 1
Iterative Procedures for Nonlinear Integral Equations
journal, October 1965
- Anderson, Donald G.
- Journal of the ACM, Vol. 12, Issue 4