DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization
Abstract
In this work, we present DFT-FE 1.0, building on DFT-FE 0.6 [Comput. Phys. Commun. 246, 106853 (2020)], to conduct fast and accurate large-scale density functional theory (DFT) calculations (reaching ~ 100,000 electrons) on both many-core CPU and hybrid CPU-GPU computing architectures. This work involves improvements in the real-space formulation—via an improved treatment of the electrostatic interactions that substantially enhances the computational efficiency—as well high-performance computing aspects, including the GPU acceleration of all the key compute kernels in DFT-FE. We demonstrate the accuracy by comparing the ground-state energies, ionic forces and cell stresses on a wide-range of benchmark systems against those obtained from widely used DFT codes. Further, we demonstrate the numerical efficiency of our implementation, which yields ~ 20× CPU-GPU speed-up by using GPU acceleration on hybrid CPU-GPU nodes. Notably, owing to the parallel-scaling of the GPU implementation, we obtain wall-times of 80–140 seconds for full ground-state calculations, with stringent accuracy, on benchmark systems containing ~ 6, 000 – 15,000 electrons.
- Authors:
-
[3];
[1]
- Univ. of Michigan, Ann Arbor, MI (United States)
- Indian Inst. of Science, Bangalore (India)
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Publication Date:
- Research Org.:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE Office of Science (SC), Basic Energy Sciences (BES); National Science Foundation (NSF); US Air Force Office of Scientific Research (AFOSR); US Army Research Office (ARO)
- OSTI Identifier:
- 1888932
- Alternate Identifier(s):
- OSTI ID: 1884863
- Grant/Contract Number:
- AC05-00OR22725; SC0008637; AC02-05CH11231; FA9550-21-1-0302; ACI-1053575; W911NF1810242
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Computer Physics Communications
- Additional Journal Information:
- Journal Volume: 280; Journal Issue: 1; Journal ID: ISSN 0010-4655
- Publisher:
- Elsevier
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS; electronic structure; real-space; spectral finite-elements; mixed-precision arithmetic; pseudopotential; all-electron; GPU
Citation Formats
Das, Sambit, Motamarri, Phani, Subramanian, Vishal, Rogers, David M., and Gavini, Vikram. DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization. United States: N. p., 2022.
Web. doi:10.1016/j.cpc.2022.108473.
Das, Sambit, Motamarri, Phani, Subramanian, Vishal, Rogers, David M., & Gavini, Vikram. DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization. United States. https://doi.org/10.1016/j.cpc.2022.108473
Das, Sambit, Motamarri, Phani, Subramanian, Vishal, Rogers, David M., and Gavini, Vikram. Wed .
"DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization". United States. https://doi.org/10.1016/j.cpc.2022.108473. https://www.osti.gov/servlets/purl/1888932.
@article{osti_1888932,
title = {DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization},
author = {Das, Sambit and Motamarri, Phani and Subramanian, Vishal and Rogers, David M. and Gavini, Vikram},
abstractNote = {In this work, we present DFT-FE 1.0, building on DFT-FE 0.6 [Comput. Phys. Commun. 246, 106853 (2020)], to conduct fast and accurate large-scale density functional theory (DFT) calculations (reaching ~ 100,000 electrons) on both many-core CPU and hybrid CPU-GPU computing architectures. This work involves improvements in the real-space formulation—via an improved treatment of the electrostatic interactions that substantially enhances the computational efficiency—as well high-performance computing aspects, including the GPU acceleration of all the key compute kernels in DFT-FE. We demonstrate the accuracy by comparing the ground-state energies, ionic forces and cell stresses on a wide-range of benchmark systems against those obtained from widely used DFT codes. Further, we demonstrate the numerical efficiency of our implementation, which yields ~ 20× CPU-GPU speed-up by using GPU acceleration on hybrid CPU-GPU nodes. Notably, owing to the parallel-scaling of the GPU implementation, we obtain wall-times of 80–140 seconds for full ground-state calculations, with stringent accuracy, on benchmark systems containing ~ 6, 000 – 15,000 electrons.},
doi = {10.1016/j.cpc.2022.108473},
journal = {Computer Physics Communications},
number = 1,
volume = 280,
place = {United States},
year = {Wed Aug 03 00:00:00 EDT 2022},
month = {Wed Aug 03 00:00:00 EDT 2022}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Tensor-structured algorithm for reduced-order scaling large-scale Kohn–Sham density functional theory calculations
journal, April 2021
- Lin, Chih-Chuen; Motamarri, Phani; Gavini, Vikram
- npj Computational Materials, Vol. 7, Issue 1
Generalized Gradient Approximation Made Simple
journal, October 1996
- Perdew, John P.; Burke, Kieron; Ernzerhof, Matthias
- Physical Review Letters, Vol. 77, Issue 18, p. 3865-3868
RESCU: A real space electronic structure method
journal, February 2016
- Michaud-Rioux, Vincent; Zhang, Lei; Guo, Hong
- Journal of Computational Physics, Vol. 307
Non-periodic finite-element formulation of orbital-free density functional theory
journal, April 2007
- Gavini, Vikram; Knap, Jaroslaw; Bhattacharya, Kaushik
- Journal of the Mechanics and Physics of Solids, Vol. 55, Issue 4
Beyond Magic Numbers: Atomic Scale Equilibrium Nanoparticle Shapes for Any Size
journal, August 2017
- Rahm, J. Magnus; Erhart, Paul
- Nano Letters, Vol. 17, Issue 9
On the adaptive finite element analysis of the Kohn-Sham equations: methods, algorithms, and implementation: On the adaptive finite element analysis of the Kohn-Sham equations: methods, algorithms, and implementation
journal, November 2015
- Davydov, Denis; Young, Toby D.; Steinmann, Paul
- International Journal for Numerical Methods in Engineering, Vol. 106, Issue 11
Projector augmented-wave method
journal, December 1994
- Blöchl, P. E.
- Physical Review B, Vol. 50, Issue 24, p. 17953-17979
Ab initio theory of the nitrogen-vacancy center in diamond
journal, September 2019
- Gali, Ádám
- Nanophotonics, Vol. 8, Issue 11
Electronic structure study of screw dislocation core energetics in Aluminum and core energetics informed forces in a dislocation aggregate
journal, July 2017
- Das, Sambit; Gavini, Vikram
- Journal of the Mechanics and Physics of Solids, Vol. 104
Real-Space Density Functional Theory on Graphical Processing Units: Computational Approach and Comparison to Gaussian Basis Set Methods
journal, September 2013
- Andrade, Xavier; Aspuru-Guzik, Alán
- Journal of Chemical Theory and Computation, Vol. 9, Issue 10
A Comparison of Exact and Model Exchange–Correlation Potentials for Molecules
journal, December 2021
- Kanungo, Bikash; Zimmerman, Paul M.; Gavini, Vikram
- The Journal of Physical Chemistry Letters, Vol. 12, Issue 50
Spin–spin interactions in defects in solids from mixed all-electron and pseudopotential first-principles calculations
journal, July 2021
- Ghosh, Krishnendu; Ma, He; Onizhuk, Mykyta
- npj Computational Materials, Vol. 7, Issue 1
QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials
journal, September 2009
- Giannozzi, Paolo; Baroni, Stefano; Bonini, Nicola
- Journal of Physics: Condensed Matter, Vol. 21, Issue 39, Article No. 395502
Optimizations of the eigensolvers in the ELPA library
journal, July 2019
- Kůs, P.; Marek, A.; Köcher, S. S.
- Parallel Computing, Vol. 85
Adaptive Finite Element Approximations for Kohn--Sham Models
journal, January 2014
- Chen, Huajie; Dai, Xiaoying; Gong, Xingao
- Multiscale Modeling & Simulation, Vol. 12, Issue 4
SPARC: Accurate and efficient finite-difference formulation and parallel implementation of Density Functional Theory: Isolated clusters
journal, March 2017
- Ghosh, Swarnava; Suryanarayana, Phanish
- Computer Physics Communications, Vol. 212
Applications of large-scale density functional theory in biology
journal, August 2016
- Cole, Daniel J.; Hine, Nicholas D. M.
- Journal of Physics: Condensed Matter, Vol. 28, Issue 39
exciting: a full-potential all-electron package implementing density-functional theory and many-body perturbation theory
journal, August 2014
- Gulans, Andris; Kontur, Stefan; Meisenbichler, Christian
- Journal of Physics: Condensed Matter, Vol. 26, Issue 36
Electronic-structure study of an edge dislocation in Aluminum and the role of macroscopic deformations on its energetics
journal, March 2015
- Iyer, Mrinal; Radhakrishnan, Balachandran; Gavini, Vikram
- Journal of the Mechanics and Physics of Solids, Vol. 76
Application of order-N first-principles DFT calculations with temperature controlled molecular dynamics to biomolecular system
journal, December 2018
- Otsuka, Takao; Taiji, Makoto
- Journal of Physics: Conference Series, Vol. 1136
Electronic structure calculations with GPAW: a real-space implementation of the projector augmented-wave method
journal, June 2010
- Enkovaara, J.; Rostgaard, C.; Mortensen, J. J.
- Journal of Physics: Condensed Matter, Vol. 22, Issue 25
The Chemistry of Deformation: How Solutes Soften Pure Metals
journal, December 2005
- Trinkle, D. R.
- Science, Vol. 310, Issue 5754
\mathcal{O}(N) methods in electronic structure calculations
journal, February 2012
- Bowler, D. R.; Miyazaki, T.
- Reports on Progress in Physics, Vol. 75, Issue 3
Efficacious Form for Model Pseudopotentials
journal, May 1982
- Kleinman, Leonard; Bylander, D. M.
- Physical Review Letters, Vol. 48, Issue 20
Fast and robust all-electron density functional theory calculations in solids using orthogonalized enriched finite elements
journal, August 2021
- Rufus, Nelson D.; Kanungo, Bikash; Gavini, Vikram
- Physical Review B, Vol. 104, Issue 8
Configurational forces in electronic structure calculations using Kohn-Sham density functional theory
journal, April 2018
- Motamarri, Phani; Gavini, Vikram
- Physical Review B, Vol. 97, Issue 16
Introducing ONETEP : Linear-scaling density functional simulations on parallel computers
journal, February 2005
- Skylaris, Chris-Kriton; Haynes, Peter D.; Mostofi, Arash A.
- The Journal of Chemical Physics, Vol. 122, Issue 8
DFT-FE – A massively parallel adaptive finite-element code for large-scale density functional theory calculations
journal, January 2020
- Motamarri, Phani; Das, Sambit; Rudraraju, Shiva
- Computer Physics Communications, Vol. 246
NWChem: A comprehensive and scalable open-source solution for large scale molecular simulations
journal, September 2010
- Valiev, M.; Bylaska, E. J.; Govind, N.
- Computer Physics Communications, Vol. 181, Issue 9, p. 1477-1489
Linear-scaling first-principles molecular dynamics with plane-waves accuracy
journal, March 2006
- Fattebert, J. -L.; Gygi, F.
- Physical Review B, Vol. 73, Issue 11
Orbital-free density functional theory simulations of dislocations in magnesium
journal, November 2011
- Shin, Ilgyou; Carter, Emily A.
- Modelling and Simulation in Materials Science and Engineering, Vol. 20, Issue 1
DFT modelling of explicit solid–solid interfaces in batteries: methods and challenges
journal, January 2020
- Leung, Kevin
- Physical Chemistry Chemical Physics, Vol. 22, Issue 19
Adaptive Finite Element Method for Solving the Exact Kohn−Sham Equation of Density Functional Theory
journal, March 2009
- Bylaska, Eric J.; Holst, Michael; Weare, John H.
- Journal of Chemical Theory and Computation, Vol. 5, Issue 4
A generic interface for parallel cell-based finite element operator application
journal, June 2012
- Kronbichler, Martin; Kormann, Katharina
- Computers & Fluids, Vol. 63
Efficient iteration scheme for self-consistent pseudopotential calculations
journal, March 1981
- Kerker, G. P.
- Physical Review B, Vol. 23, Issue 6
Large-scale DFT calculations in implicit solvent-A case study on the T4 lysozyme L99A/M102Q protein
journal, March 2012
- Dziedzic, Jacek; Fox, Stephen J.; Fox, Thomas
- International Journal of Quantum Chemistry, Vol. 113, Issue 6
All-electron Kohn–Sham density functional theory on hierarchic finite element spaces
journal, October 2013
- Schauer, Volker; Linder, Christian
- Journal of Computational Physics, Vol. 250
Ab initio modeling of dislocation core properties in metals and semiconductors
journal, February 2017
- Rodney, D.; Ventelon, L.; Clouet, E.
- Acta Materialia, Vol. 124
Beyond the local-density approximation in calculations of ground-state electronic properties
journal, August 1983
- Langreth, David C.; Mehl, M. J.
- Physical Review B, Vol. 28, Issue 4
Discrete discontinuous basis projection method for large-scale electronic structure calculations
journal, September 2018
- Xu, Qimen; Suryanarayana, Phanish; Pask, John E.
- The Journal of Chemical Physics, Vol. 149, Issue 9
Backbone charge transport in double-stranded DNA
journal, August 2020
- Zhuravel, Roman; Huang, Haichao; Polycarpou, Georgia
- Nature Nanotechnology, Vol. 15, Issue 10
Self-Consistent Equations Including Exchange and Correlation Effects
journal, November 1965
- Kohn, W.; Sham, L. J.
- Physical Review, Vol. 140, Issue 4A, p. A1133-A1138
Efficient pseudopotentials for plane-wave calculations
journal, January 1991
- Troullier, N.; Martins, José Luriaas
- Physical Review B, Vol. 43, Issue 3
The PseudoDojo: Training and grading a 85 element optimized norm-conserving pseudopotential table
journal, May 2018
- van Setten, M. J.; Giantomassi, M.; Bousquet, E.
- Computer Physics Communications, Vol. 226
Non-periodic finite-element formulation of Kohn–Sham density functional theory
journal, February 2010
- Suryanarayana, Phanish; Gavini, Vikram; Blesgen, Thomas
- Journal of the Mechanics and Physics of Solids, Vol. 58, Issue 2
Review on modeling of the anode solid electrolyte interphase (SEI) for lithium-ion batteries
journal, March 2018
- Wang, Aiping; Kadam, Sanket; Li, Hong
- npj Computational Materials, Vol. 4, Issue 1
Dislocation Core Energies and Core Fields from First Principles
journal, February 2009
- Clouet, Emmanuel; Ventelon, Lisa; Willaime, F.
- Physical Review Letters, Vol. 102, Issue 5
Self‐Consistent Molecular‐Orbital Methods. I. Use of Gaussian Expansions of Slater‐Type Atomic Orbitals
journal, September 1969
- Hehre, W. J.; Stewart, R. F.; Pople, J. A.
- The Journal of Chemical Physics, Vol. 51, Issue 6
DGDFT: A massively parallel method for large scale density functional theory calculations
journal, September 2015
- Hu, Wei; Lin, Lin; Yang, Chao
- The Journal of Chemical Physics, Vol. 143, Issue 12
Ab Initio Study of Screw Dislocations in Mo and Ta: A New Picture of Plasticity in bcc Transition Metals
journal, February 2000
- Ismail-Beigi, Sohrab; Arias, T. A.
- Physical Review Letters, Vol. 84, Issue 7
Orbital-free density functional theory study of the energetics of vacancy clustering and prismatic dislocation loop nucleation in aluminium
journal, July 2016
- Radhakrishnan, Balachandran; Gavini, Vikram
- Philosophical Magazine, Vol. 96, Issue 23
Electronic-structure calculations based on the finite-element method
journal, August 1995
- Tsuchida, Eiji; Tsukada, Masaru
- Physical Review B, Vol. 52, Issue 8
Speeding up plane-wave electronic-structure calculations using graphics-processing units
journal, July 2011
- Maintz, Stefan; Eck, Bernhard; Dronskowski, Richard
- Computer Physics Communications, Vol. 182, Issue 7
First-principles computation of material properties: the ABINIT software project
journal, November 2002
- Gonze, X.; Beuken, J. -M.; Caracas, R.
- Computational Materials Science, Vol. 25, Issue 3
Real time time-dependent density functional theory using higher order finite-element methods
journal, September 2019
- Kanungo, Bikash; Gavini, Vikram
- Physical Review B, Vol. 100, Issue 11
Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set
journal, October 1996
- Kresse, G.; Furthmüller, J.
- Physical Review B, Vol. 54, Issue 16, p. 11169-11186
Real-space local polynomial basis for solid-state electronic-structure calculations: A finite-element approach
journal, May 1999
- Pask, J. E.; Klein, B. M.; Fong, C. Y.
- Physical Review B, Vol. 59, Issue 19
Linear scaling electronic structure methods
journal, July 1999
- Goedecker, Stefan
- Reviews of Modern Physics, Vol. 71, Issue 4
Exact exchange-correlation potentials from ground-state electron densities
journal, October 2019
- Kanungo, Bikash; Zimmerman, Paul M.; Gavini, Vikram
- Nature Communications, Vol. 10, Issue 1
Adaptive finite-element method for electronic-structure calculations
journal, September 1996
- Tsuchida, Eiji; Tsukada, Masaru
- Physical Review B, Vol. 54, Issue 11
The deal.II library, Version 9.0
journal, December 2018
- Alzetta, Giovanni; Arndt, Daniel; Bangerth, Wolfgang
- Journal of Numerical Mathematics, Vol. 26, Issue 4
Two-Level Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of Large-Scale Electronic Structure Calculations
journal, April 2018
- Banerjee, Amartya S.; Lin, Lin; Suryanarayana, Phanish
- Journal of Chemical Theory and Computation, Vol. 14, Issue 6
Real-space grids and the Octopus code as tools for the development of new simulation approaches for electronic systems
journal, January 2015
- Andrade, Xavier; Strubbe, David; De Giovannini, Umberto
- Physical Chemistry Chemical Physics, Vol. 17, Issue 47
Higher-order adaptive finite-element methods for Kohn–Sham density functional theory
journal, November 2013
- Motamarri, P.; Nowak, M. R.; Leiter, K.
- Journal of Computational Physics, Vol. 253
Daubechies wavelets for high performance electronic structure calculations: The BigDFT project
journal, February 2011
- Genovese, Luigi; Videau, Brice; Ospici, Matthieu
- Comptes Rendus Mécanique, Vol. 339, Issue 2-3
PARSEC – the pseudopotential algorithm for real-space electronic structure calculations: recent advances and novel applications to nano-structures
journal, April 2006
- Kronik, Leeor; Makmal, Adi; Tiago, Murilo L.
- physica status solidi (b), Vol. 243, Issue 5
Large-scale all-electron density functional theory calculations using an enriched finite-element basis
journal, January 2017
- Kanungo, Bikash; Gavini, Vikram
- Physical Review B, Vol. 95, Issue 3
Ab initio molecular simulations with numeric atom-centered orbitals
journal, November 2009
- Blum, Volker; Gehrke, Ralf; Hanke, Felix
- Computer Physics Communications, Vol. 180, Issue 11
Spectrum-splitting approach for Fermi-operator expansion in all-electron Kohn-Sham DFT calculations
journal, January 2017
- Motamarri, Phani; Gavini, Vikram; Bhattacharya, Kaushik
- Physical Review B, Vol. 95, Issue 3
Finite element methods in ab initio electronic structure calculations
journal, April 2005
- Pask, J. E.; Sterne, P. A.
- Modelling and Simulation in Materials Science and Engineering, Vol. 13, Issue 3
Pushing the boundaries of lithium battery research with atomistic modelling on different scales
journal, December 2021
- Morgan, Lucy M.; Mercer, Michael P.; Bhandari, Arihant
- Progress in Energy, Vol. 4, Issue 1
Density functional theory calculation on many-cores hybrid central processing unit-graphic processing unit architectures
journal, July 2009
- Genovese, Luigi; Ospici, Matthieu; Deutsch, Thierry
- The Journal of Chemical Physics, Vol. 131, Issue 3
Algorithms and data structures for massively parallel generic adaptive finite element codes
journal, December 2011
- Bangerth, Wolfgang; Burstedde, Carsten; Heister, Timo
- ACM Transactions on Mathematical Software, Vol. 38, Issue 2
First principles methods using CASTEP
journal, January 2005
- Clark, Stewart J.; Segall, Matthew D.; Pickard, Chris J.
- Zeitschrift für Kristallographie - Crystalline Materials, Vol. 220, Issue 5/6
Optimization algorithm for the generation of ONCV pseudopotentials
journal, November 2015
- Schlipf, Martin; Gygi, François
- Computer Physics Communications, Vol. 196
The ELPA library: scalable parallel eigenvalue solutions for electronic structure theory and computational science
journal, May 2014
- Marek, A.; Blum, V.; Johanni, R.
- Journal of Physics: Condensed Matter, Vol. 26, Issue 21
Polarization consistent basis sets. II. Estimating the Kohn–Sham basis set limit
journal, May 2002
- Jensen, Frank
- The Journal of Chemical Physics, Vol. 116, Issue 17
Advanced capabilities for materials modelling with Quantum ESPRESSO
journal, October 2017
- Giannozzi, P.; Andreussi, O.; Brumme, T.
- Journal of Physics: Condensed Matter, Vol. 29, Issue 46
Subquadratic-scaling subspace projection method for large-scale Kohn-Sham density functional theory calculations using spectral finite-element discretization
journal, September 2014
- Motamarri, Phani; Gavini, Vikram
- Physical Review B, Vol. 90, Issue 11
cp2k: atomistic simulations of condensed matter systems
journal, June 2013
- Hutter, Jürg; Iannuzzi, Marcella; Schiffmann, Florian
- Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 4, Issue 1
Iterative Procedures for Nonlinear Integral Equations
journal, October 1965
- Anderson, Donald G.
- Journal of the ACM, Vol. 12, Issue 4