DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: DFT-FE 1.0: A massively parallel hybrid CPU-GPU density functional theory code using finite-element discretization

Journal Article · · Computer Physics Communications

In this work, we present DFT-FE 1.0, building on DFT-FE 0.6 [Comput. Phys. Commun. 246, 106853 (2020)], to conduct fast and accurate large-scale density functional theory (DFT) calculations (reaching ~ 100,000 electrons) on both many-core CPU and hybrid CPU-GPU computing architectures. This work involves improvements in the real-space formulation—via an improved treatment of the electrostatic interactions that substantially enhances the computational efficiency—as well high-performance computing aspects, including the GPU acceleration of all the key compute kernels in DFT-FE. We demonstrate the accuracy by comparing the ground-state energies, ionic forces and cell stresses on a wide-range of benchmark systems against those obtained from widely used DFT codes. Further, we demonstrate the numerical efficiency of our implementation, which yields ~ 20× CPU-GPU speed-up by using GPU acceleration on hybrid CPU-GPU nodes. Notably, owing to the parallel-scaling of the GPU implementation, we obtain wall-times of 80–140 seconds for full ground-state calculations, with stringent accuracy, on benchmark systems containing ~ 6, 000 – 15,000 electrons.

Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
Sponsoring Organization:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE Office of Science (SC), Basic Energy Sciences (BES); National Science Foundation (NSF); US Air Force Office of Scientific Research (AFOSR); US Army Research Office (ARO)
Grant/Contract Number:
AC05-00OR22725; SC0008637; AC02-05CH11231
OSTI ID:
1888932
Journal Information:
Computer Physics Communications, Journal Name: Computer Physics Communications Journal Issue: 1 Vol. 280; ISSN 0010-4655
Publisher:
ElsevierCopyright Statement
Country of Publication:
United States
Language:
English

References (83)

On the adaptive finite element analysis of the Kohn-Sham equations: methods, algorithms, and implementation: On the adaptive finite element analysis of the Kohn-Sham equations: methods, algorithms, and implementation
  • Davydov, Denis; Young, Toby D.; Steinmann, Paul
  • International Journal for Numerical Methods in Engineering, Vol. 106, Issue 11 https://doi.org/10.1002/nme.5140
journal November 2015
PARSEC – the pseudopotential algorithm for real-space electronic structure calculations: recent advances and novel applications to nano-structures journal April 2006
Large-scale DFT calculations in implicit solvent-A case study on the T4 lysozyme L99A/M102Q protein journal March 2012
cp2k: atomistic simulations of condensed matter systems
  • Hutter, Jürg; Iannuzzi, Marcella; Schiffmann, Florian
  • Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 4, Issue 1 https://doi.org/10.1002/wcms.1159
journal June 2013
First-principles computation of material properties: the ABINIT software project journal November 2002
Ab initio modeling of dislocation core properties in metals and semiconductors journal February 2017
A generic interface for parallel cell-based finite element operator application journal June 2012
Ab initio molecular simulations with numeric atom-centered orbitals journal November 2009
NWChem: A comprehensive and scalable open-source solution for large scale molecular simulations journal September 2010
Speeding up plane-wave electronic-structure calculations using graphics-processing units journal July 2011
Optimization algorithm for the generation of ONCV pseudopotentials journal November 2015
SPARC: Accurate and efficient finite-difference formulation and parallel implementation of Density Functional Theory: Isolated clusters journal March 2017
The PseudoDojo: Training and grading a 85 element optimized norm-conserving pseudopotential table journal May 2018
DFT-FE – A massively parallel adaptive finite-element code for large-scale density functional theory calculations journal January 2020
Daubechies wavelets for high performance electronic structure calculations: The BigDFT project journal February 2011
All-electron Kohn–Sham density functional theory on hierarchic finite element spaces journal October 2013
Higher-order adaptive finite-element methods for Kohn–Sham density functional theory journal November 2013
RESCU: A real space electronic structure method journal February 2016
Non-periodic finite-element formulation of orbital-free density functional theory journal April 2007
Non-periodic finite-element formulation of Kohn–Sham density functional theory journal February 2010
Electronic-structure study of an edge dislocation in Aluminum and the role of macroscopic deformations on its energetics journal March 2015
Electronic structure study of screw dislocation core energetics in Aluminum and core energetics informed forces in a dislocation aggregate journal July 2017
Optimizations of the eigensolvers in the ELPA library journal July 2019
Two-Level Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of Large-Scale Electronic Structure Calculations journal April 2018
A Comparison of Exact and Model Exchange–Correlation Potentials for Molecules journal December 2021
Beyond Magic Numbers: Atomic Scale Equilibrium Nanoparticle Shapes for Any Size journal August 2017
Real-Space Density Functional Theory on Graphical Processing Units: Computational Approach and Comparison to Gaussian Basis Set Methods journal September 2013
Adaptive Finite Element Method for Solving the Exact Kohn−Sham Equation of Density Functional Theory journal March 2009
Exact exchange-correlation potentials from ground-state electron densities journal October 2019
Review on modeling of the anode solid electrolyte interphase (SEI) for lithium-ion batteries journal March 2018
Tensor-structured algorithm for reduced-order scaling large-scale Kohn–Sham density functional theory calculations journal April 2021
Spin–spin interactions in defects in solids from mixed all-electron and pseudopotential first-principles calculations journal July 2021
Backbone charge transport in double-stranded DNA journal August 2020
Real-space grids and the Octopus code as tools for the development of new simulation approaches for electronic systems journal January 2015
DFT modelling of explicit solid–solid interfaces in batteries: methods and challenges journal January 2020
Polarization consistent basis sets. II. Estimating the Kohn–Sham basis set limit journal May 2002
Self‐Consistent Molecular‐Orbital Methods. I. Use of Gaussian Expansions of Slater‐Type Atomic Orbitals journal September 1969
Introducing ONETEP : Linear-scaling density functional simulations on parallel computers journal February 2005
Density functional theory calculation on many-cores hybrid central processing unit-graphic processing unit architectures journal July 2009
DGDFT: A massively parallel method for large scale density functional theory calculations journal September 2015
Discrete discontinuous basis projection method for large-scale electronic structure calculations journal September 2018
Orbital-free density functional theory study of the energetics of vacancy clustering and prismatic dislocation loop nucleation in aluminium journal July 2016
\mathcal{O}(N) methods in electronic structure calculations journal February 2012
QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials journal September 2009
Electronic structure calculations with GPAW: a real-space implementation of the projector augmented-wave method journal June 2010
The ELPA library: scalable parallel eigenvalue solutions for electronic structure theory and computational science journal May 2014
exciting: a full-potential all-electron package implementing density-functional theory and many-body perturbation theory journal August 2014
Applications of large-scale density functional theory in biology journal August 2016
Finite element methods in ab initio electronic structure calculations journal April 2005
Orbital-free density functional theory simulations of dislocations in magnesium journal November 2011
Advanced capabilities for materials modelling with Quantum ESPRESSO journal October 2017
Application of order-N first-principles DFT calculations with temperature controlled molecular dynamics to biomolecular system journal December 2018
Pushing the boundaries of lithium battery research with atomistic modelling on different scales journal December 2021
Self-Consistent Equations Including Exchange and Correlation Effects journal November 1965
Real time time-dependent density functional theory using higher order finite-element methods journal September 2019
Fast and robust all-electron density functional theory calculations in solids using orthogonalized enriched finite elements journal August 2021
Efficient iteration scheme for self-consistent pseudopotential calculations journal March 1981
Beyond the local-density approximation in calculations of ground-state electronic properties journal August 1983
Efficient pseudopotentials for plane-wave calculations journal January 1991
Projector augmented-wave method journal December 1994
Electronic-structure calculations based on the finite-element method journal August 1995
Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set journal October 1996
Adaptive finite-element method for electronic-structure calculations journal September 1996
Real-space local polynomial basis for solid-state electronic-structure calculations: A finite-element approach journal May 1999
Linear-scaling first-principles molecular dynamics with plane-waves accuracy journal March 2006
Subquadratic-scaling subspace projection method for large-scale Kohn-Sham density functional theory calculations using spectral finite-element discretization journal September 2014
Spectrum-splitting approach for Fermi-operator expansion in all-electron Kohn-Sham DFT calculations journal January 2017
Large-scale all-electron density functional theory calculations using an enriched finite-element basis journal January 2017
Configurational forces in electronic structure calculations using Kohn-Sham density functional theory journal April 2018
Dislocation Core Energies and Core Fields from First Principles journal February 2009
Efficacious Form for Model Pseudopotentials journal May 1982
Generalized Gradient Approximation Made Simple journal October 1996
Ab Initio Study of Screw Dislocations in Mo and Ta: A New Picture of Plasticity in bcc Transition Metals journal February 2000
Linear scaling electronic structure methods journal July 1999
The Chemistry of Deformation: How Solutes Soften Pure Metals journal December 2005
Adaptive Finite Element Approximations for Kohn--Sham Models journal January 2014
Algorithms and data structures for massively parallel generic adaptive finite element codes journal December 2011
Iterative Procedures for Nonlinear Integral Equations journal October 1965
The deal.II library, Version 9.0 journal December 2018
Ab initio theory of the nitrogen-vacancy center in diamond journal September 2019
First principles methods using CASTEP journal January 2005
SPARC: Accurate and efficient finite-difference formulation and parallel implementation of Density Functional Theory: Isolated clusters dataset January 2017
DFT-FE – A massively parallel adaptive finite-element code for large-scale density functional theory calculations dataset January 2019