TwoLevel Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of LargeScale Electronic Structure Calculations
Abstract
Copyright © 2018 American Chemical Society. We describe a novel iterative strategy for KohnSham density functional theory calculations aimed at large systems (>1,000 electrons), applicable to metals and insulators alike. In lieu of explicit diagonalization of the KohnSham Hamiltonian on every selfconsistent field (SCF) iteration, we employ a twolevel Chebyshev polynomial filter based complementary subspace strategy to (1) compute a set of vectors that span the occupied subspace of the Hamiltonian; (2) reduce subspace diagonalization to just partially occupied states; and (3) obtain those states in an efficient, scalable manner via an inner Chebyshev filter iteration. By reducing the necessary computation to just partially occupied states and obtaining these through an inner Chebyshev iteration, our approach reduces the cost of large metallic calculations significantly, while eliminating subspace diagonalization for insulating systems altogether. We describe the implementation of the method within the framework of the discontinuous Galerkin (DG) electronic structure method and show that this results in a computational scheme that can effectively tackle bulk and nano systems containing tens of thousands of electrons, with chemical accuracy, within a few minutes or less of wall clock time per SCF iteration on largescale computing platforms. We anticipate that our method will bemore »
 Authors:

 Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
 Univ. of California, Berkeley, CA (United States). Dept. of Mathematics; Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
 Georgia Inst. of Technology, Atlanta, GA (United States). College of Engineering
 Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States). Physics Division
 Publication Date:
 Research Org.:
 Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); University of California, Berkeley, CA (United States)
 Sponsoring Org.:
 USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE Office of Science (SC), Basic Energy Sciences (BES); National Science Foundation (NSF)
 OSTI Identifier:
 1512626
 Alternate Identifier(s):
 OSTI ID: 1526530
 Report Number(s):
 LLNLJRNL757223
Journal ID: ISSN 15499618; 943724
 Grant/Contract Number:
 AC5207NA27344; SC0017867; 1450372; AC0205CH11231
 Resource Type:
 Accepted Manuscript
 Journal Name:
 Journal of Chemical Theory and Computation
 Additional Journal Information:
 Journal Volume: 14; Journal Issue: 6; Journal ID: ISSN 15499618
 Publisher:
 American Chemical Society
 Country of Publication:
 United States
 Language:
 English
 Subject:
 71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS
Citation Formats
Banerjee, Amartya S., Lin, Lin, Suryanarayana, Phanish, Yang, Chao, and Pask, John E. TwoLevel Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of LargeScale Electronic Structure Calculations. United States: N. p., 2018.
Web. doi:10.1021/acs.jctc.7b01243.
Banerjee, Amartya S., Lin, Lin, Suryanarayana, Phanish, Yang, Chao, & Pask, John E. TwoLevel Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of LargeScale Electronic Structure Calculations. United States. https://doi.org/10.1021/acs.jctc.7b01243
Banerjee, Amartya S., Lin, Lin, Suryanarayana, Phanish, Yang, Chao, and Pask, John E. Mon .
"TwoLevel Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of LargeScale Electronic Structure Calculations". United States. https://doi.org/10.1021/acs.jctc.7b01243. https://www.osti.gov/servlets/purl/1512626.
@article{osti_1512626,
title = {TwoLevel Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of LargeScale Electronic Structure Calculations},
author = {Banerjee, Amartya S. and Lin, Lin and Suryanarayana, Phanish and Yang, Chao and Pask, John E.},
abstractNote = {Copyright © 2018 American Chemical Society. We describe a novel iterative strategy for KohnSham density functional theory calculations aimed at large systems (>1,000 electrons), applicable to metals and insulators alike. In lieu of explicit diagonalization of the KohnSham Hamiltonian on every selfconsistent field (SCF) iteration, we employ a twolevel Chebyshev polynomial filter based complementary subspace strategy to (1) compute a set of vectors that span the occupied subspace of the Hamiltonian; (2) reduce subspace diagonalization to just partially occupied states; and (3) obtain those states in an efficient, scalable manner via an inner Chebyshev filter iteration. By reducing the necessary computation to just partially occupied states and obtaining these through an inner Chebyshev iteration, our approach reduces the cost of large metallic calculations significantly, while eliminating subspace diagonalization for insulating systems altogether. We describe the implementation of the method within the framework of the discontinuous Galerkin (DG) electronic structure method and show that this results in a computational scheme that can effectively tackle bulk and nano systems containing tens of thousands of electrons, with chemical accuracy, within a few minutes or less of wall clock time per SCF iteration on largescale computing platforms. We anticipate that our method will be instrumental in pushing the envelope of largescale ab initio molecular dynamics. As a demonstration of this, we simulate a bulk silicon system containing 8,000 atoms at finite temperature, and obtain an average SCF step wall time of 51 s on 34,560 processors; thus allowing us to carry out 1.0 ps of ab initio molecular dynamics in approximately 28 h (of wall time).},
doi = {10.1021/acs.jctc.7b01243},
journal = {Journal of Chemical Theory and Computation},
number = 6,
volume = 14,
place = {United States},
year = {2018},
month = {4}
}
Web of Science
Figures / Tables:
Works referenced in this record:
On nearsightedness in metallic systems for $\mathcal{O}(N)$ Density Functional Theory calculations: A case study on aluminum
journal, July 2017
 Suryanarayana, Phanish
 Chemical Physics Letters, Vol. 679
SPARC: Accurate and efficient finitedifference formulation and parallel implementation of Density Functional Theory: Extended systems
dataset, January 2017
 Ghosh, Swarnava
 Mendeley
RESCU: A real space electronic structure method
journal, February 2016
 MichaudRioux, Vincent; Zhang, Lei; Guo, Hong
 Journal of Computational Physics, Vol. 307
Calculations for millions of atoms with density functional theory: linear scaling shows its potential
journal, February 2010
 Bowler, D. R.; Miyazaki, T.
 Journal of Physics: Condensed Matter, Vol. 22, Issue 7
A variational method for density functional theory calculations on metallic systems with thousands of atoms
journal, August 2013
 RuizSerrano, Álvaro; Skylaris, ChrisKriton
 The Journal of Chemical Physics, Vol. 139, Issue 5
SQDFT: Spectral Quadrature method for largescale parallel $\mathcal{O}\left(N\right)$ Kohn–Sham calculations at high temperature
journal, March 2018
 Suryanarayana, Phanish; Pratapa, Phanisri P.; Sharma, Abhiraj
 Computer Physics Communications, Vol. 224
Adaptive local basis set for Kohn–Sham density functional theory in a discontinuous Galerkin framework II: Force, vibration, and molecular dynamics calculations
journal, April 2017
 Zhang, Gaigong; Lin, Lin; Hu, Wei
 Journal of Computational Physics, Vol. 335
A spectral scheme for Kohn–Sham density functional theory of clusters
journal, April 2015
 Banerjee, Amartya S.; Elliott, Ryan S.; James, Richard D.
 Journal of Computational Physics, Vol. 287
Linearscaling densityfunctional theory with tens of thousands of atoms: Expanding the scope and scale of calculations with ONETEP
journal, July 2009
 Hine, N. D. M.; Haynes, P. D.; Mostofi, A. A.
 Computer Physics Communications, Vol. 180, Issue 7
SPARC: Accurate and efficient finitedifference formulation and parallel implementation of Density Functional Theory: Isolated clusters
journal, March 2017
 Ghosh, Swarnava; Suryanarayana, Phanish
 Computer Physics Communications, Vol. 212
Elliptic Preconditioner for Accelerating the SelfConsistent Field Iteration in KohnSham Density Functional Theory
journal, January 2013
 Lin, Lin; Yang, Chao
 SIAM Journal on Scientific Computing, Vol. 35, Issue 5
Parallel solution of partial symmetric eigenvalue problems from electronic structure calculations
journal, December 2011
 Auckenthaler, T.; Blum, V.; Bungartz, H. J.
 Parallel Computing, Vol. 37, Issue 12
Communication: Generalized canonical purification for density matrix minimization
journal, March 2016
 Truflandier, Lionel A.; Dianzinga, Rivo M.; Bowler, David R.
 The Journal of Chemical Physics, Vol. 144, Issue 9
\mathcal{O}(N) methods in electronic structure calculations
journal, February 2012
 Bowler, D. R.; Miyazaki, T.
 Reports on Progress in Physics, Vol. 75, Issue 3
Challenges in large scale quantum mechanical calculations: Challenges in large scale quantum mechanical calculations
journal, November 2016
 Ratcliff, Laura E.; Mohr, Stephan; Huhs, Georg
 Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 7, Issue 1
Optimized normconserving Vanderbilt pseudopotentials
journal, August 2013
 Hamann, D. R.
 Physical Review B, Vol. 88, Issue 8
Edge reconstruction in armchair phosphorene nanoribbons revealed by discontinuous Galerkin density functional theory
journal, January 2015
 Hu, Wei; Lin, Lin; Yang, Chao
 Physical Chemistry Chemical Physics, Vol. 17, Issue 47
Nearsightedness of electronic matter
journal, August 2005
 Prodan, E.; Kohn, W.
 Proceedings of the National Academy of Sciences, Vol. 102, Issue 33
Ab initiomolecular dynamics for liquid metals
journal, January 1993
 Kresse, G.; Hafner, J.
 Physical Review B, Vol. 47, Issue 1, p. 558561
Introducing ONETEP : Linearscaling density functional simulations on parallel computers
journal, February 2005
 Skylaris, ChrisKriton; Haynes, Peter D.; Mostofi, Arash A.
 The Journal of Chemical Physics, Vol. 122, Issue 8
Highprecision sampling for Brillouinzone integration in metals
journal, August 1989
 Methfessel, M.; Paxton, A. T.
 Physical Review B, Vol. 40, Issue 6
Unified Approach for Molecular Dynamics and DensityFunctional Theory
journal, November 1985
 Car, R.; Parrinello, M.
 Physical Review Letters, Vol. 55, Issue 22
A posteriori error estimates for discontinuous Galerkin methods using nonpolynomial basis functions Part I: Second order linear PDE
journal, July 2016
 Lin, Lin; Stamm, Benjamin
 ESAIM: Mathematical Modelling and Numerical Analysis, Vol. 50, Issue 4
Towards idempotent reduced density matrices via particlehole duality: McWeeny’s purification and beyond
journal, December 2003
 Mazziotti, David A.
 Physical Review E, Vol. 68, Issue 6
Insights into Current Limitations of Density Functional Theory
journal, August 2008
 Cohen, A. J.; MoriSanchez, P.; Yang, W.
 Science, Vol. 321, Issue 5890
SelfConsistent Equations Including Exchange and Correlation Effects
journal, November 1965
 Kohn, W.; Sham, L. J.
 Physical Review, Vol. 140, Issue 4A, p. A1133A1138
Convergence acceleration of iterative sequences. the case of scf iteration
journal, July 1980
 Pulay, Péter
 Chemical Physics Letters, Vol. 73, Issue 2
Modeling Materials
book, January 2011
 Tadmor, Ellad B.; Miller, Ronald E.
 Cambridge University Press
Accurate and efficient linear scaling DFT calculations with universal applicability
journal, January 2015
 Mohr, Stephan; Ratcliff, Laura E.; Genovese, Luigi
 Physical Chemistry Chemical Physics, Vol. 17, Issue 47
ScaLAPACK Users' Guide
book, January 1997
 Blackford, L. S.; Choi, J.; Cleary, A.
 Society for Industrial and Applied Mathematics
SIESTAPEXSI: massively parallel method for efficient and accurate ab initio materials simulation without matrix diagonalization
journal, July 2014
 Lin, Lin; García, Alberto; Huhs, Georg
 Journal of Physics: Condensed Matter, Vol. 26, Issue 30
SPARC: Accurate and efficient finitedifference formulation and parallel implementation of Density Functional Theory: Extended systems
journal, July 2017
 Ghosh, Swarnava; Suryanarayana, Phanish
 Computer Physics Communications, Vol. 216
Accelerating atomic orbitalbased electronic structure calculation via pole expansion and selected inversion
journal, June 2013
 Lin, Lin; Chen, Mohan; Yang, Chao
 Journal of Physics: Condensed Matter, Vol. 25, Issue 29
Efficiency of abinitio total energy calculations for metals and semiconductors using a planewave basis set
journal, July 1996
 Kresse, G.; Furthmüller, J.
 Computational Materials Science, Vol. 6, Issue 1, p. 1550
DGDFT: A massively parallel method for large scale density functional theory calculations
journal, September 2015
 Hu, Wei; Lin, Lin; Yang, Chao
 The Journal of Chemical Physics, Vol. 143, Issue 12
ELSI: A unified software interface for Kohn–Sham electronic structure solvers
journal, January 2018
 Yu, Victor Wenzhe; Corsetti, Fabiano; García, Alberto
 Computer Physics Communications, Vol. 222
Perspective: Methods for largescale density functional calculations on metallic systems
journal, December 2016
 Aarons, Jolyon; Sarwar, Misbah; Thompsett, David
 The Journal of Chemical Physics, Vol. 145, Issue 22
Toward the Optimal Preconditioned Eigensolver: Locally Optimal Block Preconditioned Conjugate Gradient Method
journal, January 2001
 Knyazev, Andrew V.
 SIAM Journal on Scientific Computing, Vol. 23, Issue 2
The Symmetric Eigenvalue Problem
book, January 1998
 Parlett, Beresford N.
 Classics in Applied Mathematics
<i>A posteriori</i> error estimator for adaptive local basis functions to solve Kohn–Sham density functional theory
journal, January 2015
 Kaye, Jason; Lin, Lin; Yang, Chao
 Communications in Mathematical Sciences, Vol. 13, Issue 7
Selfconsistentfield calculations using Chebyshevfiltered subspace iteration
journal, November 2006
 Zhou, Yunkai; Saad, Yousef; Tiago, Murilo L.
 Journal of Computational Physics, Vol. 219, Issue 1
Linear scaling electronic structure methods
journal, July 1999
 Goedecker, Stefan
 Reviews of Modern Physics, Vol. 71, Issue 4
Perspective: Fifty years of densityfunctional theory in chemical physics
journal, May 2014
 Becke, Axel D.
 The Journal of Chemical Physics, Vol. 140, Issue 18
A posteriori error estimates for discontinuous Galerkin methods using nonpolynomial basis functions. Part II: Eigenvalue problems
journal, September 2017
 Lin, Lin; Stamm, Benjamin
 ESAIM: Mathematical Modelling and Numerical Analysis, Vol. 51, Issue 5
Ab initio moleculardynamics simulation of the liquidmetal–amorphoussemiconductor transition in germanium
journal, May 1994
 Kresse, G.; Hafner, J.
 Physical Review B, Vol. 49, Issue 20, p. 1425114269
Cyclic density functional theory: A route to the first principles simulation of bending in nanostructures
journal, November 2016
 Banerjee, Amartya S.; Suryanarayana, Phanish
 Journal of the Mechanics and Physics of Solids, Vol. 96
Parallel selfconsistentfield calculations via Chebyshevfiltered subspace acceleration
journal, December 2006
 Zhou, Yunkai; Saad, Yousef; Tiago, Murilo L.
 Physical Review E, Vol. 74, Issue 6
Chebyshevfiltered subspace iteration method free of sparse diagonalization for solving the Kohn–Sham equation
journal, October 2014
 Zhou, Yunkai; Chelikowsky, James R.; Saad, Yousef
 Journal of Computational Physics, Vol. 274
Ground State of the Electron Gas by a Stochastic Method
journal, August 1980
 Ceperley, D. M.; Alder, B. J.
 Physical Review Letters, Vol. 45, Issue 7, p. 566569
Linear Scaling SelfConsistent Field Calculations with Millions of Atoms in the Condensed Phase
journal, March 2012
 VandeVondele, Joost; Borštnik, Urban; Hutter, Jürg
 Journal of Chemical Theory and Computation, Vol. 8, Issue 10
Linearscaling subspaceiteration algorithm with optimally localized nonorthogonal wave functions for KohnSham density functional theory
journal, March 2009
 GarcíaCervera, C. J.; Lu, Jianfeng; Xuan, Yulin
 Physical Review B, Vol. 79, Issue 11
Block Locally Optimal Preconditioned Eigenvalue Xolvers (BLOPEX) in Hypre and PETSc
journal, January 2007
 Knyazev, A. V.; Argentati, M. E.; Lashuk, I.
 SIAM Journal on Scientific Computing, Vol. 29, Issue 5
Variational minimization of atomic and molecular groundstate energies via the twoparticle reduced density matrix
journal, June 2002
 Mazziotti, David A.
 Physical Review A, Vol. 65, Issue 6
Thermal Properties of the Inhomogeneous Electron Gas
journal, March 1965
 Mermin, N. David
 Physical Review, Vol. 137, Issue 5A
An Interior Penalty Finite Element Method with Discontinuous Elements
journal, August 1982
 Arnold, Douglas N.
 SIAM Journal on Numerical Analysis, Vol. 19, Issue 4
Optimization algorithm for the generation of ONCV pseudopotentials
journal, November 2015
 Schlipf, Martin; Gygi, François
 Computer Physics Communications, Vol. 196
Periodic Pulay method for robust and efficient convergence acceleration of selfconsistent field iterations
journal, March 2016
 Banerjee, Amartya S.; Suryanarayana, Phanish; Pask, John E.
 Chemical Physics Letters, Vol. 647
The ELPA library: scalable parallel eigenvalue solutions for electronic structure theory and computational science
journal, May 2014
 Marek, A.; Blum, V.; Johanni, R.
 Journal of Physics: Condensed Matter, Vol. 26, Issue 21
Spectral Quadrature method for accurate $\mathcal{O}\left(N\right)$ electronic structure calculations of metals and insulators
journal, March 2016
 Pratapa, Phanisri P.; Suryanarayana, Phanish; Pask, John E.
 Computer Physics Communications, Vol. 200
Adaptive local basis set for Kohn–Sham density functional theory in a discontinuous Galerkin framework I: Total energy calculation
journal, February 2012
 Lin, Lin; Lu, Jianfeng; Ying, Lexing
 Journal of Computational Physics, Vol. 231, Issue 4
Perspective on density functional theory
journal, April 2012
 Burke, Kieron
 The Journal of Chemical Physics, Vol. 136, Issue 15
Chebyshev polynomial filtered subspace iteration in the discontinuous Galerkin method for largescale electronic structure calculations
journal, October 2016
 Banerjee, Amartya S.; Lin, Lin; Hu, Wei
 The Journal of Chemical Physics, Vol. 145, Issue 15
Subquadraticscaling subspace projection method for largescale KohnSham density functional theory calculations using spectral finiteelement discretization
journal, September 2014
 Motamarri, Phani; Gavini, Vikram
 Physical Review B, Vol. 90, Issue 11
Modeling Dilute Solutions Using FirstPrinciples Molecular Dynamics: Computing more than a Million Atoms with over a Million Cores
conference, November 2016
 Fattebert, JeanLuc; OseiKuffuor, Daniel; Draeger, Erik W.
 SC16: International Conference for High Performance Computing, Networking, Storage and Analysis
SPARC: Accurate and efficient finitedifference formulation and parallel implementation of Density Functional Theory: Isolated clusters
dataset, January 2017
 Ghosh, Swarnava
 Mendeley
Works referencing / citing this record:
Quantum Dynamics at Scale: Ultrafast Control of Emergent Functional Materials
conference, January 2020
 Tiwari, Subodh C.; Krishnamoorthy, Aravind; Rajak, Pankaj
 HPCAsia2020: International Conference on High Performance Computing in AsiaPacific Region, Proceedings of the International Conference on High Performance Computing in AsiaPacific Region
Symmetryadapted realspace density functional theory for cylindrical geometries: Application to large groupIV nanotubes
journal, September 2019
 Ghosh, Swarnava; Banerjee, Amartya S.; Suryanarayana, Phanish
 Physical Review B, Vol. 100, Issue 12
A Shift Selection Strategy for Parallel Shiftinvert Spectrum Slicing in Symmetric Selfconsistent Eigenvalue Computation
journal, November 2020
 WilliamsYoung, David B.; Beckman, Paul G.; Yang, Chao
 ACM Transactions on Mathematical Software, Vol. 46, Issue 4