DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Two-Level Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of Large-Scale Electronic Structure Calculations

Abstract

Copyright © 2018 American Chemical Society. We describe a novel iterative strategy for Kohn-Sham density functional theory calculations aimed at large systems (>1,000 electrons), applicable to metals and insulators alike. In lieu of explicit diagonalization of the Kohn-Sham Hamiltonian on every self-consistent field (SCF) iteration, we employ a two-level Chebyshev polynomial filter based complementary subspace strategy to (1) compute a set of vectors that span the occupied subspace of the Hamiltonian; (2) reduce subspace diagonalization to just partially occupied states; and (3) obtain those states in an efficient, scalable manner via an inner Chebyshev filter iteration. By reducing the necessary computation to just partially occupied states and obtaining these through an inner Chebyshev iteration, our approach reduces the cost of large metallic calculations significantly, while eliminating subspace diagonalization for insulating systems altogether. We describe the implementation of the method within the framework of the discontinuous Galerkin (DG) electronic structure method and show that this results in a computational scheme that can effectively tackle bulk and nano systems containing tens of thousands of electrons, with chemical accuracy, within a few minutes or less of wall clock time per SCF iteration on large-scale computing platforms. We anticipate that our method will bemore » instrumental in pushing the envelope of large-scale ab initio molecular dynamics. As a demonstration of this, we simulate a bulk silicon system containing 8,000 atoms at finite temperature, and obtain an average SCF step wall time of 51 s on 34,560 processors; thus allowing us to carry out 1.0 ps of ab initio molecular dynamics in approximately 28 h (of wall time).« less

Authors:
ORCiD logo [1]; ORCiD logo [2];  [3];  [1];  [4]
  1. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
  2. Univ. of California, Berkeley, CA (United States). Dept. of Mathematics; Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
  3. Georgia Inst. of Technology, Atlanta, GA (United States). College of Engineering
  4. Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States). Physics Division
Publication Date:
Research Org.:
Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); University of California, Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE Office of Science (SC), Basic Energy Sciences (BES); National Science Foundation (NSF)
OSTI Identifier:
1512626
Alternate Identifier(s):
OSTI ID: 1526530
Report Number(s):
LLNL-JRNL-757223
Journal ID: ISSN 1549-9618; 943724
Grant/Contract Number:  
AC52-07NA27344; SC0017867; 1450372; AC02-05CH11231
Resource Type:
Accepted Manuscript
Journal Name:
Journal of Chemical Theory and Computation
Additional Journal Information:
Journal Volume: 14; Journal Issue: 6; Journal ID: ISSN 1549-9618
Publisher:
American Chemical Society
Country of Publication:
United States
Language:
English
Subject:
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS

Citation Formats

Banerjee, Amartya S., Lin, Lin, Suryanarayana, Phanish, Yang, Chao, and Pask, John E. Two-Level Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of Large-Scale Electronic Structure Calculations. United States: N. p., 2018. Web. doi:10.1021/acs.jctc.7b01243.
Banerjee, Amartya S., Lin, Lin, Suryanarayana, Phanish, Yang, Chao, & Pask, John E. Two-Level Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of Large-Scale Electronic Structure Calculations. United States. https://doi.org/10.1021/acs.jctc.7b01243
Banerjee, Amartya S., Lin, Lin, Suryanarayana, Phanish, Yang, Chao, and Pask, John E. Mon . "Two-Level Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of Large-Scale Electronic Structure Calculations". United States. https://doi.org/10.1021/acs.jctc.7b01243. https://www.osti.gov/servlets/purl/1512626.
@article{osti_1512626,
title = {Two-Level Chebyshev Filter Based Complementary Subspace Method: Pushing the Envelope of Large-Scale Electronic Structure Calculations},
author = {Banerjee, Amartya S. and Lin, Lin and Suryanarayana, Phanish and Yang, Chao and Pask, John E.},
abstractNote = {Copyright © 2018 American Chemical Society. We describe a novel iterative strategy for Kohn-Sham density functional theory calculations aimed at large systems (>1,000 electrons), applicable to metals and insulators alike. In lieu of explicit diagonalization of the Kohn-Sham Hamiltonian on every self-consistent field (SCF) iteration, we employ a two-level Chebyshev polynomial filter based complementary subspace strategy to (1) compute a set of vectors that span the occupied subspace of the Hamiltonian; (2) reduce subspace diagonalization to just partially occupied states; and (3) obtain those states in an efficient, scalable manner via an inner Chebyshev filter iteration. By reducing the necessary computation to just partially occupied states and obtaining these through an inner Chebyshev iteration, our approach reduces the cost of large metallic calculations significantly, while eliminating subspace diagonalization for insulating systems altogether. We describe the implementation of the method within the framework of the discontinuous Galerkin (DG) electronic structure method and show that this results in a computational scheme that can effectively tackle bulk and nano systems containing tens of thousands of electrons, with chemical accuracy, within a few minutes or less of wall clock time per SCF iteration on large-scale computing platforms. We anticipate that our method will be instrumental in pushing the envelope of large-scale ab initio molecular dynamics. As a demonstration of this, we simulate a bulk silicon system containing 8,000 atoms at finite temperature, and obtain an average SCF step wall time of 51 s on 34,560 processors; thus allowing us to carry out 1.0 ps of ab initio molecular dynamics in approximately 28 h (of wall time).},
doi = {10.1021/acs.jctc.7b01243},
journal = {Journal of Chemical Theory and Computation},
number = 6,
volume = 14,
place = {United States},
year = {2018},
month = {4}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 25 works
Citation information provided by
Web of Science

Figures / Tables:

Table 1 Table 1: Values of various parameters used for the CS2CF strategy in ground-state calculations at typical electronic temperatures. Note that, as suggested in previous work, the first SCF step of a ground-state calculation employs multliple (regular) CheFSI cycles (typically, 3 − 4) while starting from randomly initialized wavefunctions.

Save / Share:

Works referenced in this record:

RESCU: A real space electronic structure method
journal, February 2016


Calculations for millions of atoms with density functional theory: linear scaling shows its potential
journal, February 2010


A variational method for density functional theory calculations on metallic systems with thousands of atoms
journal, August 2013

  • Ruiz-Serrano, Álvaro; Skylaris, Chris-Kriton
  • The Journal of Chemical Physics, Vol. 139, Issue 5
  • DOI: 10.1063/1.4817001

SQDFT: Spectral Quadrature method for large-scale parallel O ( N ) Kohn–Sham calculations at high temperature
journal, March 2018

  • Suryanarayana, Phanish; Pratapa, Phanisri P.; Sharma, Abhiraj
  • Computer Physics Communications, Vol. 224
  • DOI: 10.1016/j.cpc.2017.12.003

A spectral scheme for Kohn–Sham density functional theory of clusters
journal, April 2015

  • Banerjee, Amartya S.; Elliott, Ryan S.; James, Richard D.
  • Journal of Computational Physics, Vol. 287
  • DOI: 10.1016/j.jcp.2015.02.009

Linear-scaling density-functional theory with tens of thousands of atoms: Expanding the scope and scale of calculations with ONETEP
journal, July 2009

  • Hine, N. D. M.; Haynes, P. D.; Mostofi, A. A.
  • Computer Physics Communications, Vol. 180, Issue 7
  • DOI: 10.1016/j.cpc.2008.12.023

SPARC: Accurate and efficient finite-difference formulation and parallel implementation of Density Functional Theory: Isolated clusters
journal, March 2017


Elliptic Preconditioner for Accelerating the Self-Consistent Field Iteration in Kohn--Sham Density Functional Theory
journal, January 2013

  • Lin, Lin; Yang, Chao
  • SIAM Journal on Scientific Computing, Vol. 35, Issue 5
  • DOI: 10.1137/120880604

Parallel solution of partial symmetric eigenvalue problems from electronic structure calculations
journal, December 2011


Communication: Generalized canonical purification for density matrix minimization
journal, March 2016

  • Truflandier, Lionel A.; Dianzinga, Rivo M.; Bowler, David R.
  • The Journal of Chemical Physics, Vol. 144, Issue 9
  • DOI: 10.1063/1.4943213

\mathcal{O}(N) methods in electronic structure calculations
journal, February 2012


Challenges in large scale quantum mechanical calculations: Challenges in large scale quantum mechanical calculations
journal, November 2016

  • Ratcliff, Laura E.; Mohr, Stephan; Huhs, Georg
  • Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 7, Issue 1
  • DOI: 10.1002/wcms.1290

Optimized norm-conserving Vanderbilt pseudopotentials
journal, August 2013


Edge reconstruction in armchair phosphorene nanoribbons revealed by discontinuous Galerkin density functional theory
journal, January 2015

  • Hu, Wei; Lin, Lin; Yang, Chao
  • Physical Chemistry Chemical Physics, Vol. 17, Issue 47
  • DOI: 10.1039/C5CP00333D

Nearsightedness of electronic matter
journal, August 2005

  • Prodan, E.; Kohn, W.
  • Proceedings of the National Academy of Sciences, Vol. 102, Issue 33
  • DOI: 10.1073/pnas.0505436102

Ab initiomolecular dynamics for liquid metals
journal, January 1993


Electronic Structure
book, January 2004


Introducing ONETEP : Linear-scaling density functional simulations on parallel computers
journal, February 2005

  • Skylaris, Chris-Kriton; Haynes, Peter D.; Mostofi, Arash A.
  • The Journal of Chemical Physics, Vol. 122, Issue 8
  • DOI: 10.1063/1.1839852

High-precision sampling for Brillouin-zone integration in metals
journal, August 1989


Unified Approach for Molecular Dynamics and Density-Functional Theory
journal, November 1985


A posteriori error estimates for discontinuous Galerkin methods using non-polynomial basis functions Part I: Second order linear PDE
journal, July 2016

  • Lin, Lin; Stamm, Benjamin
  • ESAIM: Mathematical Modelling and Numerical Analysis, Vol. 50, Issue 4
  • DOI: 10.1051/m2an/2015069

Towards idempotent reduced density matrices via particle-hole duality: McWeeny’s purification and beyond
journal, December 2003


Insights into Current Limitations of Density Functional Theory
journal, August 2008


Self-Consistent Equations Including Exchange and Correlation Effects
journal, November 1965


Convergence acceleration of iterative sequences. the case of scf iteration
journal, July 1980


Modeling Materials
book, January 2011


Accurate and efficient linear scaling DFT calculations with universal applicability
journal, January 2015

  • Mohr, Stephan; Ratcliff, Laura E.; Genovese, Luigi
  • Physical Chemistry Chemical Physics, Vol. 17, Issue 47
  • DOI: 10.1039/C5CP00437C

ScaLAPACK Users' Guide
book, January 1997

  • Blackford, L. S.; Choi, J.; Cleary, A.
  • Society for Industrial and Applied Mathematics
  • DOI: 10.1137/1.9780898719642

SIESTA-PEXSI: massively parallel method for efficient and accurate ab initio materials simulation without matrix diagonalization
journal, July 2014


SPARC: Accurate and efficient finite-difference formulation and parallel implementation of Density Functional Theory: Extended systems
journal, July 2017


Accelerating atomic orbital-based electronic structure calculation via pole expansion and selected inversion
journal, June 2013


Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set
journal, July 1996


DGDFT: A massively parallel method for large scale density functional theory calculations
journal, September 2015

  • Hu, Wei; Lin, Lin; Yang, Chao
  • The Journal of Chemical Physics, Vol. 143, Issue 12
  • DOI: 10.1063/1.4931732

ELSI: A unified software interface for Kohn–Sham electronic structure solvers
journal, January 2018

  • Yu, Victor Wen-zhe; Corsetti, Fabiano; García, Alberto
  • Computer Physics Communications, Vol. 222
  • DOI: 10.1016/j.cpc.2017.09.007

Perspective: Methods for large-scale density functional calculations on metallic systems
journal, December 2016

  • Aarons, Jolyon; Sarwar, Misbah; Thompsett, David
  • The Journal of Chemical Physics, Vol. 145, Issue 22
  • DOI: 10.1063/1.4972007

Toward the Optimal Preconditioned Eigensolver: Locally Optimal Block Preconditioned Conjugate Gradient Method
journal, January 2001


The Symmetric Eigenvalue Problem
book, January 1998


<i>A posteriori</i> error estimator for adaptive local basis functions to solve Kohn–Sham density functional theory
journal, January 2015


Self-consistent-field calculations using Chebyshev-filtered subspace iteration
journal, November 2006

  • Zhou, Yunkai; Saad, Yousef; Tiago, Murilo L.
  • Journal of Computational Physics, Vol. 219, Issue 1
  • DOI: 10.1016/j.jcp.2006.03.017

Linear scaling electronic structure methods
journal, July 1999


Perspective: Fifty years of density-functional theory in chemical physics
journal, May 2014

  • Becke, Axel D.
  • The Journal of Chemical Physics, Vol. 140, Issue 18
  • DOI: 10.1063/1.4869598

A posteriori error estimates for discontinuous Galerkin methods using non-polynomial basis functions. Part II: Eigenvalue problems
journal, September 2017

  • Lin, Lin; Stamm, Benjamin
  • ESAIM: Mathematical Modelling and Numerical Analysis, Vol. 51, Issue 5
  • DOI: 10.1051/m2an/2016081

Ab initio molecular-dynamics simulation of the liquid-metal–amorphous-semiconductor transition in germanium
journal, May 1994


Cyclic density functional theory: A route to the first principles simulation of bending in nanostructures
journal, November 2016

  • Banerjee, Amartya S.; Suryanarayana, Phanish
  • Journal of the Mechanics and Physics of Solids, Vol. 96
  • DOI: 10.1016/j.jmps.2016.08.007

Parallel self-consistent-field calculations via Chebyshev-filtered subspace acceleration
journal, December 2006


Chebyshev-filtered subspace iteration method free of sparse diagonalization for solving the Kohn–Sham equation
journal, October 2014

  • Zhou, Yunkai; Chelikowsky, James R.; Saad, Yousef
  • Journal of Computational Physics, Vol. 274
  • DOI: 10.1016/j.jcp.2014.06.056

Ground State of the Electron Gas by a Stochastic Method
journal, August 1980


Linear Scaling Self-Consistent Field Calculations with Millions of Atoms in the Condensed Phase
journal, March 2012

  • VandeVondele, Joost; Borštnik, Urban; Hutter, Jürg
  • Journal of Chemical Theory and Computation, Vol. 8, Issue 10
  • DOI: 10.1021/ct200897x

Linear-scaling subspace-iteration algorithm with optimally localized nonorthogonal wave functions for Kohn-Sham density functional theory
journal, March 2009


Block Locally Optimal Preconditioned Eigenvalue Xolvers (BLOPEX) in Hypre and PETSc
journal, January 2007

  • Knyazev, A. V.; Argentati, M. E.; Lashuk, I.
  • SIAM Journal on Scientific Computing, Vol. 29, Issue 5
  • DOI: 10.1137/060661624

Thermal Properties of the Inhomogeneous Electron Gas
journal, March 1965


An Interior Penalty Finite Element Method with Discontinuous Elements
journal, August 1982

  • Arnold, Douglas N.
  • SIAM Journal on Numerical Analysis, Vol. 19, Issue 4
  • DOI: 10.1137/0719052

Optimization algorithm for the generation of ONCV pseudopotentials
journal, November 2015


Periodic Pulay method for robust and efficient convergence acceleration of self-consistent field iterations
journal, March 2016


The ELPA library: scalable parallel eigenvalue solutions for electronic structure theory and computational science
journal, May 2014


Spectral Quadrature method for accurate O ( N ) electronic structure calculations of metals and insulators
journal, March 2016

  • Pratapa, Phanisri P.; Suryanarayana, Phanish; Pask, John E.
  • Computer Physics Communications, Vol. 200
  • DOI: 10.1016/j.cpc.2015.11.005

Adaptive local basis set for Kohn–Sham density functional theory in a discontinuous Galerkin framework I: Total energy calculation
journal, February 2012


Perspective on density functional theory
journal, April 2012

  • Burke, Kieron
  • The Journal of Chemical Physics, Vol. 136, Issue 15
  • DOI: 10.1063/1.4704546

Chebyshev polynomial filtered subspace iteration in the discontinuous Galerkin method for large-scale electronic structure calculations
journal, October 2016

  • Banerjee, Amartya S.; Lin, Lin; Hu, Wei
  • The Journal of Chemical Physics, Vol. 145, Issue 15
  • DOI: 10.1063/1.4964861

Ab Initio Molecular Dynamics
book, January 2009


Modeling Dilute Solutions Using First-Principles Molecular Dynamics: Computing more than a Million Atoms with over a Million Cores
conference, November 2016

  • Fattebert, Jean-Luc; Osei-Kuffuor, Daniel; Draeger, Erik W.
  • SC16: International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2016.88

Works referencing / citing this record:

Quantum Dynamics at Scale: Ultrafast Control of Emergent Functional Materials
conference, January 2020

  • Tiwari, Subodh C.; Krishnamoorthy, Aravind; Rajak, Pankaj
  • HPCAsia2020: International Conference on High Performance Computing in Asia-Pacific Region, Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region
  • DOI: 10.1145/3368474.3368489

Symmetry-adapted real-space density functional theory for cylindrical geometries: Application to large group-IV nanotubes
journal, September 2019


A Shift Selection Strategy for Parallel Shift-invert Spectrum Slicing in Symmetric Self-consistent Eigenvalue Computation
journal, November 2020

  • Williams-Young, David B.; Beckman, Paul G.; Yang, Chao
  • ACM Transactions on Mathematical Software, Vol. 46, Issue 4
  • DOI: 10.1145/3409571