Shift-and-invert parallel spectral transformation eigensolver: Massively parallel performance for density-functional based tight-binding
Abstract
The Shift-and-invert parallel spectral transformations (SIPs), a computational approach to solve sparse eigenvalue problems, is developed for massively parallel architectures with exceptional parallel scalability and robustness. The capabilities of SIPs are demonstrated by diagonalization of density-functional based tight-binding (DFTB) Hamiltonian and overlap matrices for single-wall metallic carbon nanotubes, diamond nanowires, and bulk diamond crystals. The largest (smallest) example studied is a 128,000 (2000) atom nanotube for which ~330,000 (~5600) eigenvalues and eigenfunctions are obtained in ~190 (~5) seconds when parallelized over 266,144 (16,384) Blue Gene/Q cores. Weak scaling and strong scaling of SIPs are analyzed and the performance of SIPs is compared with other novel methods. Different matrix ordering methods are investigated to reduce the cost of the factorization step, which dominates the time-to-solution at the strong scaling limit. As a result, a parallel implementation of assembling the density matrix from the distributed eigenvectors is demonstrated.
- Authors:
-
- Argonne National Lab. (ANL), Argonne, IL (United States)
- Univ. of Alabama, Tuscaloosa, AL (United States)
- Publication Date:
- Research Org.:
- Argonne National Laboratory (ANL), Argonne, IL (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Basic Energy Sciences (BES). Chemical Sciences, Geosciences, and Biosciences Division
- OSTI Identifier:
- 1248169
- Grant/Contract Number:
- AC02-06CH11357
- Resource Type:
- Journal Article: Accepted Manuscript
- Journal Name:
- Journal of Computational Chemistry
- Additional Journal Information:
- Journal Volume: 37; Journal Issue: 4; Journal ID: ISSN 0192-8651
- Publisher:
- Wiley
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING; 36 MATERIALS SCIENCE; 37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CHEMISTRY; sparse matrices; spectrum slicing; generalized eigenvalue problem; semi-empirical methods; tight-binding DFT
Citation Formats
Zhang, Hong, Zapol, Peter, Dixon, David A., Wagner, Albert F., and Keceli, Murat. Shift-and-invert parallel spectral transformation eigensolver: Massively parallel performance for density-functional based tight-binding. United States: N. p., 2015.
Web. doi:10.1002/jcc.24254.
Zhang, Hong, Zapol, Peter, Dixon, David A., Wagner, Albert F., & Keceli, Murat. Shift-and-invert parallel spectral transformation eigensolver: Massively parallel performance for density-functional based tight-binding. United States. https://doi.org/10.1002/jcc.24254
Zhang, Hong, Zapol, Peter, Dixon, David A., Wagner, Albert F., and Keceli, Murat. 2015.
"Shift-and-invert parallel spectral transformation eigensolver: Massively parallel performance for density-functional based tight-binding". United States. https://doi.org/10.1002/jcc.24254. https://www.osti.gov/servlets/purl/1248169.
@article{osti_1248169,
title = {Shift-and-invert parallel spectral transformation eigensolver: Massively parallel performance for density-functional based tight-binding},
author = {Zhang, Hong and Zapol, Peter and Dixon, David A. and Wagner, Albert F. and Keceli, Murat},
abstractNote = {The Shift-and-invert parallel spectral transformations (SIPs), a computational approach to solve sparse eigenvalue problems, is developed for massively parallel architectures with exceptional parallel scalability and robustness. The capabilities of SIPs are demonstrated by diagonalization of density-functional based tight-binding (DFTB) Hamiltonian and overlap matrices for single-wall metallic carbon nanotubes, diamond nanowires, and bulk diamond crystals. The largest (smallest) example studied is a 128,000 (2000) atom nanotube for which ~330,000 (~5600) eigenvalues and eigenfunctions are obtained in ~190 (~5) seconds when parallelized over 266,144 (16,384) Blue Gene/Q cores. Weak scaling and strong scaling of SIPs are analyzed and the performance of SIPs is compared with other novel methods. Different matrix ordering methods are investigated to reduce the cost of the factorization step, which dominates the time-to-solution at the strong scaling limit. As a result, a parallel implementation of assembling the density matrix from the distributed eigenvectors is demonstrated.},
doi = {10.1002/jcc.24254},
url = {https://www.osti.gov/biblio/1248169},
journal = {Journal of Computational Chemistry},
issn = {0192-8651},
number = 4,
volume = 37,
place = {United States},
year = {Tue Nov 17 00:00:00 EST 2015},
month = {Tue Nov 17 00:00:00 EST 2015}
}
Web of Science
Works referenced in this record:
Simplified LCAO Method for the Periodic Potential Problem
journal, June 1954
- Slater, J. C.; Koster, G. F.
- Physical Review, Vol. 94, Issue 6
Self-consistent-charge density-functional tight-binding method for simulations of complex materials properties
journal, September 1998
- Elstner, M.; Porezag, D.; Jungnickel, G.
- Physical Review B, Vol. 58, Issue 11, p. 7260-7268
Tight-Binding Density Functional Theory: An Approximate Kohn−Sham DFT Scheme †
journal, July 2007
- Seifert, G.
- The Journal of Physical Chemistry A, Vol. 111, Issue 26
Approximate Self‐Consistent Molecular Orbital Theory. I. Invariant Procedures
journal, November 1965
- Pople, J. A.; Santry, D. P.; Segal, G. A.
- The Journal of Chemical Physics, Vol. 43, Issue 10
Ground states of molecules. 38. The MNDO method. Approximations and parameters
journal, June 1977
- Dewar, Michael J. S.; Thiel, Walter
- Journal of the American Chemical Society, Vol. 99, Issue 15
Optimization of parameters for semiempirical methods I. Method
journal, March 1989
- Stewart, James J. P.
- Journal of Computational Chemistry, Vol. 10, Issue 2
Semiempirical quantum–chemical methods
journal, July 2013
- Thiel, Walter
- Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 4, Issue 2
Inhomogeneous Electron Gas
journal, November 1964
- Hohenberg, P.; Kohn, W.
- Physical Review, Vol. 136, Issue 3B, p. B864-B871
Self-Consistent Equations Including Exchange and Correlation Effects
journal, November 1965
- Kohn, W.; Sham, L. J.
- Physical Review, Vol. 140, Issue 4A, p. A1133-A1138
Assessment of Gaussian-3 and Density Functional Theories for Enthalpies of Formation of C 1 −C 16 Alkanes †
journal, June 2000
- Redfern, Paul C.; Zapol, Peter; Curtiss, Larry A.
- The Journal of Physical Chemistry A, Vol. 104, Issue 24
n -Alkane Isodesmic Reaction Energy Errors in Density Functional Theory Are Due to Electron Correlation Effects
journal, October 2010
- Grimme, Stefan
- Organic Letters, Vol. 12, Issue 20
Triplet states via intermediate neglect of differential overlap: Benzene, pyridine and the diazines
journal, January 1976
- Ridley, Joan E.; Zerner, Michael C.
- Theoretica Chimica Acta, Vol. 42, Issue 3
Semiempirical Molecular Orbital Methods
book, January 2007
- Zerner, Michael C.
- Reviews in Computational Chemistry, Volume 2
ONIOM: A Multilayered Integrated MO + MM Method for Geometry Optimizations and Single Point Energy Predictions. A Test for Diels−Alder Reactions and Pt(P( t -Bu) 3 ) 2 + H 2 Oxidative Addition
journal, January 1996
- Svensson, Mats; Humbel, Stéphane; Froese, Robert D. J.
- The Journal of Physical Chemistry, Vol. 100, Issue 50
SIPs: Shift-and-invert parallel spectral transformations
journal, June 2007
- Zhang, Hong; Smith, Barry; Sternberg, Michael
- ACM Transactions on Mathematical Software, Vol. 33, Issue 2
Strategies for spectrum slicing based on restarted Lanczos methods
journal, March 2012
- Campos, Carmen; Roman, Jose E.
- Numerical Algorithms, Vol. 60, Issue 2
The ELPA library: scalable parallel eigenvalue solutions for electronic structure theory and computational science
journal, May 2014
- Marek, A.; Blum, V.; Johanni, R.
- Journal of Physics: Condensed Matter, Vol. 26, Issue 21
Parallel eigenvalue calculation based on multiple shift–invert Lanczos and contour integral based spectral projection method
journal, July 2014
- Aktulga, Hasan Metin; Lin, Lin; Haine, Christopher
- Parallel Computing, Vol. 40, Issue 7
New algorithms for iterative matrix-free eigensolvers in quantum chemistry
journal, December 2014
- Zuev, Dmitry; Vecharynski, Eugene; Yang, Chao
- Journal of Computational Chemistry, Vol. 36, Issue 5
Density functional tight binding
journal, March 2014
- Elstner, Marcus; Seifert, Gotthard
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 372, Issue 2011
Direct calculation of electron density in density-functional theory
journal, March 1991
- Yang, Weitao
- Physical Review Letters, Vol. 66, Issue 11
What is the best alternative to diagonalization of the Hamiltonian in large scale semiempirical calculations?
journal, January 1999
- Daniels, Andrew D.; Scuseria, Gustavo E.
- The Journal of Chemical Physics, Vol. 110, Issue 3
SIESTA-PEXSI: massively parallel method for efficient and accurate ab initio materials simulation without matrix diagonalization
journal, July 2014
- Lin, Lin; García, Alberto; Huhs, Georg
- Journal of Physics: Condensed Matter, Vol. 26, Issue 30
An iteration method for the solution of the eigenvalue problem of linear differential and integral operators
journal, October 1950
- Lanczos, C.
- Journal of Research of the National Bureau of Standards, Vol. 45, Issue 4
A Jacobi--Davidson Iteration Method for Linear Eigenvalue Problems
journal, January 2000
- Sleijpen, Gerard L. G.; Van der Vorst, Henk A.
- SIAM Review, Vol. 42, Issue 2
XIX. A demonstration of the theorem that every homogeneous quadratic polynomial is reducible by real orthogonal substitutions to the form of a sum of positive and negative squares
journal, August 1852
- Sylvester, J. J.
- The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, Vol. 4, Issue 23
SLEPc: A scalable and flexible toolkit for the solution of eigenvalue problems
journal, September 2005
- Hernandez, Vicente; Roman, Jose E.; Vidal, Vicente
- ACM Transactions on Mathematical Software, Vol. 31, Issue 3
A Fully Asynchronous Multifrontal Solver Using Distributed Dynamic Scheduling
journal, January 2001
- Amestoy, Patrick R.; Duff, Iain S.; L'Excellent, Jean-Yves
- SIAM Journal on Matrix Analysis and Applications, Vol. 23, Issue 1
Hybrid scheduling for the parallel solution of linear systems
journal, February 2006
- Amestoy, Patrick R.; Guermouche, Abdou; L’Excellent, Jean-Yves
- Parallel Computing, Vol. 32, Issue 2
A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs
journal, January 1998
- Karypis, George; Kumar, Vipin
- SIAM Journal on Scientific Computing, Vol. 20, Issue 1
A Parallel Algorithm for Multilevel Graph Partitioning and Sparse Matrix Ordering
journal, January 1998
- Karypis, George; Kumar, Vipin
- Journal of Parallel and Distributed Computing, Vol. 48, Issue 1
PT-Scotch: A tool for efficient parallel graph ordering
journal, July 2008
- Chevalier, C.; Pellegrini, F.
- Parallel Computing, Vol. 34, Issue 6-8
DFTB+, a Sparse Matrix-Based Implementation of the DFTB Method †
journal, July 2007
- Aradi, B.; Hourahine, B.; Frauenheim, Th.
- The Journal of Physical Chemistry A, Vol. 111, Issue 26
Computing the Minimum Fill-In is NP-Complete
journal, March 1981
- Yannakakis, Mihalis
- SIAM Journal on Algebraic Discrete Methods, Vol. 2, Issue 1
Reducing the bandwidth of sparse symmetric matrices
conference, January 1969
- Cuthill, E.; McKee, J.
- Proceedings of the 1969 24th national conference on -
Sparse Matrices in MATLAB: Design and Implementation
journal, January 1992
- Gilbert, John R.; Moler, Cleve; Schreiber, Robert
- SIAM Journal on Matrix Analysis and Applications, Vol. 13, Issue 1
Nested Dissection of a Regular Finite Element Mesh
journal, April 1973
- George, Alan
- SIAM Journal on Numerical Analysis, Vol. 10, Issue 2
A Sparse Self-Consistent Field Algorithm and Its Parallel Implementation: Application to Density-Functional-Based Tight Binding
journal, May 2014
- Scemama, Anthony; Renon, Nicolas; Rapacioli, Mathias
- Journal of Chemical Theory and Computation, Vol. 10, Issue 6
Parallel Computing in Quantum Chemistry
reference-book, January 2008
- Janssen, Curtis L.; Nielsen, Ida M. B.
- CRC Press
High-Performance Solvers for Dense Hermitian Eigenproblems
journal, January 2013
- Petschow, M.; Peise, E.; Bientinesi, P.
- SIAM Journal on Scientific Computing, Vol. 35, Issue 1
Elemental: A New Framework for Distributed Memory Dense Matrix Computations
journal, February 2013
- Poulson, Jack; Marker, Bryan; van de Geijn, Robert A.
- ACM Transactions on Mathematical Software, Vol. 39, Issue 2
Performance of Dense Eigensolvers on BlueGene/Q
book, January 2014
- Gutheil, Inge; Münchhalfen, Jan Felix; Grotendorst, Johannes
- Parallel Processing and Applied Mathematics
ScaLAPACK: a portable linear algebra library for distributed memory computers — design issues and performance
journal, August 1996
- Choi, J.; Demmel, J.; Dhillon, I.
- Computer Physics Communications, Vol. 97, Issue 1-2
Fast semiempirical calculations: Fast Semiempirical Calculations
journal, June 1982
- Stewart, J. J. P.; Császár, P.; Pulay, P.
- Journal of Computational Chemistry, Vol. 3, Issue 2
Calculations for millions of atoms with density functional theory: linear scaling shows its potential
journal, February 2010
- Bowler, D. R.; Miyazaki, T.
- Journal of Physics: Condensed Matter, Vol. 22, Issue 7
Kohn−Sham Density Functional Theory Electronic Structure Calculations with Linearly Scaling Computational Time and Memory Usage
journal, December 2010
- Rudberg, Elias; Rubensson, Emanuel H.; Sałek, Paweł
- Journal of Chemical Theory and Computation, Vol. 7, Issue 2
Linear Scaling Self-Consistent Field Calculations with Millions of Atoms in the Condensed Phase
journal, March 2012
- VandeVondele, Joost; Borštnik, Urban; Hutter, Jürg
- Journal of Chemical Theory and Computation, Vol. 8, Issue 10
Sparse Projected-Gradient Method As a Linear-Scaling Low-Memory Alternative to Diagonalization in Self-Consistent Field Electronic Structure Calculations
journal, December 2012
- Birgin, Ernesto G.; Martı́nez, J. M.; Martı́nez, Leandro
- Journal of Chemical Theory and Computation, Vol. 9, Issue 2
Systematic sparse matrix error control for linear scaling electronic structure calculations
journal, January 2005
- Rubensson, Emanuel H.; Sałek, Paweł
- Journal of Computational Chemistry, Vol. 26, Issue 15
Numerical Methods for Electronic Structure Calculations of Materials
journal, January 2010
- Saad, Yousef; Chelikowsky, James R.; Shontz, Suzanne M.
- SIAM Review, Vol. 52, Issue 1
Pole-Based approximation of the Fermi-Dirac function
journal, August 2009
- Lin, Lin; Lu, Jianfeng; Ying, Lexing
- Chinese Annals of Mathematics, Series B, Vol. 30, Issue 6
SelInv---An Algorithm for Selected Inversion of a Sparse Symmetric Matrix
journal, February 2011
- Lin, Lin; Yang, Chao; Meza, Juan C.
- ACM Transactions on Mathematical Software, Vol. 37, Issue 4
A Fast Parallel Algorithm for Selected Inversion of Structured Sparse Matrices with Application to 2D Electronic Structure Calculations
journal, January 2011
- Lin, Lin; Yang, Chao; Lu, Jianfeng
- SIAM Journal on Scientific Computing, Vol. 33, Issue 3
Accelerating atomic orbital-based electronic structure calculation via pole expansion and selected inversion
journal, June 2013
- Lin, Lin; Chen, Mohan; Yang, Chao
- Journal of Physics: Condensed Matter, Vol. 25, Issue 29
A spectrum slicing method for the Kohn–Sham problem
journal, March 2012
- Schofield, Grady; Chelikowsky, James R.; Saad, Yousef
- Computer Physics Communications, Vol. 183, Issue 3
Improving the scalability of a symmetric iterative eigensolver for multi-core platforms: IMPROVING THE SCALABILITY OF A SYMMETRIC ITERATIVE EIGENSOLVER
journal, September 2013
- Aktulga, Hasan Metin; Yang, Chao; Ng, Esmond G.
- Concurrency and Computation: Practice and Experience, Vol. 26, Issue 16
Works referencing / citing this record:
Three pillars for achieving quantum mechanical molecular dynamics simulations of huge systems: Divide-and-conquer, density-functional tight-binding, and massively parallel computation
journal, June 2016
- Nishizawa, Hiroaki; Nishimura, Yoshifumi; Kobayashi, Masato
- Journal of Computational Chemistry, Vol. 37, Issue 21
Three-body expansion of the fragment molecular orbital method combined with density-functional tight-binding
journal, January 2017
- Nishimoto, Yoshio; Fedorov, Dmitri G.
- Journal of Computational Chemistry, Vol. 38, Issue 7
SIESTA-SIPs: Massively parallel spectrum-slicing eigensolver for an ab initio molecular dynamics package
journal, August 2018
- Keçeli, Murat; Corsetti, Fabiano; Campos, Carmen
- Journal of Computational Chemistry, Vol. 39, Issue 22
On the achievement of high fidelity and scalability for large-scale diagonalizations in grid-based DFT simulations
journal, March 2018
- Choi, Sunghwan; Kim, Woo Youn; Yeom, Min Sun
- International Journal of Quantum Chemistry, Vol. 118, Issue 16
Recent advances in quantum‐mechanical molecular dynamics simulations of proton transfer mechanism in various water‐based environments
journal, May 2019
- Sakti, Aditya W.; Nishimura, Yoshifumi; Nakai, Hiromi
- WIREs Computational Molecular Science, Vol. 10, Issue 1