U.S. Department of EnergyOffice of Scientific and Technical Information
Massively parallel first-principles simulation of electron dynamics in materials
Abstract
Here we present a highly scalable, parallel implementation of first-principles electron dynamics coupled with molecular dynamics (MD). By using optimized kernels, network topology aware communication, and by fully distributing all terms in the time-dependent Kohn–Sham equation, we demonstrate unprecedented time to solution for disordered aluminum systems of 2000 atoms (22,000 electrons) and 5400 atoms (59,400 electrons), with wall clock time as low as 7.5 s per MD time step. Despite a significant amount of non-local communication required in every iteration, we achieved excellent strong scaling and sustained performance on the Sequoia Blue Gene/Q supercomputer at LLNL. We obtained up to 59% of the theoretical sustained peak performance on 16,384 nodes and performance of 8.75 Petaflop/s (43% of theoretical peak) on the full 98,304 node machine (1,572,864 cores). Lastly, scalable explicit electron dynamics allows for the study of phenomena beyond the reach of standard first-principles MD, in particular, materials subject to strong or rapid perturbations, such as pulsed electromagnetic radiation, particle irradiation, or strong electric currents.
- Publication Date:
- Research Org.:
- Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
- Sponsoring Org.:
- USDOE National Nuclear Security Administration (NNSA)
- OSTI Identifier:
- 1349002
- Alternate Identifier(s):
- OSTI ID: 1415865
- Report Number(s):
- LLNL-JRNL-670641
Journal ID: ISSN 0743-7315
- Grant/Contract Number:
- AC52-07NA27344
- Resource Type:
- Journal Article: Accepted Manuscript
- Journal Name:
- Journal of Parallel and Distributed Computing
- Additional Journal Information:
- Journal Volume: 106; Journal Issue: C; Journal ID: ISSN 0743-7315
- Publisher:
- Elsevier
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND SUPERFLUIDITY; 36 MATERIALS SCIENCE; 97 MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; Electron dynamics; TDDFT; Explicit time integration; Blue Gene/Q
Citation Formats
Draeger, Erik W., Andrade, Xavier, Gunnels, John A., Bhatele, Abhinav, Schleife, André, and Correa, Alfredo A. Massively parallel first-principles simulation of electron dynamics in materials. United States: N. p., 2017.
Web. doi:10.1016/j.jpdc.2017.02.005.
Draeger, Erik W., Andrade, Xavier, Gunnels, John A., Bhatele, Abhinav, Schleife, André, & Correa, Alfredo A. Massively parallel first-principles simulation of electron dynamics in materials. United States. https://doi.org/10.1016/j.jpdc.2017.02.005
Draeger, Erik W., Andrade, Xavier, Gunnels, John A., Bhatele, Abhinav, Schleife, André, and Correa, Alfredo A. 2017.
"Massively parallel first-principles simulation of electron dynamics in materials". United States. https://doi.org/10.1016/j.jpdc.2017.02.005. https://www.osti.gov/servlets/purl/1349002.
@article{osti_1349002,
title = {Massively parallel first-principles simulation of electron dynamics in materials},
author = {Draeger, Erik W. and Andrade, Xavier and Gunnels, John A. and Bhatele, Abhinav and Schleife, André and Correa, Alfredo A.},
abstractNote = {Here we present a highly scalable, parallel implementation of first-principles electron dynamics coupled with molecular dynamics (MD). By using optimized kernels, network topology aware communication, and by fully distributing all terms in the time-dependent Kohn–Sham equation, we demonstrate unprecedented time to solution for disordered aluminum systems of 2000 atoms (22,000 electrons) and 5400 atoms (59,400 electrons), with wall clock time as low as 7.5 s per MD time step. Despite a significant amount of non-local communication required in every iteration, we achieved excellent strong scaling and sustained performance on the Sequoia Blue Gene/Q supercomputer at LLNL. We obtained up to 59% of the theoretical sustained peak performance on 16,384 nodes and performance of 8.75 Petaflop/s (43% of theoretical peak) on the full 98,304 node machine (1,572,864 cores). Lastly, scalable explicit electron dynamics allows for the study of phenomena beyond the reach of standard first-principles MD, in particular, materials subject to strong or rapid perturbations, such as pulsed electromagnetic radiation, particle irradiation, or strong electric currents.},
doi = {10.1016/j.jpdc.2017.02.005},
url = {https://www.osti.gov/biblio/1349002},
journal = {Journal of Parallel and Distributed Computing},
issn = {0743-7315},
number = C,
volume = 106,
place = {United States},
year = {Tue Aug 01 00:00:00 EDT 2017},
month = {Tue Aug 01 00:00:00 EDT 2017}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Cited by: 33 works
Citation information provided by
Web of Science
Web of Science
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Modified Ehrenfest Formalism for Efficient Large-Scale ab initio Molecular Dynamics
journal, March 2009
- Andrade, Xavier; Castro, Alberto; Zueco, David
- Journal of Chemical Theory and Computation, Vol. 5, Issue 4
Application of compressed sensing to the simulation of atomic systems
journal, August 2012
- Andrade, X.; Sanders, J. N.; Aspuru-Guzik, A.
- Proceedings of the National Academy of Sciences, Vol. 109, Issue 35
Unified Approach for Molecular Dynamics and Density-Functional Theory
journal, November 1985
- Car, R.; Parrinello, M.
- Physical Review Letters, Vol. 55, Issue 22
octopus: a tool for the application of time-dependent density functional theory
journal, September 2006
- Castro, Alberto; Appel, Heiko; Oliveira, Micael
- physica status solidi (b), Vol. 243, Issue 11
Excited states dynamics in time-dependent density functional theory
journal, February 2004
- Castro, A.; Marques, M. A. L.; Alonso, J. A.
- The European Physical Journal D - Atomic, Molecular and Optical Physics, Vol. 28, Issue 2
Controlling the Dynamics of Many-Electron Systems from First Principles: A Combination of Optimal Control and Time-Dependent Density-Functional Theory
journal, October 2012
- Castro, A.; Werschnik, J.; Gross, E. K. U.
- Physical Review Letters, Vol. 109, Issue 15
Electrical conductivity for warm, dense aluminum plasmas and liquids
journal, August 2002
- Desjarlais, M. P.; Kress, J. D.; Collins, L. A.
- Physical Review E, Vol. 66, Issue 2
Current-driven atomic waterwheels
journal, February 2009
- Dundas, Daniel; McEniry, Eunan J.; Todorov, Tchavdar N.
- Nature Nanotechnology, Vol. 4, Issue 2
A survey of the parallel performance and accuracy of Poisson solvers for electronic structure calculations
journal, November 2013
- García-Risueño, Pablo; Alberdi-Rodriguez, Joseba; Oliveira, Micael J. T.
- Journal of Computational Chemistry, Vol. 35, Issue 6
Ab initio angle- and energy-resolved photoelectron spectroscopy with time-dependent density-functional theory
journal, June 2012
- De Giovannini, U.; Varsano, D.; Marques, M. A. L.
- Physical Review A, Vol. 85, Issue 6
Gordon Bell finalists I---Large-scale electronic structure calculations of high-Z metals on the BlueGene/L platform
conference, January 2006
- Gygi, Francois; Ueberhuber, Christoph W.; Lorenz, Juergen
- Proceedings of the 2006 ACM/IEEE conference on Supercomputing - SC '06
Inhomogeneous Electron Gas
journal, November 1964
- Hohenberg, P.; Kohn, W.
- Physical Review, Vol. 136, Issue 3B, p. B864-B871
Self-Consistent Equations Including Exchange and Correlation Effects
journal, November 1965
- Kohn, W.; Sham, L. J.
- Physical Review, Vol. 140, Issue 4A, p. A1133-A1138
Scalable High-Performance Algorithm for the Simulation of Exciton Dynamics. Application to the Light-Harvesting Complex II in the Presence of Resonant Vibrational Modes
journal, August 2014
- Kreisbeck, Christoph; Kramer, Tobias; Aspuru-Guzik, Alán
- Journal of Chemical Theory and Computation, Vol. 10, Issue 9
Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set
journal, October 1996
- Kresse, G.; Furthmüller, J.
- Physical Review B, Vol. 54, Issue 16, p. 11169-11186
Time-dependent quantum transport: A practical scheme using density functional theory
journal, July 2005
- Kurth, S.; Stefanucci, G.; Almbladh, C. -O.
- Physical Review B, Vol. 72, Issue 3
Ab initio quantum dynamics using coupled-cluster
journal, May 2012
- Kvaal, Simen
- The Journal of Chemical Physics, Vol. 136, Issue 19
Time-Dependent Density Functional Theory
journal, June 2004
- Marques, M. A. L.; Gross, E. K. U.
- Annual Review of Physical Chemistry, Vol. 55, Issue 1
Nonlocal Hermitian norm-conserving Vanderbilt pseudopotential
journal, March 1993
- Morrison, Ian; Bylander, D. M.; Kleinman, Leonard
- Physical Review B, Vol. 47, Issue 11
Massively-parallel electron dynamics calculations in real-time and real-space: Toward applications to nanostructures of more than ten-nanometers in size
journal, May 2014
- Noda, Masashi; Ishimura, Kazuya; Nobusada, Katsuyuki
- Journal of Computational Physics, Vol. 265
First-principles electron dynamics simulation for optical breakdown of dielectrics under an intense laser field
journal, April 2008
- Otobe, T.; Yamagiwa, M.; Iwata, J. -I.
- Physical Review B, Vol. 77, Issue 16
Time-dependent density functional theory with ultrasoft pseudopotentials: Real-time electron propagation across a molecular junction
journal, January 2006
- Qian, Xiaofeng; Li, Ju; Lin, Xi
- Physical Review B, Vol. 73, Issue 3
Conductivity of warm dense matter including electron-electron collisions
journal, April 2015
- Reinholz, H.; Röpke, G.; Rosmej, S.
- Physical Review E, Vol. 91, Issue 4
Density-Functional Theory for Time-Dependent Systems
journal, March 1984
- Runge, Erich; Gross, E. K. U.
- Physical Review Letters, Vol. 52, Issue 12
Quantum Dynamics Simulation of Electrons in Materials on High-Performance Computers
journal, September 2014
- Schleife, Andre; Draeger, Erik W.; Anisimov, Victor M.
- Computing in Science & Engineering, Vol. 16, Issue 5
Plane-wave pseudopotential implementation of explicit integrators for time-dependent Kohn-Sham equations in large-scale simulations
journal, December 2012
- Schleife, André; Draeger, Erik W.; Kanai, Yosuke
- The Journal of Chemical Physics, Vol. 137, Issue 22
Accurate atomistic first-principles calculations of electronic stopping
journal, January 2015
- Schleife, André; Kanai, Yosuke; Correa, Alfredo A.
- Physical Review B, Vol. 91, Issue 1
Real-time time-dependent density functional theory approach for frequency-dependent nonlinear optical response in photonic molecules
journal, October 2007
- Takimoto, Y.; Vila, F. D.; Rehr, J. J.
- The Journal of Chemical Physics, Vol. 127, Issue 15
Molecular dynamics in electronically excited states using time-dependent density functional theory
journal, March 2005
- Tavernelli *, Ivano; Röhrig, Ute F.; Rothlisberger, Ursula
- Molecular Physics, Vol. 103, Issue 6-8
Resonance Raman spectra of ortho -nitrophenol calculated by real-time time-dependent density functional theory
journal, January 2013
- Thomas, Martin; Latorre, Federico; Marquetand, Philipp
- The Journal of Chemical Physics, Vol. 138, Issue 4
Efficient pseudopotentials for plane-wave calculations
journal, January 1991
- Troullier, N.; Martins, José Luriaas
- Physical Review B, Vol. 43, Issue 3
Integrating the Car–Parrinello equations. I. Basic integration techniques
journal, July 1994
- Tuckerman, Mark E.; Parrinello, Michele
- The Journal of Chemical Physics, Vol. 101, Issue 2
Towards a gauge invariant method for molecular chiroptical properties in TDDFT
journal, January 2009
- Varsano, Daniele; Espinosa-Leal, Leonardo A.; Andrade, Xavier
- Physical Chemistry Chemical Physics, Vol. 11, Issue 22
Time-dependent local-density approximation in real time
journal, August 1996
- Yabana, K.; Bertsch, G. F.
- Physical Review B, Vol. 54, Issue 7
Works referencing / citing this record:
Ab initio simulations of liquid electrolytes for energy conversion and storage
journal, October 2018
- Pham, Tuan Anh
- International Journal of Quantum Chemistry, Vol. 119, Issue 1
Heavy ion ranges from first-principles electron dynamics
journal, April 2019
- Sand, Andrea E.; Ullah, Rafi; Correa, Alfredo A.
- npj Computational Materials, Vol. 5, Issue 1
Propagation of maximally localized Wannier functions in real-time TDDFT
journal, May 2019
- Yost, Dillon C.; Yao, Yi; Kanai, Yosuke
- The Journal of Chemical Physics, Vol. 150, Issue 19
Exponential integrators in time-dependent density-functional calculations
journal, December 2017
- Kidd, Daniel; Covington, Cody; Varga, Kálmán
- Physical Review E, Vol. 96, Issue 6
Core Electrons in the Electronic Stopping of Heavy Ions
journal, September 2018
- Ullah, Rafi; Artacho, Emilio; Correa, Alfredo A.
- Physical Review Letters, Vol. 121, Issue 11
Electronic stopping and proton dynamics in InP, GaP, and In0.5Ga0.5P from first principles
journal, October 2018
- Lee, Cheng-Wei; Schleife, André
- The European Physical Journal B, Vol. 91, Issue 10
Exponential Integrators in Time-Dependent Density Functional Calculations
text, January 2017
- Kidd, Daniel; Covington, Cody; Varga, Kalman
- arXiv
Electronic stopping and proton dynamics in InP, GaP, and In$_{0.5}$Ga$_{0.5}$P from first principles
text, January 2018
- Lee, Cheng-Wei; Schleife, André
- arXiv
Propagation of maximally localized Wannier functions in real-time TDDFT
text, January 2019
- Yost, Dillon C.; Yao, Yi; Kanai, Yosuke
- arXiv