TeraChem Cloud: A High-Performance Computing Service for Scalable Distributed GPU-Accelerated Electronic Structure Calculations

Seritan, Stefan; Thompson, Keiran; Martínez, Todd J.

doi:10.1021/acs.jcim.9b01152

Title: TeraChem Cloud: A High-Performance Computing Service for Scalable Distributed GPU-Accelerated Electronic Structure Calculations

Journal Article · Wed Apr 08 00:00:00 EDT 2020 · Journal of Chemical Information and Modeling

DOI:https://doi.org/10.1021/acs.jcim.9b01152· OSTI ID:1633326

^[1];

^[1]

Stanford Univ., CA (United States); SLAC National Accelerator Lab., Menlo Park, CA (United States). Photon Ultrafast Laser Science and Engineering Inst. (PULSE)

The encapsulation and commoditization of electronic structure arise naturally as interoperability, and the use of nontraditional compute resources (e.g., new hardware accelerators, cloud computing) remains important for the computational chemistry community. Here, we present TERACHEM CLOUD, a high-performance computing service (HPCS) that offers on-demand electronic structure calculations on both traditional HPC clusters and cloud-based hardware. The framework is designed using off-the-shelf web technologies and containerization to be extremely scalable and portable. Within the HPCS model, users can quickly develop new methods and algorithms in an interactive environment on their laptop while allowing TERACHEM CLOUD to distribute ab initio calculations across all available resources. This approach greatly increases the accessibility of hardware accelerators such as graphics processing units (GPUs) and flexibility for the development of new methods as additional electronic structure packages are integrated into the framework as alternative backends. Cost-performance analysis indicates that traditional nodes are the most cost-effective long-term solution, but commercial cloud providers offer cutting-edge hardware with competitive rates for short-term large-scale calculations. We demonstrate the power of the TERACHEM CLOUD framework by carrying out several showcase calculations, including the generation of 300,000 density functional theory energy and gradient evaluations on medium-sized organic molecules and reproducing 300 fs of nonadiabatic dynamics on the B800-B850 antenna complex in LH2, with the latter demonstration using over 50 Tesla V100 GPUs in a commercial cloud environment in 8 h for approximately $1250.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States)

Sponsoring Organization:: USDOE; National Science Foundation (NSF); US Department of the Navy, Office of Naval Research (ONR)

Grant/Contract Number:: AC02-76SF00515; N00014-16-1-2557; ACI-1450179

OSTI ID:: 1633326

Journal Information:: Journal of Chemical Information and Modeling, Vol. 60, Issue 4; ISSN 1549-9596

Publisher:: American Chemical SocietyCopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 18 works

Citation information provided by
Web of Science

References (61)

An Ab Initio Exciton Model Including Charge-Transfer Excited States Li, Xin; Parrish, Robert M.; Liu, Fang Journal of Chemical Theory and Computation, Vol. 13, Issue 8 https://doi.org/10.1021/acs.jctc.7b00171	journal	July 2017
970 Million Druglike Small Molecules for Virtual Screening in the Chemical Universe Database GDB-13 Blum, Lorenz C.; Reymond, Jean-Louis Journal of the American Chemical Society, Vol. 131, Issue 25 https://doi.org/10.1021/ja902302h	journal	July 2009
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2018 update Afgan, Enis; Baker, Dannon; Batut, Bérénice Nucleic Acids Research, Vol. 46, Issue W1 https://doi.org/10.1093/nar/gky379	journal	May 2018
Ab Initio Nonadiabatic Quantum Molecular Dynamics Curchod, Basile F. E.; Martínez, Todd J. Chemical Reviews, Vol. 118, Issue 7 https://doi.org/10.1021/acs.chemrev.7b00423	journal	February 2018
Molpro: a general-purpose quantum chemistry program package: Molpro Werner, Hans-Joachim; Knowles, Peter J.; Knizia, Gerald Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 2, Issue 2 https://doi.org/10.1002/wcms.82	journal	July 2011
Quantum Chemistry on Graphical Processing Units. 1. Strategies for Two-Electron Integral Evaluation Ufimtsev, Ivan S.; Martínez, Todd J. Journal of Chemical Theory and Computation, Vol. 4, Issue 2 https://doi.org/10.1021/ct700268q	journal	January 2008
Scaling Predictive Modeling in Drug Development with Cloud Computing Moghadam, Behrooz Torabi; Alvarsson, Jonathan; Holm, Marcus Journal of Chemical Information and Modeling, Vol. 55, Issue 1 https://doi.org/10.1021/ci500580y	journal	December 2014
Automated Discovery and Refinement of Reactive Molecular Dynamics Pathways Wang, Lee-Ping; McGibbon, Robert T.; Pande, Vijay S. Journal of Chemical Theory and Computation, Vol. 12, Issue 2 https://doi.org/10.1021/acs.jctc.5b00830	journal	January 2016
Singularity: Scientific containers for mobility of compute Kurtzer, Gregory M.; Sochat, Vanessa; Bauer, Michael W. PLOS ONE, Vol. 12, Issue 5 https://doi.org/10.1371/journal.pone.0177459	journal	May 2017
Cloud Parallel Processing of Tandem Mass Spectrometry Based Proteomics Data Mohammed, Yassene; Mostovenko, Ekaterina; Henneman, Alex A. Journal of Proteome Research, Vol. 11, Issue 10 https://doi.org/10.1021/pr300561q	journal	September 2012
Fast Parallel Algorithms for Short-Range Molecular Dynamics Plimpton, Steve Journal of Computational Physics, Vol. 117, Issue 1 https://doi.org/10.1006/jcph.1995.1039	journal	March 1995
Quantum and thermal effects in $H_{2}$ dissociative adsorption: Evaluation of free energy barriers in multidimensional quantum systems Mills, Greg; Jónsson, Hannes Physical Review Letters, Vol. 72, Issue 7 https://doi.org/10.1103/PhysRevLett.72.1124	journal	February 1994
Atomistic non-adiabatic dynamics of the LH2 complex with a GPU-accelerated ab initio exciton model Sisto, Aaron; Stross, Clem; van der Kamp, Marc W. Physical Chemistry Chemical Physics, Vol. 19, Issue 23 https://doi.org/10.1039/C7CP00492C	journal	January 2017
GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers Abraham, Mark James; Murtola, Teemu; Schulz, Roland SoftwareX, Vol. 1-2 https://doi.org/10.1016/j.softx.2015.06.001	journal	September 2015
Automating NEURON Simulation Deployment in Cloud Resources Stockton, David B.; Santamaria, Fidel Neuroinformatics, Vol. 15, Issue 1 https://doi.org/10.1007/s12021-016-9315-8	journal	September 2016
Charge Transfer and Polarization in Solvated Proteins from Ab Initio Molecular Dynamics Ufimtsev, Ivan S.; Luehr, Nathan; Martinez, Todd J. The Journal of Physical Chemistry Letters, Vol. 2, Issue 14 https://doi.org/10.1021/jz200697c	journal	June 2011
Ab Initio Quantum Chemistry for Protein Structures Kulik, Heather J.; Luehr, Nathan; Ufimtsev, Ivan S. The Journal of Physical Chemistry B, Vol. 116, Issue 41 https://doi.org/10.1021/jp307741u	journal	October 2012
Reversible multiple time scale molecular dynamics Tuckerman, M.; Berne, B. J.; Martyna, G. J. The Journal of Chemical Physics, Vol. 97, Issue 3 https://doi.org/10.1063/1.463137	journal	August 1992
Optimal cloud use of quartic force fields: The first purely commercial cloud computing based study for rovibrational analysis of SiCH ⁻ Fortenberry, Ryan C.; Thackston, Russell International Journal of Quantum Chemistry, Vol. 115, Issue 23 https://doi.org/10.1002/qua.24997	journal	August 2015
Nuclear Data for Astrophysics Research: A New Online Paradigm Smith, Michael S. Journal of the Korean Physical Society, Vol. 59, Issue 2(3) https://doi.org/10.3938/jkps.59.761	journal	August 2011
A climbing image nudged elastic band method for finding saddle points and minimum energy paths Henkelman, Graeme; Uberuaga, Blas P.; Jónsson, Hannes The Journal of Chemical Physics, Vol. 113, Issue 22, p. 9901-9904 https://doi.org/10.1063/1.1329672	journal	December 2000
Ab Initio Nonadiabatic Dynamics of Multichromophore Complexes: A Scalable Graphical-Processing-Unit-Accelerated Exciton Framework Sisto, Aaron; Glowacki, David R.; Martinez, Todd J. Accounts of Chemical Research, Vol. 47, Issue 9 https://doi.org/10.1021/ar500229p	journal	July 2014
Accurate sampling using Langevin dynamics Bussi, Giovanni; Parrinello, Michele Physical Review E, Vol. 75, Issue 5 https://doi.org/10.1103/PhysRevE.75.056707	journal	May 2007
Discovering chemistry with an ab initio nanoreactor Wang, Lee-Ping; Titov, Alexey; McGibbon, Robert Nature Chemistry, Vol. 6, Issue 12 https://doi.org/10.1038/nchem.2099	journal	November 2014
Borg, Omega, and Kubernetes: Lessons learned from three container-management systems over a decade Burns, Brendan; Grant, Brian; Oppenheimer, David Queue, Vol. 14, Issue 1 https://doi.org/10.1145/2898442.2898444	journal	January 2016
Understanding the Performance and Potential of Cloud Computing for Scientific Applications Sadooghi, Iman; Martin, Jesus Hernandez; Li, Tonglin IEEE Transactions on Cloud Computing, Vol. 5, Issue 2 https://doi.org/10.1109/TCC.2015.2404821	journal	April 2017
BAGEL: Brilliantly Advanced General Electronic-structure Library: BAGEL Shiozaki, Toru Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 8, Issue 1 https://doi.org/10.1002/wcms.1331	journal	August 2017
Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility Buyya, Rajkumar; Yeo, Chee Shin; Venugopal, Srikumar Future Generation Computer Systems, Vol. 25, Issue 6 https://doi.org/10.1016/j.future.2008.12.001	journal	June 2009
Multiple time step integrators in ab initio molecular dynamics Luehr, Nathan; Markland, Thomas E.; Martínez, Todd J. The Journal of Chemical Physics, Vol. 140, Issue 8 https://doi.org/10.1063/1.4866176	journal	February 2014
Fast and Accurate Modeling of Molecular Atomization Energies with Machine Learning Rupp, Matthias; Tkatchenko, Alexandre; Müller, Klaus-Robert Physical Review Letters, Vol. 108, Issue 5 https://doi.org/10.1103/PhysRevLett.108.058301	journal	January 2012
Crystal structure of an integral membrane light-harvesting complex from photosynthetic bacteria McDermott, G.; Prince, S. M.; Freer, A. A. Nature, Vol. 374, Issue 6522 https://doi.org/10.1038/374517a0	journal	April 1995
Studying chemical reactivity in a virtual environment Haag, Moritz P.; Reiher, Markus Faraday Discuss., Vol. 169 https://doi.org/10.1039/C4FD00021H	journal	January 2014
Interactive Chemical Reactivity Exploration Haag, Moritz P.; Vaucher, Alain C.; Bosson, Maël ChemPhysChem, Vol. 15, Issue 15 https://doi.org/10.1002/cphc.201402342	journal	September 2014
Quantum Chemistry on Graphical Processing Units. 3. Analytical Energy Gradients, Geometry Optimization, and First Principles Molecular Dynamics Ufimtsev, Ivan S.; Martinez, Todd J. Journal of Chemical Theory and Computation, Vol. 5, Issue 10 https://doi.org/10.1021/ct9003004	journal	August 2009
NeuroManager: a workflow analysis based simulation management engine for computational neuroscience Stockton, David B.; Santamaria, Fidel Frontiers in Neuroinformatics, Vol. 9 https://doi.org/10.3389/fninf.2015.00024	journal	October 2015
Ab Initio Interactive Molecular Dynamics on Graphical Processing Units (GPUs) Luehr, Nathan; Jin, Alex G. B.; Martínez, Todd J. Journal of Chemical Theory and Computation, Vol. 11, Issue 10 https://doi.org/10.1021/acs.jctc.5b00419	journal	September 2015
The Structure and Thermal Motion of the B800–850 LH2 Complex from Rps.acidophila at 2.0Å Resolution and 100K: New Structural Features and Functionally Relevant Motions Papiz, Miroslav Z.; Prince, Steve M.; Howard, Tina Journal of Molecular Biology, Vol. 326, Issue 5 https://doi.org/10.1016/S0022-2836(03)00024-X	journal	March 2003
Enumeration of 166 Billion Organic Small Molecules in the Chemical Universe Database GDB-17 Ruddigkeit, Lars; van Deursen, Ruud; Blum, Lorenz C. Journal of Chemical Information and Modeling, Vol. 52, Issue 11 https://doi.org/10.1021/ci300415d	journal	November 2012
Multiple-Timestep ab Initio Molecular Dynamics Using an Atomic Basis Set Partitioning Steele, Ryan P. The Journal of Physical Chemistry A, Vol. 119, Issue 50 https://doi.org/10.1021/acs.jpca.5b05850	journal	September 2015
Ab Initio Molecular Dynamics and Time-Resolved Photoelectron Spectroscopy of Electronically Excited Uracil and Thymine Hudock, Hanneli R.; Levine, Benjamin G.; Thompson, Alexis L. The Journal of Physical Chemistry A, Vol. 111, Issue 34 https://doi.org/10.1021/jp0723665	journal	August 2007
Cloud computing for comparative genomics Wall, Dennis P.; Kudtarkar, Parul; Fusaro, Vincent A. BMC Bioinformatics, Vol. 11, Issue 1 https://doi.org/10.1186/1471-2105-11-259	journal	May 2010
Biomedical Cloud Computing With Amazon Web Services Fusaro, Vincent A.; Patil, Prasad; Gafni, Erik PLoS Computational Biology, Vol. 7, Issue 8 https://doi.org/10.1371/journal.pcbi.1002147	journal	August 2011
The performance of low-cost commercial cloud computing as an alternative in computational chemistry Thackston, Russell; Fortenberry, Ryan C. Journal of Computational Chemistry, Vol. 36, Issue 12 https://doi.org/10.1002/jcc.23882	journal	March 2015
Molecular Rift: Virtual Reality for Drug Designers Norrby, Magnus; Grebner, Christoph; Eriksson, Joakim Journal of Chemical Information and Modeling, Vol. 55, Issue 11 https://doi.org/10.1021/acs.jcim.5b00544	journal	November 2015
Quantum-chemical insights from deep tensor neural networks Schütt, Kristof T.; Arbabzadah, Farhad; Chmiela, Stefan Nature Communications, Vol. 8, Issue 1 https://doi.org/10.1038/ncomms13890	journal	January 2017
Sampling molecular conformations and dynamics in a multiuser virtual reality framework O’Connor, Michael; Deeks, Helen M.; Dawn, Edward Science Advances, Vol. 4, Issue 6 https://doi.org/10.1126/sciadv.aat2731	journal	June 2018
Advances in electronic structure theory Gordon, Mark S.; Schmidt, Michael W. Theory and Applications of Computational Chemistry https://doi.org/10.1016/B978-044451719-7/50084-6	book	January 2005
Harnessing cloud computing with Galaxy Cloud Afgan, Enis; Baker, Dannon; Coraor, Nate Nature Biotechnology, Vol. 29, Issue 11 https://doi.org/10.1038/nbt.2028	journal	November 2011
Quantum Chemistry on Graphical Processing Units. 2. Direct Self-Consistent-Field Implementation Ufimtsev, Ivan S.; Martinez, Todd J. Journal of Chemical Theory and Computation, Vol. 5, Issue 4 https://doi.org/10.1021/ct800526s	journal	March 2009
Improved tangent estimate in the nudged elastic band method for finding minimum energy paths and saddle points Henkelman, Graeme; Jónsson, Hannes The Journal of Chemical Physics, Vol. 113, Issue 22 https://doi.org/10.1063/1.1323224	journal	December 2000
Low Cost, Scalable Proteomics Data Analysis Using Amazon’s Cloud Computing Services and Open Source Search Algorithms Halligan, Brian D.; Geiger, Joey F.; Vallejos, Andrew K. Journal of Proteome Research, Vol. 8, Issue 6 https://doi.org/10.1021/pr800970z	journal	June 2009
Quantum chemistry structures and properties of 134 kilo molecules Ramakrishnan, Raghunathan; Dral, Pavlo O.; Rupp, Matthias Scientific Data, Vol. 1, Issue 1 https://doi.org/10.1038/sdata.2014.22	journal	August 2014
A unified framework for the deployment, exposure and access of HPC applications as services in clouds Wong, Adam K. L.; Goscinski, Andrzej M. Future Generation Computer Systems, Vol. 29, Issue 6 https://doi.org/10.1016/j.future.2013.01.014	journal	August 2013
Enabling High-Performance Computing as a Service AbdelBaky, Moustafa; Parashar, Manish; Kim, Hyunjoo Computer, Vol. 45, Issue 10 https://doi.org/10.1109/MC.2012.293	journal	October 2012
MOLCAS-a software for multiconfigurational quantum chemistry calculations: MOLCAS Aquilante, Francesco; Pedersen, Thomas Bondo; Veryazov, Valera Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 3, Issue 2 https://doi.org/10.1002/wcms.1117	journal	September 2012
The Design, Deployment, and Evaluation of the CORAL Pre-Exascale Systems Vazhkudai, Sudharshan S.; de Supinski, Bronis R.; Bland, Arthur S. SC18: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2018.00055	conference	November 2018
LHC Cloud Computing with CernVM Segal, Ben; Buncic, Predrag; Aguado Sanchez, Carlos Proceedings of 13th International Workshop on Advanced Computing and Analysis Techniques in Physics Research — PoS(ACAT2010) https://doi.org/10.22323/1.093.0004	conference	February 2011
The Evolution of Cloud Computing in ATLAS Taylor, Ryan P.; Berghaus, Frank; Brasolin, Franco Journal of Physics: Conference Series, Vol. 664, Issue 2 https://doi.org/10.1088/1742-6596/664/2/022038	journal	December 2015
Evaluating GPU Passthrough in Xen for High Performance Cloud Computing Younge, Andrew J.; Walters, John Paul; Crago, Stephen 2014 IEEE International Parallel & Distributed Processing Symposium Workshops (IPDPSW) https://doi.org/10.1109/IPDPSW.2014.97	conference	May 2014
A comparative study of high-performance computing on the cloud Marathe, Aniruddha; Harris, Rachel; Lowenthal, David K. HPDC'13: The 22nd International Symposium on High-Performance Parallel and Distributed Computing, Proceedings of the 22nd international symposium on High-performance parallel and distributed computing https://doi.org/10.1145/2462902.2462919	conference	October 2018
A system for interactive molecular dynamics simulation Stone, John E.; Gullingsrud, Justin; Schulten, Klaus Proceedings of the 2001 symposium on Interactive 3D graphics - SI3D '01 https://doi.org/10.1145/364338.364398	conference	January 2001

Similar Records

Atomistic non-adiabatic dynamics of the LH2 complex with a GPU-accelerated ab initio exciton model

Journal Article · Tue Mar 28 00:00:00 EDT 2017 · Physical Chemistry Chemical Physics. PCCP · OSTI ID:1633326

Sisto, Aaron; Stross, Clem; van der Kamp, Marc W.; +7 more

TeraChem: A graphical processing unit-accelerated electronic structure package for large-scale ab initio molecular dynamics

Journal Article · Sun Jul 26 00:00:00 EDT 2020 · Wiley Interdisciplinary Reviews: Computational Molecular Science · OSTI ID:1633326

Seritan, Stefan; Bannwarth, Christoph; Fales, Bryan S.; +12 more

TeraChem protocol buffers (TCPB): Accelerating QM and QM/MM simulations with a client–server model

Journal Article · Wed Jan 25 00:00:00 EST 2023 · Journal of Chemical Physics · OSTI ID:1633326

Cruzeiro, Vinícius D.; Wang, Yuanheng; Pieri, Elisa; +2 more

Related Subjects

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CHEMISTRY
Ab initio calculations
Chemical calculations
Electronic structure
Cloud computing
Cluster chemistry

Title: TeraChem Cloud: A High-Performance Computing Service for Scalable Distributed GPU-Accelerated Electronic Structure Calculations

Citation Formats

References (61)

Similar Records

Related Subjects