DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: TeraChem Cloud: A High-Performance Computing Service for Scalable Distributed GPU-Accelerated Electronic Structure Calculations

Abstract

The encapsulation and commoditization of electronic structure arise naturally as interoperability, and the use of nontraditional compute resources (e.g., new hardware accelerators, cloud computing) remains important for the computational chemistry community. Here, we present TERACHEM CLOUD, a high-performance computing service (HPCS) that offers on-demand electronic structure calculations on both traditional HPC clusters and cloud-based hardware. The framework is designed using off-the-shelf web technologies and containerization to be extremely scalable and portable. Within the HPCS model, users can quickly develop new methods and algorithms in an interactive environment on their laptop while allowing TERACHEM CLOUD to distribute ab initio calculations across all available resources. This approach greatly increases the accessibility of hardware accelerators such as graphics processing units (GPUs) and flexibility for the development of new methods as additional electronic structure packages are integrated into the framework as alternative backends. Cost-performance analysis indicates that traditional nodes are the most cost-effective long-term solution, but commercial cloud providers offer cutting-edge hardware with competitive rates for short-term large-scale calculations. We demonstrate the power of the TERACHEM CLOUD framework by carrying out several showcase calculations, including the generation of 300,000 density functional theory energy and gradient evaluations on medium-sized organic molecules and reproducing 300 fsmore » of nonadiabatic dynamics on the B800-B850 antenna complex in LH2, with the latter demonstration using over 50 Tesla V100 GPUs in a commercial cloud environment in 8 h for approximately $1250.« less

Authors:
ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1]
  1. Stanford Univ., CA (United States); SLAC National Accelerator Lab., Menlo Park, CA (United States). Photon Ultrafast Laser Science and Engineering Inst. (PULSE)
Publication Date:
Research Org.:
SLAC National Accelerator Lab., Menlo Park, CA (United States)
Sponsoring Org.:
USDOE; National Science Foundation (NSF); US Department of the Navy, Office of Naval Research (ONR)
OSTI Identifier:
1633326
Grant/Contract Number:  
AC02-76SF00515; N00014-16-1-2557; ACI-1450179
Resource Type:
Accepted Manuscript
Journal Name:
Journal of Chemical Information and Modeling
Additional Journal Information:
Journal Volume: 60; Journal Issue: 4; Journal ID: ISSN 1549-9596
Publisher:
American Chemical Society
Country of Publication:
United States
Language:
English
Subject:
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CHEMISTRY; Ab initio calculations; Chemical calculations; Electronic structure; Cloud computing; Cluster chemistry

Citation Formats

Seritan, Stefan, Thompson, Keiran, and Martínez, Todd J. TeraChem Cloud: A High-Performance Computing Service for Scalable Distributed GPU-Accelerated Electronic Structure Calculations. United States: N. p., 2020. Web. doi:10.1021/acs.jcim.9b01152.
Seritan, Stefan, Thompson, Keiran, & Martínez, Todd J. TeraChem Cloud: A High-Performance Computing Service for Scalable Distributed GPU-Accelerated Electronic Structure Calculations. United States. https://doi.org/10.1021/acs.jcim.9b01152
Seritan, Stefan, Thompson, Keiran, and Martínez, Todd J. Wed . "TeraChem Cloud: A High-Performance Computing Service for Scalable Distributed GPU-Accelerated Electronic Structure Calculations". United States. https://doi.org/10.1021/acs.jcim.9b01152. https://www.osti.gov/servlets/purl/1633326.
@article{osti_1633326,
title = {TeraChem Cloud: A High-Performance Computing Service for Scalable Distributed GPU-Accelerated Electronic Structure Calculations},
author = {Seritan, Stefan and Thompson, Keiran and Martínez, Todd J.},
abstractNote = {The encapsulation and commoditization of electronic structure arise naturally as interoperability, and the use of nontraditional compute resources (e.g., new hardware accelerators, cloud computing) remains important for the computational chemistry community. Here, we present TERACHEM CLOUD, a high-performance computing service (HPCS) that offers on-demand electronic structure calculations on both traditional HPC clusters and cloud-based hardware. The framework is designed using off-the-shelf web technologies and containerization to be extremely scalable and portable. Within the HPCS model, users can quickly develop new methods and algorithms in an interactive environment on their laptop while allowing TERACHEM CLOUD to distribute ab initio calculations across all available resources. This approach greatly increases the accessibility of hardware accelerators such as graphics processing units (GPUs) and flexibility for the development of new methods as additional electronic structure packages are integrated into the framework as alternative backends. Cost-performance analysis indicates that traditional nodes are the most cost-effective long-term solution, but commercial cloud providers offer cutting-edge hardware with competitive rates for short-term large-scale calculations. We demonstrate the power of the TERACHEM CLOUD framework by carrying out several showcase calculations, including the generation of 300,000 density functional theory energy and gradient evaluations on medium-sized organic molecules and reproducing 300 fs of nonadiabatic dynamics on the B800-B850 antenna complex in LH2, with the latter demonstration using over 50 Tesla V100 GPUs in a commercial cloud environment in 8 h for approximately $1250.},
doi = {10.1021/acs.jcim.9b01152},
journal = {Journal of Chemical Information and Modeling},
number = 4,
volume = 60,
place = {United States},
year = {2020},
month = {4}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 2 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

An Ab Initio Exciton Model Including Charge-Transfer Excited States
journal, July 2017

  • Li, Xin; Parrish, Robert M.; Liu, Fang
  • Journal of Chemical Theory and Computation, Vol. 13, Issue 8
  • DOI: 10.1021/acs.jctc.7b00171

970 Million Druglike Small Molecules for Virtual Screening in the Chemical Universe Database GDB-13
journal, July 2009

  • Blum, Lorenz C.; Reymond, Jean-Louis
  • Journal of the American Chemical Society, Vol. 131, Issue 25
  • DOI: 10.1021/ja902302h

The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2018 update
journal, May 2018

  • Afgan, Enis; Baker, Dannon; Batut, Bérénice
  • Nucleic Acids Research, Vol. 46, Issue W1
  • DOI: 10.1093/nar/gky379

Ab Initio Nonadiabatic Quantum Molecular Dynamics
journal, February 2018


Molpro: a general-purpose quantum chemistry program package: Molpro
journal, July 2011

  • Werner, Hans-Joachim; Knowles, Peter J.; Knizia, Gerald
  • Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 2, Issue 2
  • DOI: 10.1002/wcms.82

Quantum Chemistry on Graphical Processing Units. 1. Strategies for Two-Electron Integral Evaluation
journal, January 2008

  • Ufimtsev, Ivan S.; Martínez, Todd J.
  • Journal of Chemical Theory and Computation, Vol. 4, Issue 2
  • DOI: 10.1021/ct700268q

Scaling Predictive Modeling in Drug Development with Cloud Computing
journal, December 2014

  • Moghadam, Behrooz Torabi; Alvarsson, Jonathan; Holm, Marcus
  • Journal of Chemical Information and Modeling, Vol. 55, Issue 1
  • DOI: 10.1021/ci500580y

Automated Discovery and Refinement of Reactive Molecular Dynamics Pathways
journal, January 2016

  • Wang, Lee-Ping; McGibbon, Robert T.; Pande, Vijay S.
  • Journal of Chemical Theory and Computation, Vol. 12, Issue 2
  • DOI: 10.1021/acs.jctc.5b00830

Singularity: Scientific containers for mobility of compute
journal, May 2017


Cloud Parallel Processing of Tandem Mass Spectrometry Based Proteomics Data
journal, September 2012

  • Mohammed, Yassene; Mostovenko, Ekaterina; Henneman, Alex A.
  • Journal of Proteome Research, Vol. 11, Issue 10
  • DOI: 10.1021/pr300561q

Fast Parallel Algorithms for Short-Range Molecular Dynamics
journal, March 1995


Atomistic non-adiabatic dynamics of the LH2 complex with a GPU-accelerated ab initio exciton model
journal, January 2017

  • Sisto, Aaron; Stross, Clem; van der Kamp, Marc W.
  • Physical Chemistry Chemical Physics, Vol. 19, Issue 23
  • DOI: 10.1039/C7CP00492C

GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers
journal, September 2015


Automating NEURON Simulation Deployment in Cloud Resources
journal, September 2016


Charge Transfer and Polarization in Solvated Proteins from Ab Initio Molecular Dynamics
journal, June 2011

  • Ufimtsev, Ivan S.; Luehr, Nathan; Martinez, Todd J.
  • The Journal of Physical Chemistry Letters, Vol. 2, Issue 14
  • DOI: 10.1021/jz200697c

Ab Initio Quantum Chemistry for Protein Structures
journal, October 2012

  • Kulik, Heather J.; Luehr, Nathan; Ufimtsev, Ivan S.
  • The Journal of Physical Chemistry B, Vol. 116, Issue 41
  • DOI: 10.1021/jp307741u

Reversible multiple time scale molecular dynamics
journal, August 1992

  • Tuckerman, M.; Berne, B. J.; Martyna, G. J.
  • The Journal of Chemical Physics, Vol. 97, Issue 3
  • DOI: 10.1063/1.463137

Cost-Effective Cloud Computing: A Case Study Using the Comparative Genomics Tool, Roundup
journal, January 2010

  • Kudtarkar, Parul; DeLuca, Todd F.; Fusaro, Vincent A.
  • Evolutionary Bioinformatics, Vol. 6
  • DOI: 10.4137/EBO.S6259

Optimal cloud use of quartic force fields: The first purely commercial cloud computing based study for rovibrational analysis of SiCH
journal, August 2015

  • Fortenberry, Ryan C.; Thackston, Russell
  • International Journal of Quantum Chemistry, Vol. 115, Issue 23
  • DOI: 10.1002/qua.24997

Nuclear Data for Astrophysics Research: A New Online Paradigm
journal, August 2011

  • Smith, Michael S.
  • Journal of the Korean Physical Society, Vol. 59, Issue 2(3)
  • DOI: 10.3938/jkps.59.761

A climbing image nudged elastic band method for finding saddle points and minimum energy paths
journal, December 2000

  • Henkelman, Graeme; Uberuaga, Blas P.; Jónsson, Hannes
  • The Journal of Chemical Physics, Vol. 113, Issue 22, p. 9901-9904
  • DOI: 10.1063/1.1329672

Ab Initio Nonadiabatic Dynamics of Multichromophore Complexes: A Scalable Graphical-Processing-Unit-Accelerated Exciton Framework
journal, July 2014

  • Sisto, Aaron; Glowacki, David R.; Martinez, Todd J.
  • Accounts of Chemical Research, Vol. 47, Issue 9
  • DOI: 10.1021/ar500229p

Accurate sampling using Langevin dynamics
journal, May 2007


Discovering chemistry with an ab initio nanoreactor
journal, November 2014

  • Wang, Lee-Ping; Titov, Alexey; McGibbon, Robert
  • Nature Chemistry, Vol. 6, Issue 12
  • DOI: 10.1038/nchem.2099

Borg, Omega, and Kubernetes: Lessons learned from three container-management systems over a decade
journal, January 2016


Understanding the Performance and Potential of Cloud Computing for Scientific Applications
journal, April 2017

  • Sadooghi, Iman; Martin, Jesus Hernandez; Li, Tonglin
  • IEEE Transactions on Cloud Computing, Vol. 5, Issue 2
  • DOI: 10.1109/TCC.2015.2404821

BAGEL: Brilliantly Advanced General Electronic-structure Library: BAGEL
journal, August 2017

  • Shiozaki, Toru
  • Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 8, Issue 1
  • DOI: 10.1002/wcms.1331

String method for the study of rare events
journal, August 2002


Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility
journal, June 2009

  • Buyya, Rajkumar; Yeo, Chee Shin; Venugopal, Srikumar
  • Future Generation Computer Systems, Vol. 25, Issue 6
  • DOI: 10.1016/j.future.2008.12.001

Multiple time step integrators in ab initio molecular dynamics
journal, February 2014

  • Luehr, Nathan; Markland, Thomas E.; Martínez, Todd J.
  • The Journal of Chemical Physics, Vol. 140, Issue 8
  • DOI: 10.1063/1.4866176

Fast and Accurate Modeling of Molecular Atomization Energies with Machine Learning
journal, January 2012


Crystal structure of an integral membrane light-harvesting complex from photosynthetic bacteria
journal, April 1995

  • McDermott, G.; Prince, S. M.; Freer, A. A.
  • Nature, Vol. 374, Issue 6522
  • DOI: 10.1038/374517a0

Studying chemical reactivity in a virtual environment
journal, January 2014


Interactive Chemical Reactivity Exploration
journal, September 2014

  • Haag, Moritz P.; Vaucher, Alain C.; Bosson, Maël
  • ChemPhysChem, Vol. 15, Issue 15
  • DOI: 10.1002/cphc.201402342

Quantum Chemistry on Graphical Processing Units. 3. Analytical Energy Gradients, Geometry Optimization, and First Principles Molecular Dynamics
journal, August 2009

  • Ufimtsev, Ivan S.; Martinez, Todd J.
  • Journal of Chemical Theory and Computation, Vol. 5, Issue 10
  • DOI: 10.1021/ct9003004

NeuroManager: a workflow analysis based simulation management engine for computational neuroscience
journal, October 2015


Ab Initio Interactive Molecular Dynamics on Graphical Processing Units (GPUs)
journal, September 2015

  • Luehr, Nathan; Jin, Alex G. B.; Martínez, Todd J.
  • Journal of Chemical Theory and Computation, Vol. 11, Issue 10
  • DOI: 10.1021/acs.jctc.5b00419

Enumeration of 166 Billion Organic Small Molecules in the Chemical Universe Database GDB-17
journal, November 2012

  • Ruddigkeit, Lars; van Deursen, Ruud; Blum, Lorenz C.
  • Journal of Chemical Information and Modeling, Vol. 52, Issue 11
  • DOI: 10.1021/ci300415d

Multiple-Timestep ab Initio Molecular Dynamics Using an Atomic Basis Set Partitioning
journal, September 2015


Ab Initio Molecular Dynamics and Time-Resolved Photoelectron Spectroscopy of Electronically Excited Uracil and Thymine
journal, August 2007

  • Hudock, Hanneli R.; Levine, Benjamin G.; Thompson, Alexis L.
  • The Journal of Physical Chemistry A, Vol. 111, Issue 34
  • DOI: 10.1021/jp0723665

Cloud computing for comparative genomics
journal, May 2010

  • Wall, Dennis P.; Kudtarkar, Parul; Fusaro, Vincent A.
  • BMC Bioinformatics, Vol. 11, Issue 1
  • DOI: 10.1186/1471-2105-11-259

Biomedical Cloud Computing With Amazon Web Services
journal, August 2011


The performance of low-cost commercial cloud computing as an alternative in computational chemistry
journal, March 2015

  • Thackston, Russell; Fortenberry, Ryan C.
  • Journal of Computational Chemistry, Vol. 36, Issue 12
  • DOI: 10.1002/jcc.23882

Molecular Rift: Virtual Reality for Drug Designers
journal, November 2015

  • Norrby, Magnus; Grebner, Christoph; Eriksson, Joakim
  • Journal of Chemical Information and Modeling, Vol. 55, Issue 11
  • DOI: 10.1021/acs.jcim.5b00544

Quantum-chemical insights from deep tensor neural networks
journal, January 2017

  • Schütt, Kristof T.; Arbabzadah, Farhad; Chmiela, Stefan
  • Nature Communications, Vol. 8, Issue 1
  • DOI: 10.1038/ncomms13890

Sampling molecular conformations and dynamics in a multiuser virtual reality framework
journal, June 2018

  • O’Connor, Michael; Deeks, Helen M.; Dawn, Edward
  • Science Advances, Vol. 4, Issue 6
  • DOI: 10.1126/sciadv.aat2731

Harnessing cloud computing with Galaxy Cloud
journal, November 2011

  • Afgan, Enis; Baker, Dannon; Coraor, Nate
  • Nature Biotechnology, Vol. 29, Issue 11
  • DOI: 10.1038/nbt.2028

Quantum Chemistry on Graphical Processing Units. 2. Direct Self-Consistent-Field Implementation
journal, March 2009

  • Ufimtsev, Ivan S.; Martinez, Todd J.
  • Journal of Chemical Theory and Computation, Vol. 5, Issue 4
  • DOI: 10.1021/ct800526s

Improved tangent estimate in the nudged elastic band method for finding minimum energy paths and saddle points
journal, December 2000

  • Henkelman, Graeme; Jónsson, Hannes
  • The Journal of Chemical Physics, Vol. 113, Issue 22
  • DOI: 10.1063/1.1323224

Low Cost, Scalable Proteomics Data Analysis Using Amazon’s Cloud Computing Services and Open Source Search Algorithms
journal, June 2009

  • Halligan, Brian D.; Geiger, Joey F.; Vallejos, Andrew K.
  • Journal of Proteome Research, Vol. 8, Issue 6
  • DOI: 10.1021/pr800970z

Quantum chemistry structures and properties of 134 kilo molecules
journal, August 2014

  • Ramakrishnan, Raghunathan; Dral, Pavlo O.; Rupp, Matthias
  • Scientific Data, Vol. 1, Issue 1
  • DOI: 10.1038/sdata.2014.22

A unified framework for the deployment, exposure and access of HPC applications as services in clouds
journal, August 2013


Enabling High-Performance Computing as a Service
journal, October 2012

  • AbdelBaky, Moustafa; Parashar, Manish; Kim, Hyunjoo
  • Computer, Vol. 45, Issue 10
  • DOI: 10.1109/MC.2012.293

MOLCAS-a software for multiconfigurational quantum chemistry calculations: MOLCAS
journal, September 2012

  • Aquilante, Francesco; Pedersen, Thomas Bondo; Veryazov, Valera
  • Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 3, Issue 2
  • DOI: 10.1002/wcms.1117

The Design, Deployment, and Evaluation of the CORAL Pre-Exascale Systems
conference, November 2018

  • Vazhkudai, Sudharshan S.; de Supinski, Bronis R.; Bland, Arthur S.
  • SC18: International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2018.00055

LHC Cloud Computing with CernVM
conference, February 2011

  • Segal, Ben; Buncic, Predrag; Aguado Sanchez, Carlos
  • Proceedings of 13th International Workshop on Advanced Computing and Analysis Techniques in Physics Research — PoS(ACAT2010)
  • DOI: 10.22323/1.093.0004

The Evolution of Cloud Computing in ATLAS
journal, December 2015


Evaluating GPU Passthrough in Xen for High Performance Cloud Computing
conference, May 2014

  • Younge, Andrew J.; Walters, John Paul; Crago, Stephen
  • 2014 IEEE International Parallel & Distributed Processing Symposium Workshops (IPDPSW)
  • DOI: 10.1109/IPDPSW.2014.97

A comparative study of high-performance computing on the cloud
conference, October 2018

  • Marathe, Aniruddha; Harris, Rachel; Lowenthal, David K.
  • HPDC'13: The 22nd International Symposium on High-Performance Parallel and Distributed Computing, Proceedings of the 22nd international symposium on High-performance parallel and distributed computing
  • DOI: 10.1145/2462902.2462919

A system for interactive molecular dynamics simulation
conference, January 2001

  • Stone, John E.; Gullingsrud, Justin; Schulten, Klaus
  • Proceedings of the 2001 symposium on Interactive 3D graphics - SI3D '01
  • DOI: 10.1145/364338.364398