Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Scaling molecular dynamics beyond 100,000 processor cores for large-scale biophysical simulations

Abstract

The growing interest in the complexity of biological interactions is continuously driving the need to increase system size in biophysical simulations, requiring not only powerful and advanced hardware but adaptable software that can accommodate a large number of atoms interacting through complex force fields. To address this, we developed and implemented strategies in the GENESIS molecular dynamics package designed for large numbers of processors. Long–range electrostatic interactions were parallelized by minimizing the number of processes involved in communication. Here, a novel algorithm was implemented for nonbonded interactions to increase single instruction multiple data (SIMD) performance, reducing memory usage for ultra large systems. Memory usage for neighbor searches in real–space nonbonded interactions was reduced by approximately 80%, leading to significant speedup. Using experimental data describing physical 3D chromatin interactions, we constructed the first atomistic model of an entire gene locus (GATA4). Taken together, these developments enabled the first billion–atom simulation of an intact biomolecular complex, achieving scaling to 65,000 processes (130,000 processor cores) with 1 ns/day performance.

Authors:
ORCiD logo [1]; ORCiD logo [2]; ORCiD logo [3];  [4]; ORCiD logo [1]; ORCiD logo [3]; ORCiD logo [1]; ORCiD logo [3];  [3]; ORCiD logo [3]; ORCiD logo [3];  [4]; ORCiD logo [1]; ORCiD logo [2]
+ Show Author Affiliations
  1. RIKEN Center for Computational Science, Kobe (Japan)
  2. Los Alamos National Lab. (LANL), Los Alamos, NM (United States); New Mexico Consortium, Los Alamos, NM (United States)
  3. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  4. New York Univ., NY (United States)
Publication Date:
Research Org.:
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE Laboratory Directed Research and Development (LDRD) Program
OSTI Identifier:
1599030
Alternate Identifier(s):
OSTI ID: 1507471
Report Number(s):
LA-UR-18-31413
Journal ID: ISSN 0192-8651
Grant/Contract Number:  
89233218CNA000001; 26119006; R35GM122562
Resource Type:
Accepted Manuscript
Journal Name:
Journal of Computational Chemistry
Additional Journal Information:
Journal Volume: 40; Journal Issue: 21; Journal ID: ISSN 0192-8651
Publisher:
Wiley
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; high performance computing; biomolecular simulation; 3D modeling; GENESIS MD software

Citation Formats

Jung, Jaewoon, Nishima, Wataru, Daniels, Marcus G., Bascom, Gavin, Kobayashi, Chigusa, Adedoyin, Adetokunbo Adelana, Wall, Michael E., Lappala, Anna, Phillips, Dominic, Fischer, William McLean, Tung, Chang‐Shung, Schlick, Tamar, Sugita, Yuji, and Sanbonmatsu, Karissa Yoshiko. Scaling molecular dynamics beyond 100,000 processor cores for large-scale biophysical simulations. United States: N. p., 2019. Web. doi:10.1002/jcc.25840.
Copy to clipboard
Jung, Jaewoon, Nishima, Wataru, Daniels, Marcus G., Bascom, Gavin, Kobayashi, Chigusa, Adedoyin, Adetokunbo Adelana, Wall, Michael E., Lappala, Anna, Phillips, Dominic, Fischer, William McLean, Tung, Chang‐Shung, Schlick, Tamar, Sugita, Yuji, & Sanbonmatsu, Karissa Yoshiko. Scaling molecular dynamics beyond 100,000 processor cores for large-scale biophysical simulations. United States. https://doi.org/10.1002/jcc.25840
Copy to clipboard
Jung, Jaewoon, Nishima, Wataru, Daniels, Marcus G., Bascom, Gavin, Kobayashi, Chigusa, Adedoyin, Adetokunbo Adelana, Wall, Michael E., Lappala, Anna, Phillips, Dominic, Fischer, William McLean, Tung, Chang‐Shung, Schlick, Tamar, Sugita, Yuji, and Sanbonmatsu, Karissa Yoshiko. Wed . "Scaling molecular dynamics beyond 100,000 processor cores for large-scale biophysical simulations". United States. https://doi.org/10.1002/jcc.25840. https://www.osti.gov/servlets/purl/1599030.
Copy to clipboard
@article{osti_1599030,
title = {Scaling molecular dynamics beyond 100,000 processor cores for large-scale biophysical simulations},
author = {Jung, Jaewoon and Nishima, Wataru and Daniels, Marcus G. and Bascom, Gavin and Kobayashi, Chigusa and Adedoyin, Adetokunbo Adelana and Wall, Michael E. and Lappala, Anna and Phillips, Dominic and Fischer, William McLean and Tung, Chang‐Shung and Schlick, Tamar and Sugita, Yuji and Sanbonmatsu, Karissa Yoshiko},
abstractNote = {The growing interest in the complexity of biological interactions is continuously driving the need to increase system size in biophysical simulations, requiring not only powerful and advanced hardware but adaptable software that can accommodate a large number of atoms interacting through complex force fields. To address this, we developed and implemented strategies in the GENESIS molecular dynamics package designed for large numbers of processors. Long–range electrostatic interactions were parallelized by minimizing the number of processes involved in communication. Here, a novel algorithm was implemented for nonbonded interactions to increase single instruction multiple data (SIMD) performance, reducing memory usage for ultra large systems. Memory usage for neighbor searches in real–space nonbonded interactions was reduced by approximately 80%, leading to significant speedup. Using experimental data describing physical 3D chromatin interactions, we constructed the first atomistic model of an entire gene locus (GATA4). Taken together, these developments enabled the first billion–atom simulation of an intact biomolecular complex, achieving scaling to 65,000 processes (130,000 processor cores) with 1 ns/day performance.},
doi = {10.1002/jcc.25840},
journal = {Journal of Computational Chemistry},
number = 21,
volume = 40,
place = {United States},
year = {Wed Apr 17 00:00:00 EDT 2019},
month = {Wed Apr 17 00:00:00 EDT 2019}
}
Copy to clipboard
Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1002/jcc.25840
Copyright Statement
Other availability
Search WorldCat Search WorldCat to find libraries that may hold this journal
Citation Metrics:
Cited by: 64 works
Citation information provided by
Web of Science
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

Works referenced in this record:

Electrostatics of nanosystems: Application to microtubules and the ribosome
journal, August 2001

  • Baker, N. A.; Sept, D.; Joseph, S.
  • Proceedings of the National Academy of Sciences, Vol. 98, Issue 18, p. 10037-10041
  • DOI: 10.1073/pnas.181342398

GENESIS: a hybrid-parallel and multi-scale molecular dynamics simulator with enhanced sampling algorithms for biomolecular and cellular simulations: GENESIS
journal, May 2015

  • Jung, Jaewoon; Mori, Takaharu; Kobayashi, Chigusa
  • Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 5, Issue 4
  • DOI: 10.1002/wcms.1220

A modified TIP3P water potential for simulation with Ewald summation
journal, November 2004

  • Price, Daniel J.; Brooks, Charles L.
  • The Journal of Chemical Physics, Vol. 121, Issue 20
  • DOI: 10.1063/1.1808117

Extension of the AMBER molecular dynamics software to Intel’s Many Integrated Core (MIC) architecture
journal, April 2016

  • Needham, Perri J.; Bhuiyan, Ashraf; Walker, Ross C.
  • Computer Physics Communications, Vol. 201
  • DOI: 10.1016/j.cpc.2015.12.025

Graphics Processing Unit Acceleration and Parallelization of GENESIS for Large-Scale Molecular Dynamics Simulations
journal, September 2016

  • Jung, Jaewoon; Naurse, Akira; Kobayashi, Chigusa
  • Journal of Chemical Theory and Computation, Vol. 12, Issue 10
  • DOI: 10.1021/acs.jctc.6b00241

Mesoscale Modeling Reveals Hierarchical Looping of Chromatin Fibers Near Gene Regulatory Elements
journal, May 2016

  • Bascom, Gavin D.; Sanbonmatsu, Karissa Y.; Schlick, Tamar
  • The Journal of Physical Chemistry B, Vol. 120, Issue 33
  • DOI: 10.1021/acs.jpcb.6b03197

Routine Access to Millisecond Time Scale Events with Accelerated Molecular Dynamics
journal, August 2012

  • Pierce, Levi C. T.; Salomon-Ferrer, Romelia; Augusto F. de Oliveira, Cesar
  • Journal of Chemical Theory and Computation, Vol. 8, Issue 9
  • DOI: 10.1021/ct300284c

Particle mesh Ewald: An N ⋅log( N ) method for Ewald sums in large systems
journal, June 1993

  • Darden, Tom; York, Darrin; Pedersen, Lee
  • The Journal of Chemical Physics, Vol. 98, Issue 12
  • DOI: 10.1063/1.464397

CHARMM36 all-atom additive protein force field: Validation based on comparison to NMR data
journal, July 2013

  • Huang, Jing; MacKerell, Alexander D.
  • Journal of Computational Chemistry, Vol. 34, Issue 25
  • DOI: 10.1002/jcc.23354

Optimizing legacy molecular dynamics software with directive-based offload
journal, October 2015

  • Michael Brown, W.; Carrillo, Jan-Michael Y.; Gavhane, Nitin
  • Computer Physics Communications, Vol. 195
  • DOI: 10.1016/j.cpc.2015.05.004

Atomic-Level Characterization of the Structural Dynamics of Proteins
journal, October 2010

Scalable molecular dynamics with NAMD
journal, January 2005

  • Phillips, James C.; Braun, Rosemary; Wang, Wei
  • Journal of Computational Chemistry, Vol. 26, Issue 16, p. 1781-1802
  • DOI: 10.1002/jcc.20289

Mature HIV-1 capsid structure by cryo-electron microscopy and all-atom molecular dynamics
journal, May 2013

  • Zhao, Gongpu; Perilla, Juan R.; Yufenyuy, Ernest L.
  • Nature, Vol. 497, Issue 7451
  • DOI: 10.1038/nature12162

Constructing irregular surfaces to enclose macromolecular complexes for mesoscale modeling using the discrete surface charge optimization (DISCO) algorithm
journal, October 2003

  • Zhang, Qing; Beard, Daniel A.; Schlick, Tamar
  • Journal of Computational Chemistry, Vol. 24, Issue 16
  • DOI: 10.1002/jcc.10337

Numerical integration of the cartesian equations of motion of a system with constraints: molecular dynamics of n-alkanes
journal, March 1977

  • Ryckaert, Jean-Paul; Ciccotti, Giovanni; Berendsen, Herman J. C.
  • Journal of Computational Physics, Vol. 23, Issue 3
  • DOI: 10.1016/0021-9991(77)90098-5

GROMACS: High performance molecular simulations through multi-level parallelism from laptops to supercomputers
journal, September 2015

Parallel implementation of 3D FFT with volumetric decomposition schemes for efficient molecular dynamics simulations
journal, March 2016

  • Jung, Jaewoon; Kobayashi, Chigusa; Imamura, Toshiyuki
  • Computer Physics Communications, Vol. 200
  • DOI: 10.1016/j.cpc.2015.10.024

Settle: An analytical version of the SHAKE and RATTLE algorithm for rigid water models
journal, October 1992

  • Miyamoto, Shuichi; Kollman, Peter A.
  • Journal of Computational Chemistry, Vol. 13, Issue 8
  • DOI: 10.1002/jcc.540130805

Efficient lookup table using a linear function of inverse distance squared
journal, August 2013

  • Jung, Jaewoon; Mori, Takaharu; Sugita, Yuji
  • Journal of Computational Chemistry, Vol. 34, Issue 28
  • DOI: 10.1002/jcc.23404

Magnesium Fluctuations Modulate RNA Dynamics in the SAM-I Riboswitch
journal, July 2012

  • Hayes, Ryan L.; Noel, Jeffrey K.; Mohanty, Udayan
  • Journal of the American Chemical Society, Vol. 134, Issue 29
  • DOI: 10.1021/ja301454u

GENESIS 1.1: A hybrid-parallel molecular dynamics simulator with enhanced sampling algorithms on multiple computational platforms
journal, July 2017

  • Kobayashi, Chigusa; Jung, Jaewoon; Matsunaga, Yasuhiro
  • Journal of Computational Chemistry, Vol. 38, Issue 25
  • DOI: 10.1002/jcc.24874

Midpoint cell method for hybrid (MPI+OpenMP) parallelization of molecular dynamics simulations
journal, March 2014

  • Jung, Jaewoon; Mori, Takaharu; Sugita, Yuji
  • Journal of Computational Chemistry, Vol. 35, Issue 14
  • DOI: 10.1002/jcc.23591

Rattle: A “velocity” version of the shake algorithm for molecular dynamics calculations
journal, October 1983

A smooth particle mesh Ewald method
journal, November 1995

  • Essmann, Ulrich; Perera, Lalith; Berkowitz, Max L.
  • The Journal of Chemical Physics, Vol. 103, Issue 19
  • DOI: 10.1063/1.470117

MOIL-opt: Energy-Conserving Molecular Dynamics on a GPU/CPU System
journal, August 2011

  • Ruymgaart, A. Peter; Cardenas, Alfredo E.; Elber, Ron
  • Journal of Chemical Theory and Computation, Vol. 7, Issue 10
  • DOI: 10.1021/ct200360f

OpenMM 7: Rapid development of high performance algorithms for molecular dynamics
journal, July 2017

Dynamics of folded proteins
journal, June 1977

  • McCammon, J. Andrew; Gelin, Bruce R.; Karplus, Martin
  • Nature, Vol. 267, Issue 5612
  • DOI: 10.1038/267585a0

ACEMD: Accelerating Biomolecular Dynamics in the Microsecond Time Scale
journal, May 2009

  • Harvey, M. J.; Giupponi, G.; Fabritiis, G. De
  • Journal of Chemical Theory and Computation, Vol. 5, Issue 6
  • DOI: 10.1021/ct9000685

Computational studies of polyurethanases from Pseudomonas
journal, January 2021

  • do Canto, Vanessa Petry; Thompson, Claudia Elizabeth; Netz, Paulo Augusto
  • Journal of Molecular Modeling, Vol. 27, Issue 2
  • DOI: 10.1007/s00894-021-04671-x

Improving the specificity of organophosphorus hydrolase to acephate by mutagenesis at its binding site: a computational study
journal, May 2021

  • Badakhshan, Reza; Mohammadi, Mozafar; Farnoosh, Gholamreza
  • Journal of Molecular Modeling, Vol. 27, Issue 6
  • DOI: 10.1007/s00894-021-04749-6

Spontaneous drying of non-polar deep-cavity cavitand pockets in aqueous solution
journal, May 2020

  • Barnett, J. Wesley; Sullivan, Matthew R.; Long, Joshua A.
  • Nature Chemistry, Vol. 12, Issue 7
  • DOI: 10.1038/s41557-020-0458-8

Rational identification and characterisation of peptide ligands for targeting polysialic acid
journal, May 2020

  • Shastry, Divya G.; Irudayanathan, Flaviyan Jerome; Williams, Asher
  • Scientific Reports, Vol. 10, Issue 1
  • DOI: 10.1038/s41598-020-64088-z
    Sort by:
    [ × clear filter / sort ]

    Works referencing / citing this record:

    Influence of p H on the formation of a polyelectrolyte complex by dissipative particle dynamics simulation: From an extended to a compact shape
    journal, July 2019

    • Meneses-Juárez, Efrain; Márquez-Beltrán, César; González-Melchor, Minerva
    • Physical Review E, Vol. 100, Issue 1
    • DOI: 10.1103/physreve.100.012505
      Sort by:
      [ × clear filter / sort ]

      Similar Records in DOE PAGES and OSTI.GOV collections: