skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Strong scaling of general-purpose molecular dynamics simulations on GPUs

Journal Article · · Computer Physics Communications
 [1];  [2];  [1];  [3]; ORCiD logo [4];  [1];  [5]; ORCiD logo [1]
  1. Univ. of Michigan, Ann Arbor, MI (United States)
  2. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). National Center for Computational Sciences
  3. Mellanox Technologies, Inc., Sunnyvale, CA (United States)
  4. Univ. of Cambridge (United Kingdom). High Performance Computing Service
  5. Univ. of Minnesota, Minneapolis, MN (United States). Dept. of Chemical Engineering and Materials Science

We describe a highly optimized implementation of MPI domain decomposition in a GPU-enabled, general-purpose molecular dynamics code, HOOMD-blue (Anderson and Glotzer, 2013). Here, our approach is inspired by a traditional CPU-based code, LAMMPS (Plimpton, 1995), but is implemented within a code that was designed for execution on GPUs from the start (Anderson et al., 2008). The software supports short-ranged pair force and bond force fields and achieves optimal GPU performance using an autotuning algorithm. We are able to demonstrate equivalent or superior scaling on up to 3375 GPUs in Lennard-Jones and dissipative particle dynamics (DPD) simulations of up to 108 million particles. GPUDirect RDMA capabilities in recent GPU generations provide better performance in full double precision calculations. For a representative polymer physics application, HOOMD-blue 1.0 provides an effective GPU vs. CPU node speed-up of 12.5x.

Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Organization:
USDOE Office of Science (SC)
Grant/Contract Number:
AC05-00OR22725
OSTI ID:
1565293
Alternate ID(s):
OSTI ID: 1246983; OSTI ID: 1565294; OSTI ID: 1820890
Journal Information:
Computer Physics Communications, Vol. 192, Issue C; ISSN 0010-4655
Publisher:
ElsevierCopyright Statement
Country of Publication:
United States
Language:
English
Citation Metrics:
Cited by: 478 works
Citation information provided by
Web of Science

References (36)

General purpose molecular dynamics simulations fully implemented on graphics processing units journal May 2008
Fast Parallel Algorithms for Short-Range Molecular Dynamics journal March 1995
Implementing molecular dynamics on hybrid high performance computers – short range forces journal April 2011
Accelerating dissipative particle dynamics simulations on GPUs: Algorithms, numerics and applications journal November 2014
A flexible algorithm for calculating pair interactions on SIMD architectures journal December 2013
Accelerating molecular modeling applications with graphics processors journal January 2007
OpenMM 4: A Reusable, Extensible, Hardware Independent Library for High Performance Molecular Simulation journal November 2012
ACEMD: Accelerating Biomolecular Dynamics in the Microsecond Time Scale journal May 2009
Lattice Boltzmann simulations on GPUs with ESPResSo journal August 2012
Highly accelerated simulations of glassy dynamics using GPUs: Caveats on limited floating-point precision journal May 2011
Universality of Block Copolymer Melts journal August 2014
GPU Based Molecular Dynamics Simulations of Polymer Rings in Concentrated Solution: Structure and Scaling journal January 2011
Micellization Studied by GPU-Accelerated Coarse-Grained Molecular Dynamics journal October 2011
Anomalous structure and scaling of ring polymer brushes journal June 2011
Decreasing Polymer Flexibility Improves Wetting and Dispersion of Polymer-Grafted Particles in a Chemically Identical Polymer Matrix journal June 2014
Rupture mechanism of liquid crystal thin films realized by large-scale molecular simulations journal January 2014
Coexistence of spinodal instability and thermal nucleation in thin-film rupture: Insights from molecular levels journal March 2014
Crossover to surface flow in supercooled unentangled polymer films journal October 2013
Highly Scalable and Memory Efficient Ultra-Coarse-Grained Molecular Dynamics Simulations journal November 2013
Stability of the double gyroid phase to nanoparticle polydispersity in polymer-tethered nanosphere systems journal January 2010
Langevin dynamics simulations of a two-dimensional colloidal crystal under confinement and shear journal June 2012
Probing the limitations of isotropic pair potentials to produce ground-state structural extremes via inverse statistical mechanics journal October 2013
Thermal and athermal three-dimensional swarms of self-propelled particles journal July 2012
Viral genome structures are optimal for capsid assembly journal June 2013
Effects of supercoiling on enhancer–promoter contacts journal August 2014
Folding and stability of helical bundle proteins from coarse-grained models: Generic Coarse-Grained Model for Proteins journal April 2013
Constant pressure molecular dynamics algorithms journal September 1994
Explicit reversible integrators for extended systems dynamics journal April 1996
Pseudo-random number generation for Brownian Dynamics and Dissipative Particle Dynamics simulations on GPU devices journal August 2011
Implementing molecular dynamics on hybrid high performance computers – Particle–particle particle-mesh journal March 2012
An optimized united atom model for simulations of polymethylene melts journal July 1995
The development of Mellanox/NVIDIA GPUDirect over InfiniBand—a new model for GPU to GPU communications journal April 2011
MVAPICH2-GPU: optimized GPU to GPU communication for InfiniBand clusters journal April 2011
GPU-Aware MPI on RDMA-Enabled Clusters: Design, Implementation and Evaluation journal October 2014
Scalable molecular dynamics with NAMD journal January 2005
GROMACS 4:  Algorithms for Highly Efficient, Load-Balanced, and Scalable Molecular Simulation journal February 2008

Cited By (136)

Non-equilibrium anisotropic colloidal single crystal growth with DNA journal November 2018
Phase behavior and design rules for plastic colloidal crystals of hard polyhedra via consideration of directional entropic forces journal January 2019
Enhanced diffusion and magnetophoresis of paramagnetic colloidal particles in rotating magnetic fields journal January 2019
Employing multi-GPU power for molecular dynamics simulation: an extension of GALAMOST journal November 2017
Fast Stokesian dynamics journal September 2019
Molecular height measurement by cell surface optical profilometry (CSOP) journal January 2020
Effects of glycol on adsorption dynamics of idealized water droplets on LTA‐3A zeolite surfaces journal February 2019
A new insight into the isotropic–nematic phase transition in lyotropic solutions of semiflexible polymers: density-functional theory tested by molecular dynamics journal January 2016
Alchemical molecular dynamics for inverse design text January 2019
Anomalous dynamics of interstitial dopants in soft crystals journal November 2016
Active binary mixtures of fast and slow hard spheres journal January 2020
Photoactive Blend Morphology Engineering through Systematically Tuning Aggregation in All-Polymer Solar Cells journal January 2018
Structure and dynamics of amphiphilic Janus spheres and spherocylinders under shear journal January 2020
The alchemical energy landscape for a pentameric cluster journal January 2020
Chain length-dependent luminescence in acceptor-doped conjugated polymers journal August 2019
Selective enhanced sampling in dihedral energy facilitates overcoming the dihedral energy increase in protein folding and accelerates the searching for protein native structure journal January 2019
The in-silico lab-on-a-chip: petascale and high-throughput simulations of microfluidics at cell resolution
  • Rossinelli, Diego; Karniadakis, George; Fatica, Massimiliano
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15 https://doi.org/10.1145/2807591.2807677
conference January 2015
Optimization and Validation of Efficient Models for Predicting Polythiophene Self-Assembly journal November 2018
Colloidal fibers and rings by cooperative assembly journal September 2019
Protracted Colored Noise Dynamics Applied to Linear Polymer Systems journal January 2018
Regulating lipid composition rationalizes acyl-tail saturation homeostasis in ectotherms posted_content December 2019
Classical molecular dynamics on graphics processing unit architectures journal August 2019
5D Entanglement in Star Polymer Dynamics journal September 2018
Semiflexible polymers confined in a slit pore with attractive walls: two-dimensional liquid crystalline order versus capillary nematization journal January 2017
Calculation of critical nucleation rates by the persistent embryo method: application to quasi hard sphere models journal January 2018
Supercharging enables organized assembly of synthetic biomolecules journal January 2019
Entropic colloidal crystallization pathways via fluid–fluid transitions and multidimensional prenucleation motifs journal July 2019
Simultaneous Nano- and Microscale Control of Nanofibrous Microspheres Self-Assembled from Star-Shaped Polymers journal May 2015
ESCO1 and CTCF enable formation of long chromatin loops by protecting cohesin STAG1 from WAPL journal September 2019
Unsupervised machine learning for detection of phase transitions in off-lattice systems. II. Applications journal November 2018
Morphology of depletant-induced erythrocyte aggregates journal January 2018
Phase behavior of flexible and semiflexible polymers in solvents of varying quality journal July 2019
On mean-field theories of dynamics in supercooled liquids journal August 2019
Tunable emergent structures and traveling waves in mixtures of passive and contact-triggered-active particles journal January 2017
Anomalous Fluctuations of Nematic Order in Solutions of Semiflexible Polymers journal May 2016
Unsupervised machine learning for detection of phase transitions in off-lattice systems. I. Foundations journal November 2018
rowan: A Python package for working with quaternions journal July 2018
Why large icosahedral viruses need scaffolding proteins journal October 2018
Strain fields in graphene induced by nanopillar mesh journal February 2019
Magic number colloidal clusters as minimum free energy structures journal December 2018
Bulk structural information from density functionals for patchy particles journal December 2018
A versatile simulation method for studying phase behavior and dynamics in colloidal rod and rod-polymer suspensions journal June 2019
PySFD: comprehensive molecular insights from significant feature differences detected among many simulated ensembles journal September 2018
Allosteric pathway selection in templated assembly journal October 2019
Hierarchical self-assembly of hard cube derivatives journal January 2019
Efficient equilibration of hard spheres with Newtonian event chains journal May 2019
Principle of corresponding states for hard polyhedron fluids text January 2019
Nematic order in solutions of semiflexible polymers: Hairpins, elastic constants, and the nematic-smectic transition journal November 2018
Inverse design of self-assembling colloidal crystals with omnidirectional photonic bandgaps journal January 2019
Theoretical principles of transcription factor traffic on folded chromatin journal April 2018
Colloidal alloys with preassembled clusters and spheres journal February 2017
Assembly of particle strings via isotropic potentials journal March 2019
Shapes within shapes: how particles arrange inside a cavity journal January 2018
Orientational ordering of lamellar structures on closed surfaces journal May 2018
Structural Color of Colloidal Clusters as a Tool to Investigate Structure and Dynamics journal November 2019
Axial dispersion of Brownian colloids in microfluidic channels journal August 2016
Influence of polymer flexibility on nanoparticle dynamics in semidilute solutions journal January 2019
Inertio-capillary cross-streamline drift of droplets in Poiseuille flow using dissipative particle dynamics simulations journal January 2018
A multi-device version of the HYFMGPU algorithm for hyperspectral scenes registration journal November 2018
Activity-controlled annealing of colloidal monolayers journal July 2019
Influence of hydrodynamic interactions on stratification in drying mixtures journal July 2018
Quasicrystalline nanocrystal superlattice with partial matching rules journal September 2016
Dynamics of single semiflexible polymers in dilute solution journal December 2016
Low-Resolution Models for the Interaction Dynamics of Coated Gold Nanoparticles with β2-microglobulin journal August 2019
Biomolecular simulations: From dynamics and mechanisms to computational assays of biological activity
  • Huggins, David J.; Biggin, Philip C.; Dämgen, Marc A.
  • Wiley Interdisciplinary Reviews: Computational Molecular Science, Vol. 9, Issue 3 https://doi.org/10.1002/wcms.1393
journal September 2018
Evaporation-induced assembly of colloidal crystals journal September 2018
Phase diagrams, mechanisms and unique characteristics of alternating-structured polymer self-assembly via simulations journal November 2018
Understanding, predicting, and tuning the fragility of vitrimeric polymers journal November 2019
Onset of transverse (shear) waves in strongly-coupled Yukawa fluids journal March 2019
Microscopic origins of the swim pressure and the anomalous surface tension of active matter journal January 2020
Size-dependent thermodynamic structural selection in colloidal crystallization journal September 2019
Identity crisis in alchemical space drives the entropic colloidal glass transition journal January 2019
Self-diffusion in single-component Yukawa fluids journal April 2018
Clathrate colloidal crystals journal March 2017
Inverse design of simple pair potentials for the self-assembly of complex structures journal November 2018
Cross-stream migration of a Brownian droplet in a polymer solution under Poiseuille flow journal January 2019
Chromatin Loop Extrusion and Chromatin Unknotting journal October 2018
Self-assembly of lobed particles into amorphous and crystalline porous structures journal January 2020
RUMD: A general purpose molecular dynamics package optimized to utilize GPU hardware down to a few thousand particles journal January 2017
Rapid sampling of stochastic displacements in Brownian dynamics simulations journal March 2017
Freezing on a sphere journal February 2018
FastNBL: fast neighbor lists establishment for molecular dynamics simulation based on bitwise operations journal April 2019
Air separation with graphene mediated by nanowindow-rim concerted motion journal May 2018
Design, Parameterization, and Implementation of Atomic Force Fields for Adsorption in Nanoporous Materials journal September 2019
Self-Diffusion in a Spatially Modulated System of Electrons on Helium journal February 2019
Static and dynamic properties of large polymer melts in equilibrium journal April 2016
Excitation spectra in fluids: How to analyze them properly journal July 2019
Doping colloidal bcc crystals — interstitial solids and meta-stable clusters journal October 2017
Hemoglobin-driven iron-directed assembly of gold nanoparticles journal January 2018
Self-assembly of semiflexible polymers confined to thin spherical shells journal January 2018
Harnessing entropy to enhance toughness in reversibly crosslinked polymer networks journal January 2019
Patchy particles by self-assembly of star copolymers on a spherical substrate: Thomson solutions in a geometric problem with a color constraint journal January 2019
Enhanced gel formation in binary mixtures of nanocolloids with short-range attraction journal January 2018
Collective modes of two-dimensional classical Coulomb fluids journal October 2018
Configurational entropy of glass-forming liquids journal April 2019
Characterization of colloidal nanocrystal surface structure using small angle neutron scattering and efficient Bayesian parameter estimation journal June 2019
Universal folding pathways of polyhedron nets journal July 2018
The race to the bottom: approaching the ideal glass? journal August 2018
Charged hydrophobic colloids at an oil–aqueous phase interface journal December 2015
The alchemical energy landscape for a pentameric cluster. text January 2020
Flow behavior of chain and star polymers and their mixtures text January 2018
ESCO1 and CTCF enable formation of long chromatin loops by protecting cohesinSTAG1 from WAPL journal February 2020
Adsorption characteristics of glycols on calcite and hematite journal July 2019
OSPREY 3.0: Open‐source protein redesign for you, with powerful new features journal October 2018
Principle of corresponding states for hard polyhedron fluids journal July 2019
Implementing molecular dynamics simulation on the Sunway TaihuLight system with heterogeneous many-core processors: Implementing molecular dynamics simulation on the Sunway TaihuLight system with heterogeneous many-core processors journal April 2018
Two-time correlations for probing the aging dynamics of glassy colloids journal January 2018
Molecular height measurement by cell surface optical profilometry (CSOP) journal June 2020
Chromatin-loop extrusion and chromatin unknotting journal September 2018
Routine million-particle simulations of epoxy curing with dissipative particle dynamics journal May 2018
Principle of corresponding states for hard polyhedron fluids text January 2019
Biomolecular simulations: from dynamics and mechanisms to computational assays of biological activity text January 2019
Exploring the zone of anisotropy and broken symmetries in DNA-mediated nanoparticle crystallization journal September 2016
Alchemical molecular dynamics for inverse design journal October 2019
OSPREY 3.0: Open-Source Protein Redesign for You, with Powerful New Features journal April 2018
RUMD: A general purpose molecular dynamics package optimized to utilize GPU hardware down to a few thousand particles text January 2015
Anomalous dynamics of interstitial dopants in soft crystals text January 2016
Rapid Sampling of Stochastic Displacements in Brownian Dynamics Simulations text January 2016
Inverse Design of Simple Pair Potentials for the Self-Assembly of Complex Structures text January 2017
The race to the bottom: approaching the ideal glass? text January 2017
Freezing on a Sphere text January 2018
Collective modes of two-dimensional classical Coulomb fluids text January 2018
Bulk structural informations from density functionals for patchy particles text January 2018
Strain field in graphene induced by nanopillar mesh text January 2018
Phase behavior of flexible and semiflexible polymers in solvents of varying quality text January 2019
Attractive versus truncated repulsive supercooled liquids: The dynamics is encoded in the pair correlation function text January 2019
Patchy particles by self-assembly of star copolymers on a spherical substrate: Thomson solutions in a geometric problem with a color constraint preprint January 2019
Active binary mixtures of fast and slow hard spheres preprint January 2019
Activity-controlled Annealing of Colloidal Monolayers text January 2019
Self-Consistent Field Lattice Model for Polymer Networks journal December 2017
Langevin Dynamics Simulations of the Exchange of Complex Coacervate Core Micelles: The Role of Nonelectrostatic Attraction and Polyelectrolyte Length journal November 2019
Disentangling the Role of Chain Conformation on the Mechanics of Polymer Tethered Particle Materials journal March 2019
Segregation in Drying Binary Colloidal Droplets journal March 2019
Thermal crumpling of perforated two-dimensional sheets journal November 2017
Evolutionary drivers of protein shape journal August 2019
Semiflexible Polymers in the Bulk and Confined by Planar Walls journal August 2016