DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Direct simulation Monte Carlo on petaflop supercomputers and beyond

Abstract

The gold-standard definition of the Direct Simulation Monte Carlo (DSMC) method is given in the 1994 book by Bird [Molecular Gas Dynamics and the Direct Simulation of Gas Flows (Clarendon Press, Oxford, UK, 1994)], which refined his pioneering earlier papers in which he first formulated the method. In the intervening 25 years, DSMC has become the method of choice for modeling rarefied gas dynamics in a variety of scenarios. The chief concern to applying DSMC to more dense or even continuum flows is its computational expense compared to continuum computational fluid dynamics methods. The dramatic (nearly billion-fold) increase in speed of the largest supercomputers over the last 30 years has thus been a key enabling factor in using DSMC to model a richer variety of flows, due to the method’s inherent parallelism. We have developed the open-source SPARTA DSMC code with the goal of running DSMC efficiently on the largest machines, both current and future. It is largely an implementation of Bird’s 1994 formulation. In this work, we describe algorithms used in SPARTA to enable DSMC to operate in parallel at the scale of many billions of particles or grid cells, or with billions of surface elements. We highlight amore » few examples of the kinds of fundamental physics questions and engineering applications that DSMC can address at these scales.« less

Authors:
 [1];  [1];  [2]; ORCiD logo [1];  [1]; ORCiD logo [1]; ORCiD logo [1]
  1. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
  2. Science and Technology Corporation at NASA Ames Research Center, Moffett Field, California 94035, USA
Publication Date:
Research Org.:
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA)
OSTI Identifier:
1559524
Report Number(s):
SAND-2019-9819J
Journal ID: ISSN 1070-6631; 678677; TRN: US2000361
Grant/Contract Number:  
AC04-94AL85000; NA0003525
Resource Type:
Accepted Manuscript
Journal Name:
Physics of Fluids
Additional Journal Information:
Journal Volume: 31; Journal Issue: 8; Journal ID: ISSN 1070-6631
Publisher:
American Institute of Physics (AIP)
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Direct Simulation Monte Carlo; DSMC; parallel algorithms; SPARTA

Citation Formats

Plimpton, S. J., Moore, S. G., Borner, A., Stagg, A. K., Koehler, T. P., Torczynski, J. R., and Gallis, M. A. Direct simulation Monte Carlo on petaflop supercomputers and beyond. United States: N. p., 2019. Web. doi:10.1063/1.5108534.
Plimpton, S. J., Moore, S. G., Borner, A., Stagg, A. K., Koehler, T. P., Torczynski, J. R., & Gallis, M. A. Direct simulation Monte Carlo on petaflop supercomputers and beyond. United States. https://doi.org/10.1063/1.5108534
Plimpton, S. J., Moore, S. G., Borner, A., Stagg, A. K., Koehler, T. P., Torczynski, J. R., and Gallis, M. A. Thu . "Direct simulation Monte Carlo on petaflop supercomputers and beyond". United States. https://doi.org/10.1063/1.5108534. https://www.osti.gov/servlets/purl/1559524.
@article{osti_1559524,
title = {Direct simulation Monte Carlo on petaflop supercomputers and beyond},
author = {Plimpton, S. J. and Moore, S. G. and Borner, A. and Stagg, A. K. and Koehler, T. P. and Torczynski, J. R. and Gallis, M. A.},
abstractNote = {The gold-standard definition of the Direct Simulation Monte Carlo (DSMC) method is given in the 1994 book by Bird [Molecular Gas Dynamics and the Direct Simulation of Gas Flows (Clarendon Press, Oxford, UK, 1994)], which refined his pioneering earlier papers in which he first formulated the method. In the intervening 25 years, DSMC has become the method of choice for modeling rarefied gas dynamics in a variety of scenarios. The chief concern to applying DSMC to more dense or even continuum flows is its computational expense compared to continuum computational fluid dynamics methods. The dramatic (nearly billion-fold) increase in speed of the largest supercomputers over the last 30 years has thus been a key enabling factor in using DSMC to model a richer variety of flows, due to the method’s inherent parallelism. We have developed the open-source SPARTA DSMC code with the goal of running DSMC efficiently on the largest machines, both current and future. It is largely an implementation of Bird’s 1994 formulation. In this work, we describe algorithms used in SPARTA to enable DSMC to operate in parallel at the scale of many billions of particles or grid cells, or with billions of surface elements. We highlight a few examples of the kinds of fundamental physics questions and engineering applications that DSMC can address at these scales.},
doi = {10.1063/1.5108534},
journal = {Physics of Fluids},
number = 8,
volume = 31,
place = {United States},
year = {Thu Aug 01 00:00:00 EDT 2019},
month = {Thu Aug 01 00:00:00 EDT 2019}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 97 works
Citation information provided by
Web of Science

Figures / Tables:

Figure 1 Figure 1: A 2D slice of a 3D hierarchical grid around the Apollo space capsule. The grid has 5 levels of refinement that track the spatially varying particle density resulting from gas flow from the bottom left to the top right of the simulation domain.

Save / Share:

Works referenced in this record:

Application highlights of the DSMC Analysis Code (DAC) software for simulating rarefied flows
journal, December 2001

  • LeBeau, G. J.; Lumpkin III, F. E.
  • Computer Methods in Applied Mechanics and Engineering, Vol. 191, Issue 6-7
  • DOI: 10.1016/s0045-7825(01)00304-8

Robust cut-cell algorithms for DSMC implementations employing multi-level Cartesian grids
journal, October 2012


An overview of Rayleigh-Taylor instability
journal, July 1984


Kokkos: Enabling manycore performance portability through polymorphic memory access patterns
journal, December 2014

  • Carter Edwards, H.; Trott, Christian R.; Sunderland, Daniel
  • Journal of Parallel and Distributed Computing, Vol. 74, Issue 12
  • DOI: 10.1016/j.jpdc.2014.07.003

Efficient Implementation of Marching Cubes' Cases with Topological Guarantees
journal, January 2003


PuMA: the Porous Microstructure Analysis software
journal, January 2018


A new algorithm for Monte Carlo simulation of Ising spin systems
journal, January 1975


High temperature permeability of fibrous materials using direct simulation Monte Carlo
journal, March 2017


Nonequilibrium flow through porous thermal protection materials, Part I: Numerical methods
journal, March 2019

  • Stern, Eric C.; Poovathingal, Savio; Nompelis, Ioannis
  • Journal of Computational Physics, Vol. 380
  • DOI: 10.1016/j.jcp.2017.09.011

The instability of liquid surfaces when accelerated in a direction perpendicular to their planes. II
journal, June 1950

  • Lewis, D. J.
  • Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, Vol. 202, Issue 1068, p. 81-96
  • DOI: 10.1098/rspa.1950.0086

Hidden surface removal using polygon area sorting
journal, August 1977

  • Weiler, Kevin; Atherton, Peter
  • ACM SIGGRAPH Computer Graphics, Vol. 11, Issue 2
  • DOI: 10.1145/965141.563896

Effects of Diffusion on Interface Instability between Gases
journal, January 1962

  • Duff, R. E.; Harlow, F. H.; Hirt, C. W.
  • Physics of Fluids, Vol. 5, Issue 4
  • DOI: 10.1063/1.1706634

Statistical simulation of the transition between regular and mach reflection in steady flows
journal, January 1998

  • Ivanov, M. S.; Gimelshein, S. F.; Markelov, G. N.
  • Computers & Mathematics with Applications, Vol. 35, Issue 1-2
  • DOI: 10.1016/s0898-1221(97)00262-9

An open source, parallel DSMC code for rarefied gas flows in arbitrary geometries
journal, December 2010


A new Strouhal–Reynolds-number relationship for the circular cylinder in the range 47<Re<2×105
journal, July 1998

  • Fey, Uwe; König, Michael; Eckelmann, Helmut
  • Physics of Fluids, Vol. 10, Issue 7
  • DOI: 10.1063/1.869675

Practical considerations on Marching Cubes 33 topological correctness
journal, November 2013


Direct simulation Monte Carlo investigation of the Richtmyer-Meshkov instability
journal, August 2015

  • Gallis, M. A.; Koehler, T. P.; Torczynski, J. R.
  • Physics of Fluids, Vol. 27, Issue 8
  • DOI: 10.1063/1.4928338

Direct Simulation Monte Carlo Simulations of Hypersonic Flows With Shock Interactions
journal, December 2005

  • Moss, James N.; Bird, Graeme A.
  • AIAA Journal, Vol. 43, Issue 12
  • DOI: 10.2514/1.12532

Exact stochastic simulation of coupled chemical reactions
journal, December 1977

  • Gillespie, Daniel T.
  • The Journal of Physical Chemistry, Vol. 81, Issue 25
  • DOI: 10.1021/j100540a008

Modeling the oxidation of low-density carbon fiber material based on micro-tomography
journal, January 2016


Gas-kinetic simulation of sustained turbulence in minimal Couette flow
journal, July 2018


An experimental assembly for precise measurement of thermal accommodation coefficients
journal, March 2011

  • Trott, Wayne M.; Castañeda, Jaime N.; Torczynski, John R.
  • Review of Scientific Instruments, Vol. 82, Issue 3
  • DOI: 10.1063/1.3571269

Nanohydrodynamics simulations: An atomistic view of the Rayleigh-Taylor instability
journal, April 2004

  • Kadau, K.; Germann, T. C.; Hadjiconstantinou, N. G.
  • Proceedings of the National Academy of Sciences, Vol. 101, Issue 16
  • DOI: 10.1073/pnas.0401228101

Marching cubes: A high resolution 3D surface construction algorithm
journal, August 1987

  • Lorensen, William E.; Cline, Harvey E.
  • ACM SIGGRAPH Computer Graphics, Vol. 21, Issue 4
  • DOI: 10.1145/37402.37422

Approach to Translational Equilibrium in a Rigid Sphere Gas
journal, January 1963


The instability of liquid surfaces when accelerated in a direction perpendicular to their planes. I
journal, March 1950

  • Taylor, Geoffrey Ingram
  • Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, Vol. 201, Issue 1065, p. 192-196
  • DOI: 10.1098/rspa.1950.0052

A general method for numerically simulating the stochastic time evolution of coupled chemical reactions
journal, December 1976


Atomistic methods in fluid simulation
journal, April 2010

  • Kadau, Kai; Barber, John L.; Germann, Timothy C.
  • Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 368, Issue 1916
  • DOI: 10.1098/rsta.2009.0218

dsmcFoam+: An OpenFOAM based direct simulation Monte Carlo solver
journal, March 2018


3D DSMC computations on a heterogeneous CPU-GPU cluster with a large number of GPUs
conference, January 2014

  • Kashkovsky, Alexander
  • PROCEEDINGS OF THE 29TH INTERNATIONAL SYMPOSIUM ON RAREFIED GAS DYNAMICS, AIP Conference Proceedings
  • DOI: 10.1063/1.4902592

On the Instability of Superposed Fluids in a Gravitational Field.
journal, July 1955

  • Layzer, David
  • The Astrophysical Journal, Vol. 122
  • DOI: 10.1086/146048

A parallel rendezvous algorithm for interpolation between multiple grids
journal, February 2004

  • Plimpton, Steven J.; Hendrickson, Bruce; Stewart, James R.
  • Journal of Parallel and Distributed Computing, Vol. 64, Issue 2
  • DOI: 10.1016/j.jpdc.2003.11.006

Review of theoretical modelling approaches of Rayleigh–Taylor instabilities and turbulent mixing
journal, April 2010

  • Abarzhi, Snezhana I.
  • Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 368, Issue 1916
  • DOI: 10.1098/rsta.2010.0020

Atomistic simulation of the Rayleigh-Taylor instability
journal, September 2006


A Parallel Rendezvous Algorithm for Interpolation Between Multiple Grids
conference, January 1998

  • Plimpton, S.; Hendrickson, B.; Stewart, J.
  • SC98 - High Performance Networking and Computing Conference, Proceedings of the IEEE/ACM SC98 Conference
  • DOI: 10.1109/sc.1998.10032

Marching cubes: A high resolution 3D surface construction algorithm
conference, January 1987

  • Lorensen, William E.; Cline, Harvey E.
  • Proceedings of the 14th annual conference on Computer graphics and interactive techniques - SIGGRAPH '87
  • DOI: 10.1145/37401.37422

Robust Cut-cell Algorithms for DSMC Implementations Employing Multi-level Cartesian Grids
conference, June 2011

  • Zhang, Chonglin; Schwartzentruber, Thomas
  • 42nd AIAA Thermophysics Conference
  • DOI: 10.2514/6.2011-3314

Hidden surface removal using polygon area sorting
conference, January 1977

  • Weiler, Kevin; Atherton, Peter
  • Proceedings of the 4th annual conference on Computer graphics and interactive techniques - SIGGRAPH '77
  • DOI: 10.1145/563858.563896

Works referencing / citing this record:

Graeme A. Bird
journal, November 2019