## This content will become publicly available on August 1, 2020

# Direct simulation Monte Carlo on petaflop supercomputers and beyond

## Abstract

The gold-standard definition of the Direct Simulation Monte Carlo (DSMC) method is given in the 1994 book by Bird [Molecular Gas Dynamics and the Direct Simulation of Gas Flows (Clarendon Press, Oxford, UK, 1994)], which refined his pioneering earlier papers in which he first formulated the method. In the intervening 25 years, DSMC has become the method of choice for modeling rarefied gas dynamics in a variety of scenarios. The chief concern to applying DSMC to more dense or even continuum flows is its computational expense compared to continuum computational fluid dynamics methods. The dramatic (nearly billion-fold) increase in speed of the largest supercomputers over the last 30 years has thus been a key enabling factor in using DSMC to model a richer variety of flows, due to the method’s inherent parallelism. We have developed the open-source SPARTA DSMC code with the goal of running DSMC efficiently on the largest machines, both current and future. It is largely an implementation of Bird’s 1994 formulation. In this work, we describe algorithms used in SPARTA to enable DSMC to operate in parallel at the scale of many billions of particles or grid cells, or with billions of surface elements. We highlight amore »

- Authors:

- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
- Science and Technology Corporation at NASA Ames Research Center, Moffett Field, California 94035, USA

- Publication Date:

- Research Org.:
- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

- Sponsoring Org.:
- USDOE National Nuclear Security Administration (NNSA)

- OSTI Identifier:
- 1559524

- Report Number(s):
- SAND-2019-9819J

Journal ID: ISSN 1070-6631; 678677

- Grant/Contract Number:
- AC04-94AL85000; NA0003525

- Resource Type:
- Accepted Manuscript

- Journal Name:
- Physics of Fluids

- Additional Journal Information:
- Journal Volume: 31; Journal Issue: 8; Journal ID: ISSN 1070-6631

- Publisher:
- American Institute of Physics (AIP)

- Country of Publication:
- United States

- Language:
- English

- Subject:
- 97 MATHEMATICS AND COMPUTING; Direct Simulation Monte Carlo; DSMC; parallel algorithms; SPARTA

### Citation Formats

```
Plimpton, S. J., Moore, S. G., Borner, A., Stagg, A. K., Koehler, T. P., Torczynski, J. R., and Gallis, M. A. Direct simulation Monte Carlo on petaflop supercomputers and beyond. United States: N. p., 2019.
Web. doi:10.1063/1.5108534.
```

```
Plimpton, S. J., Moore, S. G., Borner, A., Stagg, A. K., Koehler, T. P., Torczynski, J. R., & Gallis, M. A. Direct simulation Monte Carlo on petaflop supercomputers and beyond. United States. doi:10.1063/1.5108534.
```

```
Plimpton, S. J., Moore, S. G., Borner, A., Stagg, A. K., Koehler, T. P., Torczynski, J. R., and Gallis, M. A. Thu .
"Direct simulation Monte Carlo on petaflop supercomputers and beyond". United States. doi:10.1063/1.5108534.
```

```
@article{osti_1559524,
```

title = {Direct simulation Monte Carlo on petaflop supercomputers and beyond},

author = {Plimpton, S. J. and Moore, S. G. and Borner, A. and Stagg, A. K. and Koehler, T. P. and Torczynski, J. R. and Gallis, M. A.},

abstractNote = {The gold-standard definition of the Direct Simulation Monte Carlo (DSMC) method is given in the 1994 book by Bird [Molecular Gas Dynamics and the Direct Simulation of Gas Flows (Clarendon Press, Oxford, UK, 1994)], which refined his pioneering earlier papers in which he first formulated the method. In the intervening 25 years, DSMC has become the method of choice for modeling rarefied gas dynamics in a variety of scenarios. The chief concern to applying DSMC to more dense or even continuum flows is its computational expense compared to continuum computational fluid dynamics methods. The dramatic (nearly billion-fold) increase in speed of the largest supercomputers over the last 30 years has thus been a key enabling factor in using DSMC to model a richer variety of flows, due to the method’s inherent parallelism. We have developed the open-source SPARTA DSMC code with the goal of running DSMC efficiently on the largest machines, both current and future. It is largely an implementation of Bird’s 1994 formulation. In this work, we describe algorithms used in SPARTA to enable DSMC to operate in parallel at the scale of many billions of particles or grid cells, or with billions of surface elements. We highlight a few examples of the kinds of fundamental physics questions and engineering applications that DSMC can address at these scales.},

doi = {10.1063/1.5108534},

journal = {Physics of Fluids},

number = 8,

volume = 31,

place = {United States},

year = {2019},

month = {8}

}

Works referenced in this record:

##
Application highlights of the DSMC Analysis Code (DAC) software for simulating rarefied flows

journal, December 2001

- LeBeau, G. J.; Lumpkin III, F. E.
- Computer Methods in Applied Mechanics and Engineering, Vol. 191, Issue 6-7

##
Robust cut-cell algorithms for DSMC implementations employing multi-level Cartesian grids

journal, October 2012

- Zhang, Chonglin; Schwartzentruber, Thomas E.
- Computers & Fluids, Vol. 69

##
An overview of Rayleigh-Taylor instability

journal, July 1984

- Sharp, D. H.
- Physica D: Nonlinear Phenomena, Vol. 12, Issue 1-3

##
Kokkos: Enabling manycore performance portability through polymorphic memory access patterns

journal, December 2014

- Carter Edwards, H.; Trott, Christian R.; Sunderland, Daniel
- Journal of Parallel and Distributed Computing, Vol. 74, Issue 12

##
Efficient Implementation of Marching Cubes' Cases with Topological Guarantees

journal, January 2003

- Lewiner, Thomas; Lopes, Hélio; Vieira, Antônio Wilson
- Journal of Graphics Tools, Vol. 8, Issue 2

##
Rayleigh–Taylor and Richtmyer–Meshkov instability induced flow, turbulence, and mixing. I

journal, December 2017

- Zhou, Ye
- Physics Reports, Vol. 720-722

##
Rayleigh–Taylor and Richtmyer–Meshkov instability induced flow, turbulence, and mixing. II

journal, December 2017

- Zhou, Ye
- Physics Reports, Vol. 723-725

##
PuMA: the Porous Microstructure Analysis software

journal, January 2018

- Ferguson, Joseph C.; Panerai, Francesco; Borner, Arnaud
- SoftwareX, Vol. 7

##
A new algorithm for Monte Carlo simulation of Ising spin systems

journal, January 1975

- Bortz, A. B.; Kalos, M. H.; Lebowitz, J. L.
- Journal of Computational Physics, Vol. 17, Issue 1

##
High temperature permeability of fibrous materials using direct simulation Monte Carlo

journal, March 2017

- Borner, Arnaud; Panerai, Francesco; Mansour, Nagi N.
- International Journal of Heat and Mass Transfer, Vol. 106

##
Nonequilibrium flow through porous thermal protection materials, Part I: Numerical methods

journal, March 2019

- Stern, Eric C.; Poovathingal, Savio; Nompelis, Ioannis
- Journal of Computational Physics, Vol. 380

##
Hidden surface removal using polygon area sorting

journal, August 1977

- Weiler, Kevin; Atherton, Peter
- ACM SIGGRAPH Computer Graphics, Vol. 11, Issue 2

##
Effects of Diffusion on Interface Instability between Gases

journal, January 1962

- Duff, R. E.; Harlow, F. H.; Hirt, C. W.
- Physics of Fluids, Vol. 5, Issue 4

##
Statistical simulation of the transition between regular and mach reflection in steady flows

journal, January 1998

- Ivanov, M. S.; Gimelshein, S. F.; Markelov, G. N.
- Computers & Mathematics with Applications, Vol. 35, Issue 1-2

##
An open source, parallel DSMC code for rarefied gas flows in arbitrary geometries

journal, December 2010

- Scanlon, T. J.; Roohi, E.; White, C.
- Computers & Fluids, Vol. 39, Issue 10

##
A new Strouhal–Reynolds-number relationship for the circular cylinder in the range 47<Re<2×105

journal, July 1998

- Fey, Uwe; König, Michael; Eckelmann, Helmut
- Physics of Fluids, Vol. 10, Issue 7

##
Practical considerations on Marching Cubes 33 topological correctness

journal, November 2013

- Custodio, Lis; Etiene, Tiago; Pesco, Sinesio
- Computers & Graphics, Vol. 37, Issue 7

##
Direct simulation Monte Carlo investigation of the Richtmyer-Meshkov instability

journal, August 2015

- Gallis, M. A.; Koehler, T. P.; Torczynski, J. R.
- Physics of Fluids, Vol. 27, Issue 8

##
Direct Simulation Monte Carlo Simulations of Hypersonic Flows With Shock Interactions

journal, December 2005

- Moss, James N.; Bird, Graeme A.
- AIAA Journal, Vol. 43, Issue 12

##
Exact stochastic simulation of coupled chemical reactions

journal, December 1977

- Gillespie, Daniel T.
- The Journal of Physical Chemistry, Vol. 81, Issue 25

##
Modeling the oxidation of low-density carbon fiber material based on micro-tomography

journal, January 2016

- Ferguson, Joseph C.; Panerai, Francesco; Lachaud, Jean
- Carbon, Vol. 96

##
Gas-kinetic simulation of sustained turbulence in minimal Couette flow

journal, July 2018

- Gallis, M. A.; Torczynski, J. R.; Bitter, N. P.
- Physical Review Fluids, Vol. 3, Issue 7

##
An experimental assembly for precise measurement of thermal accommodation coefficients

journal, March 2011

- Trott, Wayne M.; Castañeda, Jaime N.; Torczynski, John R.
- Review of Scientific Instruments, Vol. 82, Issue 3

##
Nanohydrodynamics simulations: An atomistic view of the Rayleigh-Taylor instability

journal, April 2004

- Kadau, K.; Germann, T. C.; Hadjiconstantinou, N. G.
- Proceedings of the National Academy of Sciences, Vol. 101, Issue 16

##
Marching cubes: A high resolution 3D surface construction algorithm

journal, August 1987

- Lorensen, William E.; Cline, Harvey E.
- ACM SIGGRAPH Computer Graphics, Vol. 21, Issue 4

##
Approach to Translational Equilibrium in a Rigid Sphere Gas

journal, January 1963

- Bird, G. A.
- Physics of Fluids, Vol. 6, Issue 10

##
A general method for numerically simulating the stochastic time evolution of coupled chemical reactions

journal, December 1976

- Gillespie, Daniel T.
- Journal of Computational Physics, Vol. 22, Issue 4

##
Atomistic methods in fluid simulation

journal, April 2010

- Kadau, Kai; Barber, John L.; Germann, Timothy C.
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 368, Issue 1916

##
dsmcFoam+: An OpenFOAM based direct simulation Monte Carlo solver

journal, March 2018

- White, C.; Borg, M. K.; Scanlon, T. J.
- Computer Physics Communications, Vol. 224

##
Investigation of the ellipsoidal-statistical Bhatnagar–Gross–Krook kinetic model applied to gas-phase transport of heat and tangential momentum between parallel walls

journal, March 2011

- Gallis, M. A.; Torczynski, J. R.
- Physics of Fluids, Vol. 23, Issue 3

##
On the Instability of Superposed Fluids in a Gravitational Field.

journal, July 1955

- Layzer, David
- The Astrophysical Journal, Vol. 122

##
A parallel rendezvous algorithm for interpolation between multiple grids

journal, February 2004

- Plimpton, Steven J.; Hendrickson, Bruce; Stewart, James R.
- Journal of Parallel and Distributed Computing, Vol. 64, Issue 2

##
Review of theoretical modelling approaches of Rayleigh–Taylor instabilities and turbulent mixing

journal, April 2010

- Abarzhi, Snezhana I.
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 368, Issue 1916

##
Atomistic simulation of the Rayleigh-Taylor instability

journal, September 2006

- Barber, J. L.; Kadau, K.; Germann, T. C.
- Journal of Physics: Conference Series, Vol. 46