Direct simulation Monte Carlo on petaflop supercomputers and beyond

Plimpton, S. J.; Moore, S. G.; Borner, A.; Stagg, A. K.; Koehler, T. P.; Torczynski, J. R.; Gallis, M. A.

doi:10.1063/1.5108534

Title: Direct simulation Monte Carlo on petaflop supercomputers and beyond

Journal Article · Thu Aug 01 00:00:00 EDT 2019 · Physics of Fluids

DOI:https://doi.org/10.1063/1.5108534· OSTI ID:1559524

Plimpton, S. J. ^[1]; Moore, S. G. ^[1]; Borner, A. ^[2];

^[1]; Koehler, T. P. ^[1];

^[1];

^[1]

Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Science and Technology Corporation at NASA Ames Research Center, Moffett Field, California 94035, USA

The gold-standard definition of the Direct Simulation Monte Carlo (DSMC) method is given in the 1994 book by Bird [Molecular Gas Dynamics and the Direct Simulation of Gas Flows (Clarendon Press, Oxford, UK, 1994)], which refined his pioneering earlier papers in which he first formulated the method. In the intervening 25 years, DSMC has become the method of choice for modeling rarefied gas dynamics in a variety of scenarios. The chief concern to applying DSMC to more dense or even continuum flows is its computational expense compared to continuum computational fluid dynamics methods. The dramatic (nearly billion-fold) increase in speed of the largest supercomputers over the last 30 years has thus been a key enabling factor in using DSMC to model a richer variety of flows, due to the method’s inherent parallelism. We have developed the open-source SPARTA DSMC code with the goal of running DSMC efficiently on the largest machines, both current and future. It is largely an implementation of Bird’s 1994 formulation. In this work, we describe algorithms used in SPARTA to enable DSMC to operate in parallel at the scale of many billions of particles or grid cells, or with billions of surface elements. We highlight a few examples of the kinds of fundamental physics questions and engineering applications that DSMC can address at these scales.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: AC04-94AL85000; NA0003525

OSTI ID:: 1559524

Report Number(s):: SAND-2019-9819J; 678677; TRN: US2000361

Journal Information:: Physics of Fluids, Vol. 31, Issue 8; ISSN 1070-6631

Publisher:: American Institute of Physics (AIP)Copyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 97 works

Citation information provided by
Web of Science

References (42)

Application highlights of the DSMC Analysis Code (DAC) software for simulating rarefied flows LeBeau, G. J.; Lumpkin III, F. E. Computer Methods in Applied Mechanics and Engineering, Vol. 191, Issue 6-7 https://doi.org/10.1016/s0045-7825(01)00304-8	journal	December 2001
Robust cut-cell algorithms for DSMC implementations employing multi-level Cartesian grids Zhang, Chonglin; Schwartzentruber, Thomas E. Computers & Fluids, Vol. 69 https://doi.org/10.1016/j.compfluid.2012.08.013	journal	October 2012
An overview of Rayleigh-Taylor instability Sharp, D. H. Physica D: Nonlinear Phenomena, Vol. 12, Issue 1-3 https://doi.org/10.1016/0167-2789(84)90510-4	journal	July 1984
Kokkos: Enabling manycore performance portability through polymorphic memory access patterns Carter Edwards, H.; Trott, Christian R.; Sunderland, Daniel Journal of Parallel and Distributed Computing, Vol. 74, Issue 12 https://doi.org/10.1016/j.jpdc.2014.07.003	journal	December 2014
Efficient Implementation of Marching Cubes' Cases with Topological Guarantees Lewiner, Thomas; Lopes, Hélio; Vieira, Antônio Wilson Journal of Graphics Tools, Vol. 8, Issue 2 https://doi.org/10.1080/10867651.2003.10487582	journal	January 2003
Rayleigh–Taylor and Richtmyer–Meshkov instability induced flow, turbulence, and mixing. I Zhou, Ye Physics Reports, Vol. 720-722 https://doi.org/10.1016/j.physrep.2017.07.005	journal	December 2017
Rayleigh–Taylor and Richtmyer–Meshkov instability induced flow, turbulence, and mixing. II Zhou, Ye Physics Reports, Vol. 723-725 https://doi.org/10.1016/j.physrep.2017.07.008	journal	December 2017
PuMA: the Porous Microstructure Analysis software Ferguson, Joseph C.; Panerai, Francesco; Borner, Arnaud SoftwareX, Vol. 7 https://doi.org/10.1016/j.softx.2018.03.001	journal	January 2018
A new algorithm for Monte Carlo simulation of Ising spin systems Bortz, A. B.; Kalos, M. H.; Lebowitz, J. L. Journal of Computational Physics, Vol. 17, Issue 1 https://doi.org/10.1016/0021-9991(75)90060-1	journal	January 1975
High temperature permeability of fibrous materials using direct simulation Monte Carlo Borner, Arnaud; Panerai, Francesco; Mansour, Nagi N. International Journal of Heat and Mass Transfer, Vol. 106 https://doi.org/10.1016/j.ijheatmasstransfer.2016.10.113	journal	March 2017
Nonequilibrium flow through porous thermal protection materials, Part I: Numerical methods Stern, Eric C.; Poovathingal, Savio; Nompelis, Ioannis Journal of Computational Physics, Vol. 380 https://doi.org/10.1016/j.jcp.2017.09.011	journal	March 2019
The instability of liquid surfaces when accelerated in a direction perpendicular to their planes. II Lewis, D. J. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, Vol. 202, Issue 1068, p. 81-96 https://doi.org/10.1098/rspa.1950.0086	journal	June 1950
Hidden surface removal using polygon area sorting Weiler, Kevin; Atherton, Peter ACM SIGGRAPH Computer Graphics, Vol. 11, Issue 2 https://doi.org/10.1145/965141.563896	journal	August 1977
Effects of Diffusion on Interface Instability between Gases Duff, R. E.; Harlow, F. H.; Hirt, C. W. Physics of Fluids, Vol. 5, Issue 4 https://doi.org/10.1063/1.1706634	journal	January 1962
Statistical simulation of the transition between regular and mach reflection in steady flows Ivanov, M. S.; Gimelshein, S. F.; Markelov, G. N. Computers & Mathematics with Applications, Vol. 35, Issue 1-2 https://doi.org/10.1016/s0898-1221(97)00262-9	journal	January 1998
An open source, parallel DSMC code for rarefied gas flows in arbitrary geometries Scanlon, T. J.; Roohi, E.; White, C. Computers & Fluids, Vol. 39, Issue 10 https://doi.org/10.1016/j.compfluid.2010.07.014	journal	December 2010
A new Strouhal–Reynolds-number relationship for the circular cylinder in the range 47<Re<2×105 Fey, Uwe; König, Michael; Eckelmann, Helmut Physics of Fluids, Vol. 10, Issue 7 https://doi.org/10.1063/1.869675	journal	July 1998
Practical considerations on Marching Cubes 33 topological correctness Custodio, Lis; Etiene, Tiago; Pesco, Sinesio Computers & Graphics, Vol. 37, Issue 7 https://doi.org/10.1016/j.cag.2013.04.004	journal	November 2013
Turbulence: The Legacy of A. N. Kolmogorov Frisch, Uriel https://doi.org/10.1017/CBO9781139170666	book	January 1996
Direct simulation Monte Carlo investigation of the Richtmyer-Meshkov instability Gallis, M. A.; Koehler, T. P.; Torczynski, J. R. Physics of Fluids, Vol. 27, Issue 8 https://doi.org/10.1063/1.4928338	journal	August 2015
Direct Simulation Monte Carlo Simulations of Hypersonic Flows With Shock Interactions Moss, James N.; Bird, Graeme A. AIAA Journal, Vol. 43, Issue 12 https://doi.org/10.2514/1.12532	journal	December 2005
Exact stochastic simulation of coupled chemical reactions Gillespie, Daniel T. The Journal of Physical Chemistry, Vol. 81, Issue 25 https://doi.org/10.1021/j100540a008	journal	December 1977
Modeling the oxidation of low-density carbon fiber material based on micro-tomography Ferguson, Joseph C.; Panerai, Francesco; Lachaud, Jean Carbon, Vol. 96 https://doi.org/10.1016/j.carbon.2015.08.113	journal	January 2016
Gas-kinetic simulation of sustained turbulence in minimal Couette flow Gallis, M. A.; Torczynski, J. R.; Bitter, N. P. Physical Review Fluids, Vol. 3, Issue 7 https://doi.org/10.1103/physrevfluids.3.071402	journal	July 2018
An experimental assembly for precise measurement of thermal accommodation coefficients Trott, Wayne M.; Castañeda, Jaime N.; Torczynski, John R. Review of Scientific Instruments, Vol. 82, Issue 3 https://doi.org/10.1063/1.3571269	journal	March 2011
Nanohydrodynamics simulations: An atomistic view of the Rayleigh-Taylor instability Kadau, K.; Germann, T. C.; Hadjiconstantinou, N. G. Proceedings of the National Academy of Sciences, Vol. 101, Issue 16 https://doi.org/10.1073/pnas.0401228101	journal	April 2004
Marching cubes: A high resolution 3D surface construction algorithm Lorensen, William E.; Cline, Harvey E. ACM SIGGRAPH Computer Graphics, Vol. 21, Issue 4 https://doi.org/10.1145/37402.37422	journal	August 1987
Approach to Translational Equilibrium in a Rigid Sphere Gas Bird, G. A. Physics of Fluids, Vol. 6, Issue 10 https://doi.org/10.1063/1.1710976	journal	January 1963
The instability of liquid surfaces when accelerated in a direction perpendicular to their planes. I Taylor, Geoffrey Ingram Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, Vol. 201, Issue 1065, p. 192-196 https://doi.org/10.1098/rspa.1950.0052	journal	March 1950
A general method for numerically simulating the stochastic time evolution of coupled chemical reactions Gillespie, Daniel T. Journal of Computational Physics, Vol. 22, Issue 4 https://doi.org/10.1016/0021-9991(76)90041-3	journal	December 1976
Atomistic methods in fluid simulation Kadau, Kai; Barber, John L.; Germann, Timothy C. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 368, Issue 1916 https://doi.org/10.1098/rsta.2009.0218	journal	April 2010
dsmcFoam+: An OpenFOAM based direct simulation Monte Carlo solver White, C.; Borg, M. K.; Scanlon, T. J. Computer Physics Communications, Vol. 224 https://doi.org/10.1016/j.cpc.2017.09.030	journal	March 2018
Investigation of the ellipsoidal-statistical Bhatnagar–Gross–Krook kinetic model applied to gas-phase transport of heat and tangential momentum between parallel walls Gallis, M. A.; Torczynski, J. R. Physics of Fluids, Vol. 23, Issue 3 https://doi.org/10.1063/1.3558869	journal	March 2011
3D DSMC computations on a heterogeneous CPU-GPU cluster with a large number of GPUs Kashkovsky, Alexander PROCEEDINGS OF THE 29TH INTERNATIONAL SYMPOSIUM ON RAREFIED GAS DYNAMICS, AIP Conference Proceedings https://doi.org/10.1063/1.4902592	conference	January 2014
On the Instability of Superposed Fluids in a Gravitational Field. Layzer, David The Astrophysical Journal, Vol. 122 https://doi.org/10.1086/146048	journal	July 1955
A parallel rendezvous algorithm for interpolation between multiple grids Plimpton, Steven J.; Hendrickson, Bruce; Stewart, James R. Journal of Parallel and Distributed Computing, Vol. 64, Issue 2 https://doi.org/10.1016/j.jpdc.2003.11.006	journal	February 2004
Review of theoretical modelling approaches of Rayleigh–Taylor instabilities and turbulent mixing Abarzhi, Snezhana I. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 368, Issue 1916 https://doi.org/10.1098/rsta.2010.0020	journal	April 2010
Atomistic simulation of the Rayleigh-Taylor instability Barber, J. L.; Kadau, K.; Germann, T. C. Journal of Physics: Conference Series, Vol. 46 https://doi.org/10.1088/1742-6596/46/1/008	journal	September 2006
A Parallel Rendezvous Algorithm for Interpolation Between Multiple Grids Plimpton, S.; Hendrickson, B.; Stewart, J. SC98 - High Performance Networking and Computing Conference, Proceedings of the IEEE/ACM SC98 Conference https://doi.org/10.1109/sc.1998.10032	conference	January 1998
Marching cubes: A high resolution 3D surface construction algorithm Lorensen, William E.; Cline, Harvey E. Proceedings of the 14th annual conference on Computer graphics and interactive techniques - SIGGRAPH '87 https://doi.org/10.1145/37401.37422	conference	January 1987
Robust Cut-cell Algorithms for DSMC Implementations Employing Multi-level Cartesian Grids Zhang, Chonglin; Schwartzentruber, Thomas 42nd AIAA Thermophysics Conference https://doi.org/10.2514/6.2011-3314	conference	June 2011
Hidden surface removal using polygon area sorting Weiler, Kevin; Atherton, Peter Proceedings of the 4th annual conference on Computer graphics and interactive techniques - SIGGRAPH '77 https://doi.org/10.1145/563858.563896	conference	January 1977