Robust verification of stochastic simulation codes

Radtke, Gregg A.; Martin, Nevin; Moore, Christopher H.; Huang, Andy; Cartwright, Keith L.

doi:10.1016/j.jcp.2021.110855

Robust verification of stochastic simulation codes

Journal Article · Wed Nov 17 00:00:00 EST 2021 · Journal of Computational Physics

DOI:https://doi.org/10.1016/j.jcp.2021.110855· OSTI ID:1834336

Radtke, Gregg A. ^[1]; Martin, Nevin ^[1]; Moore, Christopher H. ^[1]; ^[1]; Cartwright, Keith L. ^[1]

Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

We introduce a robust verification tool for computational codes, which we call Stochastic Robust Extrapolation based Error Quantification (StREEQ). Unlike the prevalent Grid Convergence Index (GCI) [1] method, our approach is suitable for both stochastic and deterministic computational codes and is generalizable to any number of discretization variables. Building on ideas introduced in the Robust Verification [2] approach, we estimate the converged solution and orders of convergence with uncertainty using multiple fits of a discretization error model. In contrast to Robust Verification, we perform these fits to many bootstrap samples yielding a larger set of predictions with smoother statistics. Here, bootstrap resampling is performed on the lack-of-fit errors for deterministic code responses, and directly on the noisy data set for stochastic responses. This approach lends a degree of robustness to the overall results, capable of yielding precise verification results for sufficiently resolved data sets, and appropriately expanding the uncertainty when the data set does not support a precise result. For stochastic responses, a credibility assessment is also performed to give the analyst an indication of the trustworthiness of the results. Furthermore, this approach is suitable for both code and solution verification, and is particularly useful for solution verification of high-consequence simulations..

View Accepted Manuscript (DOE)

Research Organization:: Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: NA0003525

OSTI ID:: 1834336

Alternate ID(s):: OSTI ID: 1834343
OSTI ID: 1862496
OSTI ID: 23203574

Report Number(s):: SAND--2020-9723J; SAND--2021-11699J; 699567

Journal Information:: Journal of Computational Physics, Journal Name: Journal of Computational Physics Vol. 451; ISSN 0021-9991

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (16)

A convergence proof for Bird's direct simulation Monte Carlo method for the Boltzmann equation Wagner, Wolfgang Journal of Statistical Physics, Vol. 66, Issue 3-4 https://doi.org/10.1007/BF01055714	journal	February 1992
A theoretical derivation of the Condensed History Algorithm Larsen, Edward W. Annals of Nuclear Energy, Vol. 19, Issue 10-12 https://doi.org/10.1016/0306-4549(92)90013-2	journal	October 1992
The use of non-linear regression analysis and the F test for model discrimination with dose-response curves and ligand binding data Bardsley, William G.; McGinlay, Philip B. Journal of Theoretical Biology, Vol. 126, Issue 2 https://doi.org/10.1016/S0022-5193(87)80228-X	journal	May 1987
Review of code and solution verification procedures for computational simulation Roy, Christopher J. Journal of Computational Physics, Vol. 205, Issue 1 https://doi.org/10.1016/j.jcp.2004.10.036	journal	May 2005
Robust verification analysis Rider, William; Witkowski, Walt; Kamm, James R. Journal of Computational Physics, Vol. 307 https://doi.org/10.1016/j.jcp.2015.11.054	journal	February 2016
Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median Leys, Christophe; Ley, Christophe; Klein, Olivier Journal of Experimental Social Psychology, Vol. 49, Issue 4 https://doi.org/10.1016/j.jesp.2013.03.013	journal	July 2013
Direct simulation Monte Carlo convergence behavior of the hard-sphere-gas thermal conductivity for Fourier heat flow Rader, D. J.; Gallis, M. A.; Torczynski, J. R. Physics of Fluids, Vol. 18, Issue 7 https://doi.org/10.1063/1.2213640	journal	July 2006
Exact solutions of the Boltzmann equation Krook, Max; Wu, Tai Tsun Physics of Fluids, Vol. 20, Issue 10 https://doi.org/10.1063/1.861780	journal	January 1977
Robust Tests for the Equality of Variances Brown, Morton B.; Forsythe, Alan B. Journal of the American Statistical Association, Vol. 69, Issue 346 https://doi.org/10.1080/01621459.1974.10482955	journal	June 1974
A Simplex Method for Function Minimization Nelder, J. A.; Mead, R. The Computer Journal, Vol. 7, Issue 4 https://doi.org/10.1093/comjnl/7.4.308	journal	January 1965
Validation Test of Geant4 Simulation of Electron Backscattering Kim, Sung Hun; Pia, Maria Grazia; Basaglia, Tullio IEEE Transactions on Nuclear Science, Vol. 62, Issue 2 https://doi.org/10.1109/TNS.2015.2401055	journal	April 2015
Linear Inversion of Band-Limited Reflection Seismograms Santosa, Fadil; Symes, William W. SIAM Journal on Scientific and Statistical Computing, Vol. 7, Issue 4 https://doi.org/10.1137/0907087	journal	October 1986
The Convergence Rate of Finite Difference Schemes in the Presence of Shocks Engquist, Bjorn; Sjögreen, Björn SIAM Journal on Numerical Analysis, Vol. 35, Issue 6 https://doi.org/10.1137/S0036142997317584	journal	December 1998
Testing for Lack of Fit in Nonlinear Regression Neill, James W. The Annals of Statistics, Vol. 16, Issue 2 https://doi.org/10.1214/aos/1176350831	journal	June 1988
A Lack-of-Fit Test for Econometric Applications to Cross-Section Data Doran, Howard E.; Kmenta, Jan The Review of Economics and Statistics, Vol. 68, Issue 2 https://doi.org/10.2307/1925519	journal	May 1986
EMPIRE-PIC: A Performance Portable Unstructured Particle-in-Cell Code sci, global Communications in Computational Physics, Vol. 30, Issue 4 https://doi.org/10.4208/cicp.OA-2020-0261	journal	June 2021

Similar Records

Stochastic Richardson Extrapolation Based Numerical Error Estimation for Kinetic Plasma Simulations

Technical Report · Thu Oct 01 00:00:00 EDT 2015 · OSTI ID:1504853

Numerical Uncertainty Estimation for Stochastic Particle-in-Cell Simulations Applied to Verification and Validation.

Journal Article · Mon Nov 30 23:00:00 EST 2015 · OSTI ID:1427197

Code Verification of a Warm Electron Diode Using the EMPIRE Plasma Simulation Code.

Technical Report · Mon Jul 01 00:00:00 EDT 2019 · OSTI ID:1762913

Related Subjects

97 MATHEMATICS AND COMPUTING
Convergence
Error estimation
Monte Carlo methods
Plasma simulation
Solution verification
Verification

Robust verification of stochastic simulation codes

Citation Formats

References (16)

Similar Records

Related Subjects