skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Robust verification analysis

Abstract

We introduce a new methodology for inferring the accuracy of computational simulations through the practice of solution verification. We demonstrate this methodology on examples from computational heat transfer, fluid dynamics and radiation transport. Our methodology is suited to both well- and ill-behaved sequences of simulations. Our approach to the analysis of these sequences of simulations incorporates expert judgment into the process directly via a flexible optimization framework, and the application of robust statistics. The expert judgment is systematically applied as constraints to the analysis, and together with the robust statistics guards against over-emphasis on anomalous analysis results. We have named our methodology Robust Verification. Our methodology is based on utilizing multiple constrained optimization problems to solve the verification model in a manner that varies the analysis' underlying assumptions. Constraints applied in the analysis can include expert judgment regarding convergence rates (bounds and expectations) as well as bounding values for physical quantities (e.g., positivity of energy or density). This approach then produces a number of error models, which are then analyzed through robust statistical techniques (median instead of mean statistics). This provides self-contained, data and expert informed error estimation including uncertainties for both the solution itself and order of convergence. Ourmore » method produces high quality results for the well-behaved cases relatively consistent with existing practice. The methodology can also produce reliable results for ill-behaved circumstances predicated on appropriate expert judgment. We demonstrate the method and compare the results with standard approaches used for both code and solution verification on well-behaved and ill-behaved simulations.« less

Authors:
 [1];  [2];  [3];  [1]
  1. Sandia National Laboratories, Center for Computing Research, Albuquerque, NM 87185 (United States)
  2. Sandia National Laboratories, Verification and Validation, Uncertainty Quantification, Credibility Processes Department, Engineering Sciences Center, Albuquerque, NM 87185 (United States)
  3. Los Alamos National Laboratory, Methods and Algorithms Group, Computational Physics Division, Los Alamos, NM 87545 (United States)
Publication Date:
OSTI Identifier:
22570224
Resource Type:
Journal Article
Resource Relation:
Journal Name: Journal of Computational Physics; Journal Volume: 307; Other Information: Copyright (c) 2015 Elsevier Science B.V., Amsterdam, The Netherlands, All rights reserved.; Country of input: International Atomic Energy Agency (IAEA)
Country of Publication:
United States
Language:
English
Subject:
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS; CONVERGENCE; ERRORS; HEAT TRANSFER; MATHEMATICAL SOLUTIONS; OPTIMIZATION; RADIATION TRANSPORT; SIMULATION; STATISTICS; VERIFICATION

Citation Formats

Rider, William, E-mail: wjrider@sandia.gov, Witkowski, Walt, Kamm, James R., and Wildey, Tim. Robust verification analysis. United States: N. p., 2016. Web. doi:10.1016/J.JCP.2015.11.054.
Rider, William, E-mail: wjrider@sandia.gov, Witkowski, Walt, Kamm, James R., & Wildey, Tim. Robust verification analysis. United States. doi:10.1016/J.JCP.2015.11.054.
Rider, William, E-mail: wjrider@sandia.gov, Witkowski, Walt, Kamm, James R., and Wildey, Tim. 2016. "Robust verification analysis". United States. doi:10.1016/J.JCP.2015.11.054.
@article{osti_22570224,
title = {Robust verification analysis},
author = {Rider, William, E-mail: wjrider@sandia.gov and Witkowski, Walt and Kamm, James R. and Wildey, Tim},
abstractNote = {We introduce a new methodology for inferring the accuracy of computational simulations through the practice of solution verification. We demonstrate this methodology on examples from computational heat transfer, fluid dynamics and radiation transport. Our methodology is suited to both well- and ill-behaved sequences of simulations. Our approach to the analysis of these sequences of simulations incorporates expert judgment into the process directly via a flexible optimization framework, and the application of robust statistics. The expert judgment is systematically applied as constraints to the analysis, and together with the robust statistics guards against over-emphasis on anomalous analysis results. We have named our methodology Robust Verification. Our methodology is based on utilizing multiple constrained optimization problems to solve the verification model in a manner that varies the analysis' underlying assumptions. Constraints applied in the analysis can include expert judgment regarding convergence rates (bounds and expectations) as well as bounding values for physical quantities (e.g., positivity of energy or density). This approach then produces a number of error models, which are then analyzed through robust statistical techniques (median instead of mean statistics). This provides self-contained, data and expert informed error estimation including uncertainties for both the solution itself and order of convergence. Our method produces high quality results for the well-behaved cases relatively consistent with existing practice. The methodology can also produce reliable results for ill-behaved circumstances predicated on appropriate expert judgment. We demonstrate the method and compare the results with standard approaches used for both code and solution verification on well-behaved and ill-behaved simulations.},
doi = {10.1016/J.JCP.2015.11.054},
journal = {Journal of Computational Physics},
number = ,
volume = 307,
place = {United States},
year = 2016,
month = 2
}
  • Cited by 2
  • The complexity of ecological system requires investigators to deal with aggregates: individuals aggregated into populations or populations aggregated into communities or trophic levels. A number of theoretical studies have indicated the dangers of dealing with the dynamics of aggregated systems. These studies have focused on mathematical models so the results have been of marginal interest to the field ecologist. The present paper develops practical criteria which can be applied when the underlying model is unknown or of little immediate interest. Our approach has been to examine 40 alternative models and to state conclusions that are valid, no matter what themore » underlying dynamics of the system. Theoretical studies indicate that no error is introduced when two components with identical turnover times are aggregated. This study shows that, or certain problems, the components can be aggregated with < 10% error, even if the turnover rates differ by a factor of three. Criteria can be stated even more practically in terms of the ratio between the standing crops of the components. When measurement errors are considered explicity, the criteria depend upon the accuracy with which aggregated and disaggregated measures can be estimated.« less
  • The assumed [beta]-pdf is widely used in turbulent flame simulation with the moment closure method. However, implementation of the [beta]-pdf in turbulent flame simulation may sometimes encounter singularity difficulties in numerical calculation. The study proposes a robust [beta]-pdf treatment to overcome these numerical difficulties. The proposed [beta]-pdf treatment associated with the partial equilibrium chemistry model and the k-[epsilon] turbulence model is firstly applied to the case of H[sub 2]/N[sub 2]-air nonpremixed flame to demonstrate its robustness. Then, the present method is applied to the study of thermal NO formation in H[sub 2]-air nonpremixed flames. Comparison of results shows the presentmore » model to be capable of yielding fair agreement with the measurements. It is competitive with other more sophisticated methods, such as the pdf evolution method and the conditional moment closure method.« less
  • Robust regression using iteratively reweighted least squares is described and compared with traditional least-squares regression. Application of both techniques to a collection of actual data sets demonstrates that the performance of robust regression equals, and often exceeds, that of conventional least-square regression. The practical advantages of robust methods to the chemist are discussed.