skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Concepts of Model Verification and Validation

Abstract

Model verification and validation (V&V) is an enabling methodology for the development of computational models that can be used to make engineering predictions with quantified confidence. Model V&V procedures are needed by government and industry to reduce the time, cost, and risk associated with full-scale testing of products, materials, and weapon systems. Quantifying the confidence and predictive accuracy of model calculations provides the decision-maker with the information necessary for making high-consequence decisions. The development of guidelines and procedures for conducting a model V&V program are currently being defined by a broad spectrum of researchers. This report reviews the concepts involved in such a program. Model V&V is a current topic of great interest to both government and industry. In response to a ban on the production of new strategic weapons and nuclear testing, the Department of Energy (DOE) initiated the Science-Based Stockpile Stewardship Program (SSP). An objective of the SSP is to maintain a high level of confidence in the safety, reliability, and performance of the existing nuclear weapons stockpile in the absence of nuclear testing. This objective has challenged the national laboratories to develop high-confidence tools and methods that can be used to provide credible models needed for stockpilemore » certification via numerical simulation. There has been a significant increase in activity recently to define V&V methods and procedures. The U.S. Department of Defense (DoD) Modeling and Simulation Office (DMSO) is working to develop fundamental concepts and terminology for V&V applied to high-level systems such as ballistic missile defense and battle management simulations. The American Society of Mechanical Engineers (ASME) has recently formed a Standards Committee for the development of V&V procedures for computational solid mechanics models. The Defense Nuclear Facilities Safety Board (DNFSB) has been a proponent of model V&V for all safety-related nuclear facility design, analyses, and operations. In fact, DNFSB 2002-1 recommends to the DOE and National Nuclear Security Administration (NNSA) that a V&V process be performed for all safety related software and analysis. Model verification and validation are the primary processes for quantifying and building credibility in numerical models. Verification is the process of determining that a model implementation accurately represents the developer's conceptual description of the model and its solution. Validation is the process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model. Both verification and validation are processes that accumulate evidence of a model's correctness or accuracy for a specific scenario; thus, V&V cannot prove that a model is correct and accurate for all possible scenarios, but, rather, it can provide evidence that the model is sufficiently accurate for its intended use. Model V&V is fundamentally different from software V&V. Code developers developing computer programs perform software V&V to ensure code correctness, reliability, and robustness. In model V&V, the end product is a predictive model based on fundamental physics of the problem being solved. In all applications of practical interest, the calculations involved in obtaining solutions with the model require a computer code, e.g., finite element or finite difference analysis. Therefore, engineers seeking to develop credible predictive models critically need model V&V guidelines and procedures. The expected outcome of the model V&V process is the quantified level of agreement between experimental data and model prediction, as well as the predictive accuracy of the model. This report attempts to describe the general philosophy, definitions, concepts, and processes for conducting a successful V&V program. This objective is motivated by the need for highly accurate numerical models for making predictions to support the SSP, and also by the lack of guidelines, standards and procedures for performing V&V for complex numerical models.« less

Authors:
; ; ; ; ;
Publication Date:
Research Org.:
Los Alamos National Lab., Los Alamos, NM (US)
Sponsoring Org.:
US DOE (US)
OSTI Identifier:
835920
Report Number(s):
LA-14167
TRN: US0500431
DOE Contract Number:
W-7405-ENG-36
Resource Type:
Technical Report
Resource Relation:
Other Information: PBD: 30 Oct 2004
Country of Publication:
United States
Language:
English
Subject:
45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEFENSE; 11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS; ACCURACY; BALLISTIC MISSILE DEFENSE; COMPUTER CODES; ENGINEERS; FORECASTING; IMPLEMENTATION; NUCLEAR FACILITIES; NUCLEAR WEAPONS; PHYSICS; RELIABILITY; SAFETY; SECURITY; SIMULATION; STOCKPILES; TESTING; US DOD; VALIDATION; VERIFICATION; WEAPONS; 99 GENERAL AND MISCELLANEOUS//MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE

Citation Formats

B.H.Thacker, S.W.Doebling, F.M.Hemez, M.C. Anderson, J.E. Pepin, and E.A. Rodriguez. Concepts of Model Verification and Validation. United States: N. p., 2004. Web. doi:10.2172/835920.
B.H.Thacker, S.W.Doebling, F.M.Hemez, M.C. Anderson, J.E. Pepin, & E.A. Rodriguez. Concepts of Model Verification and Validation. United States. doi:10.2172/835920.
B.H.Thacker, S.W.Doebling, F.M.Hemez, M.C. Anderson, J.E. Pepin, and E.A. Rodriguez. Sat . "Concepts of Model Verification and Validation". United States. doi:10.2172/835920. https://www.osti.gov/servlets/purl/835920.
@article{osti_835920,
title = {Concepts of Model Verification and Validation},
author = {B.H.Thacker and S.W.Doebling and F.M.Hemez and M.C. Anderson and J.E. Pepin and E.A. Rodriguez},
abstractNote = {Model verification and validation (V&V) is an enabling methodology for the development of computational models that can be used to make engineering predictions with quantified confidence. Model V&V procedures are needed by government and industry to reduce the time, cost, and risk associated with full-scale testing of products, materials, and weapon systems. Quantifying the confidence and predictive accuracy of model calculations provides the decision-maker with the information necessary for making high-consequence decisions. The development of guidelines and procedures for conducting a model V&V program are currently being defined by a broad spectrum of researchers. This report reviews the concepts involved in such a program. Model V&V is a current topic of great interest to both government and industry. In response to a ban on the production of new strategic weapons and nuclear testing, the Department of Energy (DOE) initiated the Science-Based Stockpile Stewardship Program (SSP). An objective of the SSP is to maintain a high level of confidence in the safety, reliability, and performance of the existing nuclear weapons stockpile in the absence of nuclear testing. This objective has challenged the national laboratories to develop high-confidence tools and methods that can be used to provide credible models needed for stockpile certification via numerical simulation. There has been a significant increase in activity recently to define V&V methods and procedures. The U.S. Department of Defense (DoD) Modeling and Simulation Office (DMSO) is working to develop fundamental concepts and terminology for V&V applied to high-level systems such as ballistic missile defense and battle management simulations. The American Society of Mechanical Engineers (ASME) has recently formed a Standards Committee for the development of V&V procedures for computational solid mechanics models. The Defense Nuclear Facilities Safety Board (DNFSB) has been a proponent of model V&V for all safety-related nuclear facility design, analyses, and operations. In fact, DNFSB 2002-1 recommends to the DOE and National Nuclear Security Administration (NNSA) that a V&V process be performed for all safety related software and analysis. Model verification and validation are the primary processes for quantifying and building credibility in numerical models. Verification is the process of determining that a model implementation accurately represents the developer's conceptual description of the model and its solution. Validation is the process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model. Both verification and validation are processes that accumulate evidence of a model's correctness or accuracy for a specific scenario; thus, V&V cannot prove that a model is correct and accurate for all possible scenarios, but, rather, it can provide evidence that the model is sufficiently accurate for its intended use. Model V&V is fundamentally different from software V&V. Code developers developing computer programs perform software V&V to ensure code correctness, reliability, and robustness. In model V&V, the end product is a predictive model based on fundamental physics of the problem being solved. In all applications of practical interest, the calculations involved in obtaining solutions with the model require a computer code, e.g., finite element or finite difference analysis. Therefore, engineers seeking to develop credible predictive models critically need model V&V guidelines and procedures. The expected outcome of the model V&V process is the quantified level of agreement between experimental data and model prediction, as well as the predictive accuracy of the model. This report attempts to describe the general philosophy, definitions, concepts, and processes for conducting a successful V&V program. This objective is motivated by the need for highly accurate numerical models for making predictions to support the SSP, and also by the lack of guidelines, standards and procedures for performing V&V for complex numerical models.},
doi = {10.2172/835920},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Sat Oct 30 00:00:00 EDT 2004},
month = {Sat Oct 30 00:00:00 EDT 2004}
}

Technical Report:

Save / Share:
  • This report describes the methodology and results of independent verification and validation performed on a combat model in its design stage. The combat model is the Future Theater Level Model (FTLM), under development by The Joint Staff/J-8. J-8 has undertaken its development to provide an analysis tool that addresses the uncertainties of combat more directly than previous models and yields more rapid study results. The methodology adopted for this verification and validation consisted of document analyses. Included were detailed examination of the FTLM design documents (at all stages of development), the FTLM Mission Needs Statement, and selected documentation for othermore » theater level combat models. These documents were compared to assess the FTLM as to its design stage, its purpose as an analytical combat model, and its capabilities as specified in the Mission Needs Statement. The conceptual design passed those tests. The recommendations included specific modifications as well as a recommendation for continued development. The methodology is significant because independent verification and validation have not been previously reported as being performed on a combat model in its design stage. The results are significant because The Joint Staff/J-8 will be using the recommendations from this study in determining whether to proceed with develop of the model.« less
  • This study details the verification and validation (V and V) of the Comprehensive Operational Support Evaluation Model for Space (COSEMS). COSEMS is an Ada-based simulation which models spacecraft constellation support concepts such as support from the ground and on-orbit support. While the model is intended for use in analyzing Strategic Defense System concepts, it can easily evaluate non-military satellite constellations. The VV was confined to a subset of the over 200 subprograms which comprise COSEMS. This subset covered random number generation, reliability, orbital mechanics, and mission planning functions. The study used traces and comparison to other models to perform themore » VV. An input/output analysis was also performed to ascertain the ease of use of COSEMS and the utility of its output. The analysis showed that the areas under investigation performed according to the model and that the model approximated real-world behavior except for orbital motion. The part of the model governing orbital perturbations due to the non-spherical earth omitted rotation of the line-of-apsides. The analysis also revealed that the Ada code and the input/output format are highly machine dependent, which restricts the program from coming into widespread use and limits the usefulness of the output.« less
  • Recent modifications to the EQ3/6 geochemical modeling software package provide a new option which can be used to compute the activity of water and the activity coefficients of solute species in both brines and dilute solutions. This option is based on equations proposed by Pitzer (1973) which allow approximation of mean molal activity coefficients (/gamma//sub /plus minus//) and osmotic coefficients (/phi/) up to high ionic strengths, and which, together with an appropriate ion-splitting convention, also afford calculation of individual ion activity coefficients (/gamma//sub i/) in brines. Values of /gamma//sub /plus minus// and /phi/ generated by EQ3/6 for binary and compositionallymore » higher order systems compare favorably with their experimental and hand-calculated counterparts. The addition of Pitzer's equations to EQ3/6 as an optional method for calculating activity coefficients represents a significant improvement over the previous versions of the codes, which were limited to the use of a simple extended form of the Debye-Hueckel equation (the B-dot equation of Helgeson, 1969). Test runs using EQ3/6 to calculate solubility limits in simple binary and ternary systems (NaCl-H/sub 2/O, KCl-H/sub 2/O, and KCl-NaCl-H/sub 2/O) confirm the capability of the codes to accurately predict geochemical equilibria between brines and evaporite minerals in the sample systems. The test runs were sufficiently comprehensive to verify the accuracy of the calculational procedures and to partially validate the capability of the codes to dependably model the geochemical behavior of aqueous electrolyte solutions having ionic strengths as high as /approximately/6 m, subject to the availability of the requisite thermodynamic data and activity coefficient parameters. 24 refs., 6 figs., 8 tabs.« less
  • This document is the verification and validation final report for the Decision Analysis Model for Assessment of Tank Waste Remediation System Waste Treatment Strategies. This model is also known as the INSIGHT Model.
  • The concepts of Verification and Validation (V&V) can be oversimplified in a succinct manner by saying that 'verification is doing things right' and 'validation is doing the right thing'. In the world of the Finite Element Method (FEM) and computational analysis, it is sometimes said that 'verification means solving the equations right' and 'validation means solving the right equations'. In other words, if one intends to give an answer to the equation '2+2=', then one must run the resulting code to assure that the answer '4' results. However, if the nature of the physics or engineering problem being addressed withmore » this code is multiplicative rather than additive, then even though Verification may succeed (2+2=4 etc), Validation may fail because the equations coded are not those needed to address the real world (multiplicative) problem. We have previously provided a 4-step 'ABCD' quantitative implementation for a quantitative V&V process: (A) Plan the analyses and validation testing that may be needed along the way. Assure that the code[s] chosen have sufficient documentation of software quality and Code Verification (i.e., does 2+2=4?). Perform some calibration analyses and calibration based sensitivity studies (these are not validated sensitivities but are useful for planning purposes). Outline the data and validation analyses that will be needed to turn the calibrated model (and calibrated sensitivities) into validated quantities. (B) Solution Verification: For the system or component being modeled, quantify the uncertainty and error estimates due to spatial, temporal, and iterative discretization during solution. (C) Validation over the data domain: Perform a quantitative validation to provide confidence-bounded uncertainties on the quantity of interest over the domain of available data. (D) Predictive Adequacy: Extend the model validation process of 'C' out to the application domain of interest, which may be outside the domain of available data in one or more planes of multi-dimensional space. Part 'D' should provide the numerical information about the model and its predictive capability such that given a requirement, an adequacy assessment can be made to determine of more validation analyses or data are needed.« less