skip to main content

SciTech ConnectSciTech Connect

Title: Validation in the Absence of Observed Events

Here our paper addresses the problem of validating models in the absence of observed events, in the area of Weapons of Mass Destruction terrorism risk assessment. We address that problem with a broadened definition of “Validation,” based on “backing up” to the reason why modelers and decision makers seek validation, and from that basis re-define validation as testing how well the model can advise decision makers in terrorism risk management decisions. We develop that into two conditions: Validation must be based on cues available in the observable world; and it must focus on what can be done to affect that observable world, i.e. risk management. That in turn leads to two foci: 1.) the risk generating process, 2.) best use of available data. Based on our experience with nine WMD terrorism risk assessment models, we then describe three best use of available data pitfalls: SME confidence bias, lack of SME cross-referencing, and problematic initiation rates. Those two foci and three pitfalls provide a basis from which we define validation in this context in terms of four tests -- Does the model: … capture initiation? … capture the sequence of events by which attack scenarios unfold? … consider unanticipated scenarios? …more » consider alternative causal chains? Finally, we corroborate our approach against three key validation tests from the DOD literature: Is the model a correct representation of the simuland? To what degree are the model results comparable to the real world? Over what range of inputs are the model results useful?« less
 [1] ;  [2]
  1. Decision Strategies, LLC, New York, NY (United States)
  2. Old Dominion Univ., Norfolk, VA (United States). Virginia Modeling, Analysis and Simulation Center
Publication Date:
OSTI Identifier:
Report Number(s):
Journal ID: ISSN 0272-4332
Grant/Contract Number:
Accepted Manuscript
Journal Name:
Risk Analysis
Additional Journal Information:
Journal Volume: 36; Journal Issue: 4; Journal ID: ISSN 0272-4332
Research Org:
Lawrence Livermore National Laboratory (LLNL), Livermore, CA
Sponsoring Org:
Country of Publication:
United States
45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEFENSE Validation; WMD Terrorism Risk Assessment; WMD Terrorism Risk Management