Validation in the Absence of Observed Events
Abstract
Here our paper addresses the problem of validating models in the absence of observed events, in the area of Weapons of Mass Destruction terrorism risk assessment. We address that problem with a broadened definition of “Validation,” based on “backing up” to the reason why modelers and decision makers seek validation, and from that basis re-define validation as testing how well the model can advise decision makers in terrorism risk management decisions. We develop that into two conditions: Validation must be based on cues available in the observable world; and it must focus on what can be done to affect that observable world, i.e. risk management. That in turn leads to two foci: 1.) the risk generating process, 2.) best use of available data. Based on our experience with nine WMD terrorism risk assessment models, we then describe three best use of available data pitfalls: SME confidence bias, lack of SME cross-referencing, and problematic initiation rates. Those two foci and three pitfalls provide a basis from which we define validation in this context in terms of four tests -- Does the model: … capture initiation? … capture the sequence of events by which attack scenarios unfold? … consider unanticipated scenarios? …more »
- Authors:
-
- Decision Strategies, LLC, New York, NY (United States)
- Old Dominion Univ., Norfolk, VA (United States). Virginia Modeling, Analysis and Simulation Center
- Publication Date:
- Research Org.:
- Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1262178
- Report Number(s):
- LLNL-JRNL-669713
Journal ID: ISSN 0272-4332
- Grant/Contract Number:
- AC52-07NA27344
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Risk Analysis
- Additional Journal Information:
- Journal Volume: 36; Journal Issue: 4; Journal ID: ISSN 0272-4332
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEFENSE; Validation; WMD Terrorism Risk Assessment; WMD Terrorism Risk Management
Citation Formats
Lathrop, John, and Ezell, Barry. Validation in the Absence of Observed Events. United States: N. p., 2015.
Web. doi:10.1111/risa.12442.
Lathrop, John, & Ezell, Barry. Validation in the Absence of Observed Events. United States. https://doi.org/10.1111/risa.12442
Lathrop, John, and Ezell, Barry. Wed .
"Validation in the Absence of Observed Events". United States. https://doi.org/10.1111/risa.12442. https://www.osti.gov/servlets/purl/1262178.
@article{osti_1262178,
title = {Validation in the Absence of Observed Events},
author = {Lathrop, John and Ezell, Barry},
abstractNote = {Here our paper addresses the problem of validating models in the absence of observed events, in the area of Weapons of Mass Destruction terrorism risk assessment. We address that problem with a broadened definition of “Validation,” based on “backing up” to the reason why modelers and decision makers seek validation, and from that basis re-define validation as testing how well the model can advise decision makers in terrorism risk management decisions. We develop that into two conditions: Validation must be based on cues available in the observable world; and it must focus on what can be done to affect that observable world, i.e. risk management. That in turn leads to two foci: 1.) the risk generating process, 2.) best use of available data. Based on our experience with nine WMD terrorism risk assessment models, we then describe three best use of available data pitfalls: SME confidence bias, lack of SME cross-referencing, and problematic initiation rates. Those two foci and three pitfalls provide a basis from which we define validation in this context in terms of four tests -- Does the model: … capture initiation? … capture the sequence of events by which attack scenarios unfold? … consider unanticipated scenarios? … consider alternative causal chains? Finally, we corroborate our approach against three key validation tests from the DOD literature: Is the model a correct representation of the simuland? To what degree are the model results comparable to the real world? Over what range of inputs are the model results useful?},
doi = {10.1111/risa.12442},
journal = {Risk Analysis},
number = 4,
volume = 36,
place = {United States},
year = {Wed Jul 22 00:00:00 EDT 2015},
month = {Wed Jul 22 00:00:00 EDT 2015}
}
Search WorldCat to find libraries that may hold this journalWeb of Science
Works referenced in this record:
Probabilistic Risk Analysis and Bioterrorism Risk
journal, March 2009
- Ezell, Barry Charles; Winterfeldt, Detlof von
- Biosecurity and Bioterrorism: Biodefense Strategy, Practice, and Science, Vol. 7, Issue 1
Response to Parnell, Smith, and Moxley, Intelligent Adversary Risk Analysis: A Bioterrorism Risk Management Model, Risk Analysis, Vol. 30, No. 1, 2010
journal, April 2010
- Ezell, Barry C.; Collins, Andrew J.
- Risk Analysis, Vol. 30, Issue 6
A Mathematical Model of the Risk of Nuclear Terrorism
journal, September 2006
- Bunn, Matthew
- The ANNALS of the American Academy of Political and Social Science, Vol. 607, Issue 1
Towards a practical method to validate decision support systems
journal, July 1998
- Borenstein, Denis
- Decision Support Systems, Vol. 23, Issue 3
Efforts and Programs of the Department of Defense Relating to the Prevention, Mitigation, and Treatment of Blast Injuries
report, October 2008
- Defense, Department of
Use of expert judgment in NUREG-1150
journal, May 1991
- Ortiz, N. R.; Wheeler, T. A.; Breeding, R. J.
- Nuclear Engineering and Design, Vol. 126, Issue 3
Validation of the lagrangian particle dispersion model FLEXPART against large-scale tracer experiment data
journal, December 1998
- Stohl, A.; Hittenberger, M.; Wotawa, G.
- Atmospheric Environment, Vol. 32, Issue 24
Modelling blast loads on buildings in complex city geometries
journal, October 2005
- Remennikov, Alex M.; Rose, Timothy A.
- Computers & Structures, Vol. 83, Issue 27