Applicability Analysis of Validation Evidence for Biomedical Computational Models
Abstract
Computational modeling has the potential to revolutionize medicine the way it transformed engineering. However, despite decades of work, there has only been limited progress to successfully translate modeling research to patient care. One major difficulty which often occurs with biomedical computational models is an inability to perform validation in a setting that closely resembles how the model will be used. For example, for a biomedical model that makes in vivo clinically relevant predictions, direct validation of predictions may be impossible for ethical, technological, or financial reasons. Unavoidable limitations inherent to the validation process lead to challenges in evaluating the credibility of biomedical model predictions. Therefore, when evaluating biomedical models, it is critical to rigorously assess applicability, that is, the relevance of the computational model, and its validation evidence to the proposed context of use (COU). However, there are no well-established methods for assessing applicability. In this paper, we present a novel framework for performing applicability analysis and demonstrate its use with a medical device computational model. The framework provides a systematic, step-by-step method for breaking down the broad question of applicability into a series of focused questions, which may be addressed using supporting evidence and subject matter expertise. The frameworkmore »
- Authors:
-
- Food and Drug Administration (FDA), Silver Spring, MD (United States). Office of Science and Engineering Laboratories (OSEL). Center for Devices and Radiological Health (CDRH)
- Food and Drug Administration (FDA), Silver Spring, MD (United States). Office of Science and Engineering Laboratories (OSEL). Center for Devices and Radiological Health (CDRH)
- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
- Publication Date:
- Research Org.:
- Food and Drug Administration (FDA), Silver Spring, MD (United States); Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
- Sponsoring Org.:
- USDOE; Food and Drug Administration (FDA) (United States)
- OSTI Identifier:
- 1399491
- Report Number(s):
- SAND2017-2092J
Journal ID: ISSN 2377-2158; 651131
- Grant/Contract Number:
- AC04-94AL85000
- Resource Type:
- Journal Article: Accepted Manuscript
- Journal Name:
- Journal of Verification, Validation and Uncertainty Quantification
- Additional Journal Information:
- Journal Volume: 2; Journal Issue: 2; Journal ID: ISSN 2377-2158
- Publisher:
- ASME
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING; computer simulation; simulation; stents; biomedicine
Citation Formats
Pathmanathan, Pras, Gray, Richard A., Romero, Vicente J., and Morrison, Tina M. Applicability Analysis of Validation Evidence for Biomedical Computational Models. United States: N. p., 2017.
Web. doi:10.1115/1.4037671.
Pathmanathan, Pras, Gray, Richard A., Romero, Vicente J., & Morrison, Tina M. Applicability Analysis of Validation Evidence for Biomedical Computational Models. United States. https://doi.org/10.1115/1.4037671
Pathmanathan, Pras, Gray, Richard A., Romero, Vicente J., and Morrison, Tina M. 2017.
"Applicability Analysis of Validation Evidence for Biomedical Computational Models". United States. https://doi.org/10.1115/1.4037671. https://www.osti.gov/servlets/purl/1399491.
@article{osti_1399491,
title = {Applicability Analysis of Validation Evidence for Biomedical Computational Models},
author = {Pathmanathan, Pras and Gray, Richard A. and Romero, Vicente J. and Morrison, Tina M.},
abstractNote = {Computational modeling has the potential to revolutionize medicine the way it transformed engineering. However, despite decades of work, there has only been limited progress to successfully translate modeling research to patient care. One major difficulty which often occurs with biomedical computational models is an inability to perform validation in a setting that closely resembles how the model will be used. For example, for a biomedical model that makes in vivo clinically relevant predictions, direct validation of predictions may be impossible for ethical, technological, or financial reasons. Unavoidable limitations inherent to the validation process lead to challenges in evaluating the credibility of biomedical model predictions. Therefore, when evaluating biomedical models, it is critical to rigorously assess applicability, that is, the relevance of the computational model, and its validation evidence to the proposed context of use (COU). However, there are no well-established methods for assessing applicability. In this paper, we present a novel framework for performing applicability analysis and demonstrate its use with a medical device computational model. The framework provides a systematic, step-by-step method for breaking down the broad question of applicability into a series of focused questions, which may be addressed using supporting evidence and subject matter expertise. The framework can be used for model justification, model assessment, and validation planning. While motivated by biomedical models, it is relevant to a broad range of disciplines and underlying physics. Finally, the proposed applicability framework could help overcome some of the barriers inherent to validation of, and aid clinical implementation of, biomedical models.},
doi = {10.1115/1.4037671},
url = {https://www.osti.gov/biblio/1399491},
journal = {Journal of Verification, Validation and Uncertainty Quantification},
issn = {2377-2158},
number = 2,
volume = 2,
place = {United States},
year = {Thu Sep 07 00:00:00 EDT 2017},
month = {Thu Sep 07 00:00:00 EDT 2017}
}
Works referenced in this record:
In silico clinical trials: how computer simulation will transform the biomedical industry
journal, May 2016
- Viceconti, Marco; Henney, Adriano; Morley-Fletcher, Edwin
- International Journal of Clinical Trials, Vol. 3, Issue 2
Incorporation of stochastic engineering models as prior information in Bayesian medical device trials
journal, March 2017
- Haddad, Tarek; Himes, Adam; Thompson, Laura
- Journal of Biopharmaceutical Statistics, Vol. 27, Issue 6
Computational Medicine: Translating Models to Clinical Care
journal, October 2012
- Winslow, R. L.; Trayanova, N.; Geman, D.
- Science Translational Medicine, Vol. 4, Issue 158
Predictive Medicine: Computational Techniques in Therapeutic Decision-Making
journal, January 1999
- Taylor, Charles A.; Draney, Mary T.; Ku, Joy P.
- Computer Aided Surgery, Vol. 4, Issue 5
Ensuring reliability of safety-critical clinical applications of computational cardiac models
journal, January 2013
- Pathmanathan, Pras; Gray, Richard A.
- Frontiers in Physiology, Vol. 4
Defining predictive maturity for validated numerical simulations
journal, April 2010
- Hemez, François; Atamturktur, H. Sezer; Unal, Cetin
- Computers & Structures, Vol. 88, Issue 7-8
Risk-based verification, validation, and accreditation process
conference, April 2010
- Elele, James N.; Smith, Jeremy
- SPIE Defense, Security, and Sensing, SPIE Proceedings
Verification, validation, and predictive capability in computational engineering and physics
journal, September 2004
- Oberkampf, William L.; Trucano, Timothy G.; Hirsch, Charles
- Applied Mechanics Reviews, Vol. 57, Issue 5
Sandia Verification and Validation Challenge Problem: A PCMM-Based Approach to Assessing Prediction Credibility
journal, February 2016
- Beghini, Lauren L.; Hough, Patricia D.
- Journal of Verification, Validation and Uncertainty Quantification, Vol. 1, Issue 1
Calibration, validation, and sensitivity analysis: What's what
journal, October 2006
- Trucano, T. G.; Swiler, L. P.; Igusa, T.
- Reliability Engineering & System Safety, Vol. 91, Issue 10-11
Bayesian calibration of computer models
journal, August 2001
- Kennedy, Marc C.; O'Hagan, Anthony
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 63, Issue 3
Type X and Y Errors and Data & amp; Model Conditioning for Systematic Uncertainty in Model Calibration, Validation, and Extrapolation
conference, April 2008
- Romero, Vicente J.
- SAE World Congress & Exhibition, SAE Technical Paper Series
Design considerations for studies of the biomechanical environment of the femoropopliteal arteries
journal, September 2013
- Ansari, Farzana; Pack, Lindsay K.; Brooks, Steven S.
- Journal of Vascular Surgery, Vol. 58, Issue 3
Effects of Cyclic Loading on the Uniaxial Behavior of Nitinol
journal, December 2010
- Schlun, M.; Zipse, A.; Dreher, G.
- Journal of Materials Engineering and Performance, Vol. 20, Issue 4-5
Incorporation of stochastic engineering models as prior information in Bayesian medical device trials
text, January 2017
- Haddad, Tarek; Himes, Adam; Thompson, Laura
- Taylor & Francis
in silico Clinical Trials: How Computer Simulation will Transform the Biomedical Industry
text, January 2016
- Viceconti, Marco; Henney, Adriano; Morley-Fletcher, Edwin
- Avicenna Consortium
Works referencing / citing this record:
Off‐axis loads cause failure of the distal radius at lower magnitudes than axial loads: A side‐to‐side experimental study
journal, January 2020
- Johnston, James D.; McDonald, Matthew P.; Kontulainen, Saija A.
- Journal of Orthopaedic Research, Vol. 38, Issue 8
Patient-Specific Cardiovascular Computational Modeling: Diversity of Personalization and Challenges
journal, March 2018
- Gray, Richard A.; Pathmanathan, Pras
- Journal of Cardiovascular Translational Research, Vol. 11, Issue 2
Study on the Accuracy of Structural and FSI Heart Valves Simulations
journal, August 2018
- Luraghi, Giulia; Migliavacca, Francesco; Rodriguez Matas, Josè Fèlix
- Cardiovascular Engineering and Technology, Vol. 9, Issue 4
Advancing Regulatory Science With Computational Modeling for Medical Devices at the FDA's Office of Science and Engineering Laboratories
journal, September 2018
- Morrison, Tina M.; Pathmanathan, Pras; Adwan, Mariam
- Frontiers in Medicine, Vol. 5
Validation and Trustworthiness of Multiscale Models of Cardiac Electrophysiology
journal, February 2018
- Pathmanathan, Pras; Gray, Richard A.
- Frontiers in Physiology, Vol. 9
Functionalized Anatomical Models for Computational Life Sciences
journal, November 2018
- Neufeld, Esra; Lloyd, Bryn; Schneider, Beatrice
- Frontiers in Physiology, Vol. 9
Credibility Evidence for Computational Patient Models Used in the Development of Physiological Closed-Loop Controlled Devices for Critical Care Medicine
journal, March 2019
- Parvinian, Bahram; Pathmanathan, Pras; Daluwatte, Chathuri
- Frontiers in Physiology, Vol. 10
Comprehensive Uncertainty Quantification and Sensitivity Analysis for Cardiac Action Potential Models
journal, June 2019
- Pathmanathan, Pras; Cordeiro, Jonathan M.; Gray, Richard A.
- Frontiers in Physiology, Vol. 10
Functionalized Anatomical Models for Computational Life Sciences
text, January 2018
- Neufeld, Esra; Lloyd, Bryn; Schneider, Beatrice
- ETH Zurich
Consideration of a Credibility Assessment Framework in Model‐Informed Drug Development: Potential Application to Physiologically‐Based Pharmacokinetic Modeling and Simulation
journal, November 2019
- Kuemmel, Colleen; Yang, Yuching; Zhang, Xinyuan
- CPT: Pharmacometrics & Systems Pharmacology, Vol. 9, Issue 1
Advancing Regulatory Science With Computational Modeling for Medical Devices at the FDA's Office of Science and Engineering Laboratories
journal, September 2018
- Morrison, Tina M.; Pathmanathan, Pras; Adwan, Mariam
- Frontiers in Medicine, Vol. 5
Impact of Modeling Assumptions on Stability Predictions in Reverse Total Shoulder Arthroplasty
journal, August 2018
- Dharia, Mehul A.; Bischoff, Jeffrey E.; Schneider, David
- Frontiers in Physiology, Vol. 9
Functionalized Anatomical Models for Computational Life Sciences
journal, November 2018
- Neufeld, Esra; Lloyd, Bryn; Schneider, Beatrice
- Frontiers in Physiology, Vol. 9
Credibility Evidence for Computational Patient Models Used in the Development of Physiological Closed-Loop Controlled Devices for Critical Care Medicine
journal, March 2019
- Parvinian, Bahram; Pathmanathan, Pras; Daluwatte, Chathuri
- Frontiers in Physiology, Vol. 10