Characterization and Valuation of the Uncertainty of Calibrated Parameters in Microsimulation Decision Models

Alarid-Escudero, Fernando; Knudsen, Amy B.; Ozik, Jonathan; Collier, Nicholson; Kuntz, Karen M.

doi:10.3389/fphys.2022.780917

Characterization and Valuation of the Uncertainty of Calibrated Parameters in Microsimulation Decision Models

Journal Article · Mon May 09 00:00:00 EDT 2022 · Frontiers in Physiology

DOI:https://doi.org/10.3389/fphys.2022.780917· OSTI ID:2469484

Alarid-Escudero, Fernando ^[1]; Knudsen, Amy B. ^[2]; Ozik, Jonathan ^[3]; Collier, Nicholson ^[3]; Kuntz, Karen M. ^[4]

Center for Research and Teaching in Economics (CIDE) (Mexico)
Massachusetts General Hospital, Boston, MA (United States)
Argonne National Laboratory (ANL), Argonne, IL (United States); Univ. of Chicago, IL (United States)
Univ. of Minnesota, Minneapolis, MN (United States). School of Public Health

We evaluated the implications of different approaches to characterize the uncertainty of calibrated parameters of microsimulation decision models (DMs) and quantified the value of such uncertainty in decision making. We calibrated the natural history model of CRC to simulated epidemiological data with different degrees of uncertainty and obtained the joint posterior distribution of the parameters using a Bayesian approach. We conducted a probabilistic sensitivity analysis (PSA) on all the model parameters with different characterizations of the uncertainty of the calibrated parameters. We estimated the value of uncertainty of the various characterizations with a value of information analysis. We conducted all analyses using high-performance computing resources running the Extreme-scale Model Exploration with Swift (EMEWS) framework. The posterior distribution had a high correlation among some parameters. The parameters of the Weibull hazard function for the age of onset of adenomas had the highest posterior correlation of -0.958. When comparing full posterior distributions and the maximum-a-posteriori estimate of the calibrated parameters, there is little difference in the spread of the distribution of the CEA outcomes with a similar expected value of perfect information (EVPI) of $$\$$$$653 and $$\$$$$685, respectively, at a willingness-to-pay (WTP) threshold of $$\$$$$66,000 per quality-adjusted life year (QALY). Ignoring correlation on the calibrated parameters’ posterior distribution produced the broadest distribution of CEA outcomes and the highest EVPI of $$\$$$$809 at the same WTP threshold. Different characterizations of the uncertainty of calibrated parameters affect the expected value of eliminating parametric uncertainty on the CEA. Ignoring inherent correlation among calibrated parameters on a PSA overestimates the value of uncertainty.

View Accepted Manuscript (DOE)

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: AC02-06CH11357

OSTI ID:: 2469484

Journal Information:: Frontiers in Physiology, Journal Name: Frontiers in Physiology Vol. 13; ISSN 1664-042X

Publisher:: Frontiers Media S.A.Copyright Statement

Country of Publication:: United States

Language:: English

References (46)

Cost-effectiveness of Screening for Colorectal Cancer in the General Population Frazier, A. Lindsay JAMA, Vol. 284, Issue 15 https://doi.org/10.1001/jama.284.15.1954	journal	October 2000
A hierarchical non‐homogenous Poisson model for meta‐analysis of adenoma counts Rutter, Carolyn M.; Yu, Onchee; Miglioretti, Diana L. Statistics in Medicine, Vol. 26, Issue 1 https://doi.org/10.1002/sim.2460	journal	December 2005
Statistical and Computational Inverse Problems Kaipio, Jari P.; Somersalo, Erkki Applied Mathematical Sciences https://doi.org/10.1007/b138659	book	January 2005
Bayesian calibration, validation, and uncertainty quantification of diffuse interface models of tumor growth Hawkins-Daarud, Andrea; Prudhomme, Serge; van der Zee, Kristoffer G. Journal of Mathematical Biology, Vol. 67, Issue 6-7 https://doi.org/10.1007/s00285-012-0595-9	journal	October 2012
Artificial neural network assisted Bayesian calibration of climate models Hauser, Tristan; Keats, Andrew; Tarasov, Lev Climate Dynamics, Vol. 39, Issue 1-2 https://doi.org/10.1007/s00382-011-1168-0	journal	September 2011
Bayesian Methods for Calibrating Health Policy Models: A Tutorial Menzies, Nicolas A.; Soeteman, Djøra I.; Pandya, Ankur PharmacoEconomics, Vol. 35, Issue 6 https://doi.org/10.1007/s40273-017-0494-4	journal	February 2017
Bayesian analysis of computer code outputs: A tutorial O’Hagan, A. Reliability Engineering & System Safety, Vol. 91, Issue 10-11 https://doi.org/10.1016/j.ress.2005.11.025	journal	October 2006
Designs for Computer Experiments Sacks, Jerome; Schiller, Susannah B.; Welch, William J. Technometrics, Vol. 31, Issue 1 https://doi.org/10.1080/00401706.1989.10488474	journal	February 1989
Calibration of Stochastic Computer Simulators Using Likelihood Emulation Oakley, Jeremy E.; Youngman, Benjamin D. Technometrics, Vol. 59, Issue 1 https://doi.org/10.1080/00401706.2015.1125391	journal	January 2017
Multiparameter Calibration of a Natural History Model of Cervical Cancer Kim, J. J.; Kuntz, K. M.; Stout, N. K. American Journal of Epidemiology, Vol. 166, Issue 2 https://doi.org/10.1093/aje/kwm086	journal	June 2007
Bayesian inference for the uncertainty distribution of computer model outputs Oakley, J. Biometrika, Vol. 89, Issue 4 https://doi.org/10.1093/biomet/89.4.769	journal	December 2002
A Simplex Method for Function Minimization Nelder, J. A.; Mead, R. The Computer Journal, Vol. 7, Issue 4 https://doi.org/10.1093/comjnl/7.4.308	journal	January 1965
Using Multiple Outcomes of Sexual Behavior to Provide Insights Into Chlamydia Transmission and the Effectiveness of Prevention Interventions in Adolescents Enns, Eva Andrea; Kao, Szu-Yu; Kozhimannil, Katy Backes Sexually Transmitted Diseases, Vol. 44, Issue 10 https://doi.org/10.1097/olq.0000000000000653	journal	October 2017
Extreme-Scale Dynamic Exploration of a Distributed Agent-Based Model With the EMEWS Framework Ozik, Jonathan; Collier, Nicholson T.; Wozniak, Justin M. IEEE Transactions on Computational Social Systems, Vol. 5, Issue 3 https://doi.org/10.1109/tcss.2018.2859189	journal	September 2018
Bayesian calibration of computer models Kennedy, Marc C.; O'Hagan, Anthony Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 63, Issue 3 https://doi.org/10.1111/1467-9868.00294	journal	August 2001
Probabilistic sensitivity analysis of complex models: a Bayesian approach Oakley, Jeremy E.; O'Hagan, Anthony Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 66, Issue 3 https://doi.org/10.1111/j.1467-9868.2004.05304.x	journal	August 2004
Expected Value of Perfect Information: An Empirical Example of Reducing Decision Uncertainty by Conducting Additional Research Oostenbrink, Jan B.; Al, Maiwenn J.; Oppe, Mark Value in Health, Vol. 11, Issue 7 https://doi.org/10.1111/j.1524-4733.2008.00389.x	journal	December 2008
Calibration of Disease Simulation Model Using an Engineering Approach Kong, Chung Yin; McMahon, Pamela M.; Gazelle, G. Scott Value in Health, Vol. 12, Issue 4 https://doi.org/10.1111/j.1524-4733.2008.00484.x	journal	June 2009
Utility Valuations for Outcome States of Colorectal Cancer Ness, Reid M.; Holmes, Ann M.; Klein, Robert American Journal of Gastroenterology, Vol. 94, Issue 6 https://doi.org/10.1111/j.1572-0241.1999.01157.x	journal	June 1999
Polyp Miss Rate Determined by Tandem Colonoscopy: A Systematic Review van Rijn, Jeroen C.; Reitsma, Johannes B.; Stoker, Jaap The American Journal of Gastroenterology, Vol. 101, Issue 2 https://doi.org/10.1111/j.1572-0241.2006.00390.x	journal	February 2006
Probabilistic Analysis of Cost-Effectiveness Models: Choosing between Treatment Strategies for Gastroesophageal Reflux Disease Briggs, A. H.; Goeree, R.; Blackhouse, G. Medical Decision Making, Vol. 22, Issue 4 https://doi.org/10.1177/027298902400448867	journal	July 2002
Estimation of Markov Chain Transition Probabilities and Rates from Fully and Partially Observed Data: Uncertainty Propagation, Evidence Synthesis, and Model Calibration Welton, Nicky J.; Ades, A. E. Medical Decision Making, Vol. 25, Issue 6 https://doi.org/10.1177/0272989x05282637	journal	November 2005
Dynamic Microsimulation Models for Health Outcomes: A Review Rutter, Carolyn M.; Zaslavsky, Alan M.; Feuer, Eric J. Medical Decision Making, Vol. 31, Issue 1 https://doi.org/10.1177/0272989x10369005	journal	May 2010
Bayesian Calibration of a Natural History Model with Application to a Population Model for Colorectal Cancer Whyte, Sophie; Walsh, Cathal; Chilcott, Jim Medical Decision Making, Vol. 31, Issue 4 https://doi.org/10.1177/0272989x10384738	journal	December 2010
A Systematic Comparison of Microsimulation Models of Colorectal Cancer Kuntz, Karen M.; Lansdorp-Vogelaar, Iris; Rutter, Carolyn M. Medical Decision Making, Vol. 31, Issue 4 https://doi.org/10.1177/0272989x11408730	journal	June 2011
Model Parameter Estimation and Uncertainty Analysis: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group–6 Briggs, Andrew H.; Weinstein, Milton C.; Fenwick, Elisabeth A. L. Medical Decision Making, Vol. 32, Issue 5 https://doi.org/10.1177/0272989x12458348	journal	September 2012
Microsimulation Modeling for Health Decision Sciences Using R: A Tutorial Krijkamp, Eline M.; Alarid-Escudero, Fernando; Enns, Eva A. Medical Decision Making, Vol. 38, Issue 3 https://doi.org/10.1177/0272989x18754513	journal	March 2018
Nonidentifiability in Model Calibration and Implications for Medical Decision Making Alarid-Escudero, Fernando; MacLehose, Richard F.; Peralta, Yadira Medical Decision Making, Vol. 38, Issue 7 https://doi.org/10.1177/0272989x18792283	journal	September 2018
Methods for Model Calibration under High Uncertainty: Modeling Cholera in Bangladesh Ryckman, Theresa; Luby, Stephen; Owens, Douglas K. Medical Decision Making, Vol. 40, Issue 5 https://doi.org/10.1177/0272989x20938683	journal	July 2020
Cost-effectiveness analysis of colorectal cancer screening with stool DNA testing in intermediate-incidence countries Wu, Grace Hui-Min; Wang, Yi-Ming; Yen, Amy Ming-Fang BMC Cancer, Vol. 6, Issue 1 https://doi.org/10.1186/1471-2407-6-136	journal	May 2006
Sequentially calibrating a Bayesian microsimulation model to incorporate new information and assumptions DeYoreo, Maria; Rutter, Carolyn M.; Ozik, Jonathan BMC Medical Informatics and Decision Making, Vol. 22, Issue 1 https://doi.org/10.1186/s12911-021-01726-0	journal	January 2022
Computing Normalizing Constants for Finite Mixture Models via Incremental Mixture Importance Sampling (IMIS) Steele, Russell J.; Raftery, Adrian E.; Emond, Mary J. Journal of Computational and Graphical Statistics, Vol. 15, Issue 3 https://doi.org/10.1198/106186006x132358	journal	September 2006
Bayesian Calibration of Microsimulation Models Rutter, Carolyn M.; Miglioretti, Diana L.; Savarino, James E. Journal of the American Statistical Association, Vol. 104, Issue 488 https://doi.org/10.1198/jasa.2009.ap07466	journal	December 2009
On Model Expansion, Model Contraction, Identifiability and Prior Information: Two Illustrative Scenarios Involving Mismeasured Variables Gustafson, Paul Statistical Science, Vol. 20, Issue 2 https://doi.org/10.1214/088342305000000098	journal	May 2005
Microsimulation model calibration using incremental mixture approximate Bayesian computation Rutter, Carolyn M.; Ozik, Jonathan; DeYoreo, Maria The Annals of Applied Statistics, Vol. 13, Issue 4 https://doi.org/10.1214/19-aoas1279	journal	December 2019
Design and Analysis of Computer Experiments Sacks, Jerome; Welch, William J.; Mitchell, Toby J. Statistical Science, Vol. 4, Issue 4 https://doi.org/10.1214/ss/1177012413	journal	November 1989
A theoretical foundation for state-transition cohort models in health decision analysis Iskandar, Rowan PLOS ONE, Vol. 13, Issue 12 https://doi.org/10.1371/journal.pone.0205543	journal	December 2018
Calibration Methods Used in Cancer Simulation Models and Suggested Reporting Guidelines Stout, Natasha K.; Knudsen, Amy B.; Kong, Chung Yin PharmacoEconomics, Vol. 27, Issue 7 https://doi.org/10.2165/11314830-000000000-00000	journal	July 2009
The Value of Value of Information Eckermann, Simon; Karnon, Jon; Willan, Andrew R. PharmacoEconomics, Vol. 28, Issue 9 https://doi.org/10.2165/11537370-000000000-00000	journal	September 2010
Incorporating Calibrated Model Parameters into Sensitivity Analyses Taylor, Douglas C. A.; Pawar, Vivek; Kruzikas, Denise T. PharmacoEconomics, Vol. 30, Issue 2 https://doi.org/10.2165/11593360-000000000-00000	journal	February 2012
Nonidentifiability in Model Calibration and Implications for Medical Decision Making Alarid-Escudero, Fernando; MacLehose, Richard F.; Peralta, Yadira Figshare https://doi.org/10.25384/sage.c.4243478	collection	January 2018
A review and critique of modelling in prioritising and designing screening programmes Karnon, J.; Goyder, E.; Tappenden, P. Health Technology Assessment, Vol. 11, Issue 52 https://doi.org/10.3310/hta11520	journal	December 2007
BayCANN: Streamlining Bayesian Calibration With Artificial Neural Network Metamodeling Jalal, Hawre; Trikalinos, Thomas A.; Alarid-Escudero, Fernando Frontiers in Physiology, Vol. 12 https://doi.org/10.3389/fphys.2021.662314	journal	May 2021
Sequentially calibrating a Bayesian microsimulation model to incorporate new information and assumptions DeYoreo, Maria; Rutter, Carolyn M.; Ozik, Jonathan figshare https://doi.org/10.6084/m9.figshare.c.5790862	collection	January 2022
Cost-Effectiveness of Human Papillomavirus Vaccination and Cervical Cancer Screening in Women Older Than 30 Years in the United States Kim, Jane J. Annals of Internal Medicine, Vol. 151, Issue 8 https://doi.org/10.7326/0003-4819-151-8-200910200-00007	journal	October 2009
Prevalence of Advanced Colorectal Neoplasia in White and Black Patients Undergoing Screening Colonoscopy in a Safety-Net Hospital Schroy, Paul C.; Coe, Alison; Chen, Clara A. Annals of Internal Medicine, Vol. 159, Issue 1 https://doi.org/10.7326/0003-4819-159-1-201307020-00004	journal	July 2013

Similar Records

Microsimulation model calibration using incremental mixture approximate Bayesian computation

Journal Article · Sat Nov 30 23:00:00 EST 2019 · The Annals of Applied Statistics · OSTI ID:1607644

Sequentially calibrating a Bayesian microsimulation model to incorporate new information and assumptions

Journal Article · Tue Jan 11 23:00:00 EST 2022 · BMC Medical Informatics and Decision Making (Online) · OSTI ID:1879856

Key uncertainties in climate change policy: Results from ICAM-2

Conference · Sat Dec 30 23:00:00 EST 1995 · OSTI ID:465909

Related Subjects

97 MATHEMATICS AND COMPUTING
Bayesian
EMEWS
calibration
decision-analytic models
high-performance computing
microsimulation models
uncertainty quantification
value of information analysis

Characterization and Valuation of the Uncertainty of Calibrated Parameters in Microsimulation Decision Models

Citation Formats

References (46)

Similar Records

Related Subjects