skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Evaluating marginal likelihood with thermodynamic integration method and comparison with several other numerical methods

Abstract

Evaluating marginal likelihood is the most critical and computationally expensive task, when conducting Bayesian model averaging to quantify parametric and model uncertainties. The evaluation is commonly done by using Laplace approximations to evaluate semianalytical expressions of the marginal likelihood or by using Monte Carlo (MC) methods to evaluate arithmetic or harmonic mean of a joint likelihood function. This study introduces a new MC method, i.e., thermodynamic integration, which has not been attempted in environmental modeling. Instead of using samples only from prior parameter space (as in arithmetic mean evaluation) or posterior parameter space (as in harmonic mean evaluation), the thermodynamic integration method uses samples generated gradually from the prior to posterior parameter space. This is done through a path sampling that conducts Markov chain Monte Carlo simulation with different power coefficient values applied to the joint likelihood function. The thermodynamic integration method is evaluated using three analytical functions by comparing the method with two variants of the Laplace approximation method and three MC methods, including the nested sampling method that is recently introduced into environmental modeling. The thermodynamic integration method outperforms the other methods in terms of their accuracy, convergence, and consistency. The thermodynamic integration method is also applied tomore » a synthetic case of groundwater modeling with four alternative models. The application shows that model probabilities obtained using the thermodynamic integration method improves predictive performance of Bayesian model averaging. As a result, the thermodynamic integration method is mathematically rigorous, and its MC implementation is computationally general for a wide range of environmental problems.« less

Authors:
 [1];  [2];  [2];  [2];  [3];  [4];  [5]
  1. Hefei Univ. of Technology, Hefei (China); Florida State Univ., Tallahassee, FL (United States)
  2. Florida State Univ., Tallahassee, FL (United States)
  3. Nanjing Univ., Nanjing (China)
  4. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
  5. Hefei Univ. of Technology, Hefei (China)
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1326496
Alternate Identifier(s):
OSTI ID: 1402380
Grant/Contract Number:  
AC05-00OR22725; DE‐SC0008272
Resource Type:
Accepted Manuscript
Journal Name:
Water Resources Research
Additional Journal Information:
Journal Volume: 52; Journal Issue: 2; Journal ID: ISSN 0043-1397
Publisher:
American Geophysical Union (AGU)
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Bayesian model averaging; arithmetic mean; harmonic mean; nested sampling; model uncertainty; Markov chain Monte Carlo

Citation Formats

Liu, Peigui, Elshall, Ahmed S., Ye, Ming, Beerli, Peter, Zeng, Xiankui, Lu, Dan, and Tao, Yuezan. Evaluating marginal likelihood with thermodynamic integration method and comparison with several other numerical methods. United States: N. p., 2016. Web. https://doi.org/10.1002/2014WR016718.
Liu, Peigui, Elshall, Ahmed S., Ye, Ming, Beerli, Peter, Zeng, Xiankui, Lu, Dan, & Tao, Yuezan. Evaluating marginal likelihood with thermodynamic integration method and comparison with several other numerical methods. United States. https://doi.org/10.1002/2014WR016718
Liu, Peigui, Elshall, Ahmed S., Ye, Ming, Beerli, Peter, Zeng, Xiankui, Lu, Dan, and Tao, Yuezan. Fri . "Evaluating marginal likelihood with thermodynamic integration method and comparison with several other numerical methods". United States. https://doi.org/10.1002/2014WR016718. https://www.osti.gov/servlets/purl/1326496.
@article{osti_1326496,
title = {Evaluating marginal likelihood with thermodynamic integration method and comparison with several other numerical methods},
author = {Liu, Peigui and Elshall, Ahmed S. and Ye, Ming and Beerli, Peter and Zeng, Xiankui and Lu, Dan and Tao, Yuezan},
abstractNote = {Evaluating marginal likelihood is the most critical and computationally expensive task, when conducting Bayesian model averaging to quantify parametric and model uncertainties. The evaluation is commonly done by using Laplace approximations to evaluate semianalytical expressions of the marginal likelihood or by using Monte Carlo (MC) methods to evaluate arithmetic or harmonic mean of a joint likelihood function. This study introduces a new MC method, i.e., thermodynamic integration, which has not been attempted in environmental modeling. Instead of using samples only from prior parameter space (as in arithmetic mean evaluation) or posterior parameter space (as in harmonic mean evaluation), the thermodynamic integration method uses samples generated gradually from the prior to posterior parameter space. This is done through a path sampling that conducts Markov chain Monte Carlo simulation with different power coefficient values applied to the joint likelihood function. The thermodynamic integration method is evaluated using three analytical functions by comparing the method with two variants of the Laplace approximation method and three MC methods, including the nested sampling method that is recently introduced into environmental modeling. The thermodynamic integration method outperforms the other methods in terms of their accuracy, convergence, and consistency. The thermodynamic integration method is also applied to a synthetic case of groundwater modeling with four alternative models. The application shows that model probabilities obtained using the thermodynamic integration method improves predictive performance of Bayesian model averaging. As a result, the thermodynamic integration method is mathematically rigorous, and its MC implementation is computationally general for a wide range of environmental problems.},
doi = {10.1002/2014WR016718},
journal = {Water Resources Research},
number = 2,
volume = 52,
place = {United States},
year = {2016},
month = {2}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 3 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Bayes Factors
journal, June 1995


Maximum likelihood Bayesian averaging of spatial variability models in unsaturated fractured tuff: MAXIMUM LIKELIHOOD BAYESIAN MODEL AVERAGING
journal, May 2004

  • Ye, Ming; Neuman, Shlomo P.; Meyer, Philip D.
  • Water Resources Research, Vol. 40, Issue 5
  • DOI: 10.1029/2003WR002557

Nested Sampling
conference, January 2004

  • Skilling, John
  • BAYESIAN INFERENCE AND MAXIMUM ENTROPY METHODS IN SCIENCE AND ENGINEERING: 24th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering, AIP Conference Proceedings
  • DOI: 10.1063/1.1835238

Marginal likelihood estimation via power posteriors
journal, July 2008

  • Friel, N.; Pettitt, A. N.
  • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 70, Issue 3
  • DOI: 10.1111/j.1467-9868.2007.00650.x

Simulating normalizing constants: from importance sampling to bridge sampling to path sampling
journal, May 1998


Nested sampling algorithm for subsurface flow model selection, uncertainty quantification, and nonlinear calibration: NESTED SAMPLING FOR SUBSURFACE FLOW MODELS
journal, December 2013

  • Elsheikh, A. H.; Wheeler, M. F.; Hoteit, I.
  • Water Resources Research, Vol. 49, Issue 12
  • DOI: 10.1002/2012WR013406

Treatment of input uncertainty in hydrologic modeling: Doing hydrology backward with Markov chain Monte Carlo simulation: FORCING DATA ERROR USING MCMC SAMPLING
journal, December 2008

  • Vrugt, Jasper A.; ter Braak, Cajo J. F.; Clark, Martyn P.
  • Water Resources Research, Vol. 44, Issue 12
  • DOI: 10.1029/2007WR006720

Accelerating Markov Chain Monte Carlo Simulation by Differential Evolution with Self-Adaptive Randomized Subspace Sampling
journal, January 2009

  • Vrugt, J. A.; ter Braak, C. J. F.; Diks, C. G. H.
  • International Journal of Nonlinear Sciences and Numerical Simulation, Vol. 10, Issue 3
  • DOI: 10.1515/IJNSNS.2009.10.3.273

Bayesian inference in physics
journal, September 2011


Hydrological model selection: A Bayesian alternative: HYDROLOGICAL MODEL SELECTION
journal, October 2005

  • Marshall, Lucy; Nott, David; Sharma, Ashish
  • Water Resources Research, Vol. 41, Issue 10
  • DOI: 10.1029/2004WR003719

Review of surrogate modeling in water resources: REVIEW
journal, July 2012

  • Razavi, Saman; Tolson, Bryan A.; Burn, Donald H.
  • Water Resources Research, Vol. 48, Issue 7
  • DOI: 10.1029/2011WR011527

Assessment of parametric uncertainty for groundwater reactive transport modeling
journal, May 2014

  • Shi, Xiaoqing; Ye, Ming; Curtis, Gary P.
  • Water Resources Research, Vol. 50, Issue 5
  • DOI: 10.1002/2013WR013755

A multimodel data assimilation framework via the ensemble Kalman filter
journal, May 2014


Estimation of Aquifer Parameters Under Transient and Steady State Conditions: 1. Maximum Likelihood Method Incorporating Prior Information
journal, February 1986


Assessing five evolving microbial enzyme models against field measurements from a semiarid savannah-What are the mechanisms of soil respiration pulses?
journal, September 2014

  • Zhang, Xia; Niu, Guo-Yue; Elshall, Ahmed S.
  • Geophysical Research Letters, Vol. 41, Issue 18
  • DOI: 10.1002/2014GL061399

Bayesian analysis of data-worth considering model and parameter uncertainties
journal, February 2012


A Model-Averaging Method for Assessing Groundwater Conceptual Model Uncertainty
journal, August 2010


Computing the Bayes Factor from a Markov Chain Monte Carlo Simulation of the Posterior Distribution
journal, September 2012


Estimating the evidence - a review
journal, January 2012


Model selection on solid ground: Rigorous comparison of nine ways to evaluate Bayesian model evidence
journal, December 2014

  • Schöniger, Anneli; Wöhling, Thomas; Samaniego, Luis
  • Water Resources Research, Vol. 50, Issue 12
  • DOI: 10.1002/2014WR016062

Postaudit evaluation of conceptual model uncertainty for a glacial aquifer groundwater flow and contaminant transport model
journal, January 2010


Unified Framework to Evaluate Panmixia and Migration Direction Among Multiple Sampling Locations
journal, February 2010


Computing Bayes Factors Using Thermodynamic Integration
journal, April 2006


Hydrologic data assimilation using particle Markov chain Monte Carlo simulation: Theory, concepts and applications
journal, January 2013


Maximum likelihood Bayesian averaging of uncertain model predictions
journal, November 2003

  • Neuman, S. P.
  • Stochastic Environmental Research and Risk Assessment (SERRA), Vol. 17, Issue 5
  • DOI: 10.1007/s00477-003-0151-7

On model selection criteria in multimodel analysis: ON MODEL SELECTION CRITERIA IN MULTIMODEL ANALYSIS
journal, March 2008

  • Ye, Ming; Meyer, Philip D.; Neuman, Shlomo P.
  • Water Resources Research, Vol. 44, Issue 3
  • DOI: 10.1029/2008WR006803

Nested sampling for general Bayesian computation
journal, December 2006


A Bayesian approach for inverse modeling, data assimilation, and conditional simulation of spatial random fields
journal, October 2010

  • Rubin, Yoram; Chen, Xingyuan; Murakami, Haruko
  • Water Resources Research, Vol. 46, Issue 10
  • DOI: 10.1029/2009WR008799

Marginal Likelihood From the Metropolis–Hastings Output
journal, March 2001

  • Chib, Siddhartha; Jeliazkov, Ivan
  • Journal of the American Statistical Association, Vol. 96, Issue 453
  • DOI: 10.1198/016214501750332848

Improving Marginal Likelihood Estimation for Bayesian Phylogenetic Model Selection
journal, December 2010


Markov Chain Monte Carlo Methods for Computing Bayes Factors: A Comparative Review
journal, September 2001

  • Han, Cong; Carlin, Bradley P.
  • Journal of the American Statistical Association, Vol. 96, Issue 455
  • DOI: 10.1198/016214501753208780

Multimodel Bayesian analysis of data-worth applied to unsaturated fractured tuffs
journal, January 2012


Model complexity control for hydrologic prediction: MODEL COMPLEXITY CONTROL
journal, August 2008

  • Schoups, G.; van de Giesen, N. C.; Savenije, H. H. G.
  • Water Resources Research, Vol. 44, Issue 12
  • DOI: 10.1029/2008WR006836

Using MCMC chain outputs to efficiently estimate Bayes factors
journal, October 2011

  • Morey, Richard D.; Rouder, Jeffrey N.; Pratte, Michael S.
  • Journal of Mathematical Psychology, Vol. 55, Issue 5
  • DOI: 10.1016/j.jmp.2011.06.004

Dependence of Bayesian Model Selection Criteria and Fisher Information Matrix on Sample Size
journal, October 2011


David Draper and E. I. George, and a rejoinder by the authors
journal, November 1999

  • Volinsky, Chris T.; Raftery, Adrian E.; Madigan, David
  • Statistical Science, Vol. 14, Issue 4
  • DOI: 10.1214/ss/1009212519

Markov Chain Sampling Methods for Dirichlet Process Mixture Models
journal, June 2000

  • Neal, Radford M.
  • Journal of Computational and Graphical Statistics, Vol. 9, Issue 2
  • DOI: 10.2307/1390653

    Works referencing / citing this record:

    Erratum: Tidal Deformabilities and Radii of Neutron Stars from the Observation of GW170817 [Phys. Rev. Lett. 121 , 091102 (2018)]
    journal, December 2018


    Hydrogeological Model Selection Among Complex Spatial Priors
    journal, August 2019

    • Brunetti, C.; Bianchi, M.; Pirot, G.
    • Water Resources Research, Vol. 55, Issue 8
    • DOI: 10.1029/2019wr024840

    A Primer for Model Selection: The Decisive Role of Model Complexity
    journal, March 2018

    • Höge, Marvin; Wöhling, Thomas; Nowak, Wolfgang
    • Water Resources Research, Vol. 54, Issue 3
    • DOI: 10.1002/2017wr021902

    Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling
    journal, August 2018

    • Elshall, Ahmed S.; Ye, Ming; Pei, Yongzhen
    • Stochastic Environmental Research and Risk Assessment, Vol. 32, Issue 10
    • DOI: 10.1007/s00477-018-1592-3