On Model Selection Criteria in Multimodel Analysis
Hydrologic systems are open and complex, rendering them prone to multiple conceptualizations and mathematical descriptions. There has been a growing tendency to postulate several alternative hydrologic models for a site and use model selection criteria to (a) rank these models, (b) eliminate some of them and/or (c) weigh and average predictions and statistics generated by multiple models. This has led to some debate among hydrogeologists about the merits and demerits of common model selection (also known as model discrimination or information) criteria such as AIC [Akaike, 1974], AICc [Hurvich and Tsai, 1989], BIC [Schwartz, 1978] and KIC [Kashyap, 1982] and some lack of clarity about the proper interpretation and mathematical representation of each criterion. In particular, whereas we [Neuman, 2003; Ye et al., 2004, 2005; Meyer et al., 2007] have based our approach to multimodel hydrologic ranking and inference on the Bayesian criterion KIC (which reduces asymptotically to BIC), Poeter and Anderson [2005] and Poeter and Hill [2007] have voiced a preference for the information-theoretic criterion AICc (which reduces asymptotically to AIC). Their preference stems in part from a perception that KIC and BIC require a "true" or "quasi-true" model to be in the set of alternatives while AIC and AICc are free of such an unreasonable requirement. We examine the model selection literature to find that (a) all published rigorous derivations of AIC and AICc require that the (true) model having generated the observational data be in the set of candidate models; (b) though BIC and KIC were originally derived by assuming that such a model is in the set, BIC has been rederived by Cavanaugh and Neath [1999] without the need for such an assumption; (c) KIC reduces to BIC as the number of observations becomes large relative to the number of adjustable model parameters, implying that it likewise does not require the existence of a true model in the set of alternatives; (d) if a true model is in the set, BIC and KIC select with probability one the true model as sample size increases, a consistency property not shared by AIC and AICc; (e) published comparisons between BIC and AIC (none consider KIC and few consider AICc) tend to rely on the consistency of BIC, which does not apply when a true model is not in the set; and (f) all four criteria have been used with various degrees of success in such situations. We explain why KIC is the only criterion accounting validly for the likelihood of prior parameter estimates, elucidate the unique role that the Fisher information matrix plays in KIC, and demonstrate through an example that it imbues KIC with desirable model selection properties not shared by AIC, AICc or BIC. Our example appears to provide the first comprehensive test of how AIC, AICc, BIC and KIC weigh and rank alternative models in light of the models’ predictive performance under cross-validation with real hydrologic data.
- Research Organization:
- Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC05-76RL01830
- OSTI ID:
- 927977
- Report Number(s):
- PNNL-SA-59158; WRERAQ; 401001060; TRN: US200816%%930
- Journal Information:
- Water Resources Research, 44(3):Art. no. W03428, Vol. 44, Issue 3; ISSN 0043-1397
- Country of Publication:
- United States
- Language:
- English
Similar Records
Effects of error covariance structure on estimation of model averaging weights and predictive performance
On Evaluation of Recharge Model Uncertainty: a Priori and a Posteriori