Bayesian projection pursuit regression

Collins, Gavin Quinn; Francom, Devin Craig; Rumsey, Kellin Norman

doi:10.1007/s11222-023-10334-z

Bayesian projection pursuit regression

Journal Article · Sat Nov 04 00:00:00 EDT 2023 · Statistics and Computing

DOI:https://doi.org/10.1007/s11222-023-10334-z· OSTI ID:2280566

Collins, Gavin Quinn ^[1]; ^[2]; ^[2]

Los Alamos National Laboratory (LANL), Los Alamos, NM (United States); The Ohio State Univ., Columbus, OH (United States)
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

In projection pursuit regression (PPR), a univariate response variable is approximated by the sum of $$M$$ “ridge functions,” which are flexible functions of one-dimensional projections of a multivariate input variable. Traditionally, optimization routines are used to choose the projection directions and ridge functions via a sequential algorithm, and $$M$$ is typically chosen via cross-validation. Here, we introduce a novel Bayesian version of PPR, which has the benefit of accurate uncertainty quantification. To infer appropriate projection directions and ridge functions, we apply novel adaptations of methods used for the single ridge function case ($$M$$=1), called the Bayesian Single Index Model; and use a Reversible Jump Markov chain Monte Carlo algorithm to infer the number of ridge functions $$M$$. We evaluate the predictive ability of our model in 20 simulated scenarios and for 23 real datasets, in a bake-off against an array of state-of-the-art regression methods. Finally, we generalize this methodology and demonstrate the ability to accurately model multivariate response variables. Its effective performance indicates that Bayesian Projection Pursuit Regression is a valuable addition to the existing regression toolbox.

View Accepted Manuscript (DOE)

Research Organization:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE Laboratory Directed Research and Development (LDRD) Program; USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: 89233218CNA000001

OSTI ID:: 2280566

Report Number(s):: LA-UR--22-21174

Journal Information:: Statistics and Computing, Journal Name: Statistics and Computing Journal Issue: 1 Vol. 34; ISSN 0960-3174

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (35)

The Elements of Statistical Learning Hastie, Trevor; Tibshirani, Robert; Friedman, Jerome Springer Series in Statistics https://doi.org/10.1007/978-0-387-84858-7	book	January 2009
A Generalized Linear Model With “Gaussian” Regressor Variables Brillinger, David R. Selected Works of David Brillinger https://doi.org/10.1007/978-1-4614-1344-8_34	book	November 2011
Posterior odds ratios for selected regression hypotheses Zellner, A.; Siow, A. Trabajos de Estadistica Y de Investigacion Operativa, Vol. 31, Issue 1 https://doi.org/10.1007/BF02888369	journal	February 1980
Efficient sampling schemes for Bayesian MARS models with many predictors Nott, David J.; Kuk, Anthony Y. C.; Duc, Hiep Statistics and Computing, Vol. 15, Issue 2 https://doi.org/10.1007/s11222-005-6201-x	journal	April 2005
Bayesian estimation and variable selection for single index models Wang, Hai-Bin Computational Statistics & Data Analysis, Vol. 53, Issue 7 https://doi.org/10.1016/j.csda.2008.12.010	journal	May 2009
Bayesian MARS Denison, D. G. T.; Mallick, B. K.; Smith, A. F. M. Statistics and Computing, Vol. 8, Issue 4, p. 337-346 https://doi.org/10.1023/A:1008824606259	journal	December 1998
Random Forests Breiman, Leo Machine Learning, Vol. 45, Issue 1, p. 5-32 https://doi.org/10.1023/A:1010933404324	journal	January 2001
Model of plastic deformation for extreme loading conditions Preston, Dean L.; Tonks, Davis L.; Wallace, Duane C. Journal of Applied Physics, Vol. 93, Issue 1 https://doi.org/10.1063/1.1524706	journal	January 2003
Bayesian calibration of strength parameters using hydrocode simulations of symmetric impact shock experiments of Al-5083 Walters, David J.; Biswas, Ayan; Lawrence, Earl C. Journal of Applied Physics, Vol. 124, Issue 20 https://doi.org/10.1063/1.5051442	journal	November 2018
Gaussian Process Single-Index Models as Emulators for Computer Experiments Gramacy, Robert B.; Lian, Heng Technometrics, Vol. 54, Issue 1 https://doi.org/10.1080/00401706.2012.650527	journal	February 2012
Projection Pursuit Regression Friedman, Jerome H.; Stuetzle, Werner Journal of the American Statistical Association, Vol. 76, Issue 376 https://doi.org/10.1080/01621459.1981.10477729	journal	December 1981
Inferring Atmospheric Release Characteristics in a Large Computer Experiment Using Bayesian Adaptive Splines Francom, Devin; Sansó, Bruno; Bulaevskaya, Vera Journal of the American Statistical Association https://doi.org/10.1080/01621459.2018.1562933	journal	January 2019
Predicting vertical ground reaction force during running using novel piezoresponsive sensors and accelerometry Seeley, Matthew K.; Evans-Pickett, Alyssa; Collins, Gavin Q. Journal of Sports Sciences, Vol. 38, Issue 16 https://doi.org/10.1080/02640414.2020.1757361	journal	May 2020
Calculation Methods for Wallenius' Noncentral Hypergeometric Distribution Fog, Agner Communications in Statistics - Simulation and Computation, Vol. 37, Issue 2 https://doi.org/10.1080/03610910701790269	journal	February 2008
A Gaussian process regression approach to a single-index model Choi, Taeryon; Shi, Jian Q.; Wang, Bo Journal of Nonparametric Statistics, Vol. 23, Issue 1 https://doi.org/10.1080/10485251003768019	journal	March 2011
Local Gaussian Process Approximation for Large Computer Experiments Gramacy, Robert B.; Apley, Daniel W. Journal of Computational and Graphical Statistics, Vol. 24, Issue 2 https://doi.org/10.1080/10618600.2014.914442	journal	April 2015
Regression and Classification Using Gaussian Process Priors Neal, Radford M. Bayesian Statistics 6 https://doi.org/10.1093/oso/9780198504856.003.0021	book	August 1999
A Projection Pursuit Algorithm for Exploratory Data Analysis Friedman, J. H.; Tukey, J. W. IEEE Transactions on Computers, Vol. C-23, Issue 9 https://doi.org/10.1109/T-C.1974.224051	journal	September 1974
Regression Shrinkage and Selection Via the Lasso Tibshirani, Robert Journal of the Royal Statistical Society: Series B (Methodological), Vol. 58, Issue 1 https://doi.org/10.1111/j.2517-6161.1996.tb02080.x	journal	January 1996
Visualizing the effects of predictor variables in black box supervised learning models Apley, Daniel W.; Zhu, Jingyu Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 82, Issue 4 https://doi.org/10.1111/rssb.12377	journal	June 2020
A Novel Method to Characterize Walking and Running Energy Expenditure Evans, Alyssa; Collins, Gavin Q.; Rosquist, Parker G. Journal for the Measurement of Physical Behaviour, Vol. 1, Issue 3 https://doi.org/10.1123/jmpb.2018-0008	journal	September 2018
Multidimensional Additive Spline Approximation Friedman, Jerome H.; Grosse, Eric; Stuetzle, Werner SIAM Journal on Scientific and Statistical Computing, Vol. 4, Issue 2 https://doi.org/10.1137/0904023	journal	June 1983
Mixtures of g Priors for Bayesian Variable Selection Liang, Feng; Paulo, Rui; Molina, German Journal of the American Statistical Association, Vol. 103, Issue 481 https://doi.org/10.1198/016214507000001337	journal	March 2008
An Introduction to Neural Networks Gurney, Kevin https://doi.org/10.1201/9781315273570	book	January 1997
BART: Bayesian additive regression trees Chipman, Hugh A.; George, Edward I.; McCulloch, Robert E. The Annals of Applied Statistics, Vol. 4, Issue 1 https://doi.org/10.1214/09-AOAS285	journal	March 2010
Rank-Normalization, Folding, and Localization: An Improved Rˆ for Assessing Convergence of MCMC (with Discussion) Vehtari, Aki; Gelman, Andrew; Simpson, Daniel Bayesian Analysis, Vol. 16, Issue 2 https://doi.org/10.1214/20-BA1221	journal	June 2021
machine. Friedman, Jerome H. The Annals of Statistics, Vol. 29, Issue 5 https://doi.org/10.1214/aos/1013203451	journal	October 2001
Multivariate Adaptive Regression Splines Friedman, Jerome H. The Annals of Statistics, Vol. 19, Issue 1 https://doi.org/10.1214/aos/1176347963	journal	March 1991
Bayesian backfitting (with comments and a rejoinder by the authors Hastie, Trevor; Tibshirani, Robert Statistical Science, Vol. 15, Issue 3 https://doi.org/10.1214/ss/1009212815	journal	August 2000
Categorical Inputs, Sensitivity Analysis, Optimization and Importance Tempering with tgp Version 2, an R Package for Treed Gaussian Process Models Gramacy, Robert B.; Taddy, Matthew Journal of Statistical Software, Vol. 33, Issue 6 https://doi.org/10.18637/jss.v033.i06	journal	January 2010
BASS : An R Package for Fitting and Performing Sensitivity Analysis of Bayesian Adaptive Spline Surfaces Francom, Devin; Sansó, Bruno Journal of Statistical Software, Vol. 94, Issue 8 https://doi.org/10.18637/jss.v094.i08	journal	January 2020
Reversible Jump Markov Chain Monte Carlo Computation and Bayesian Model Determination Green, Peter J. Biometrika, Vol. 82, Issue 4, p. 711 https://doi.org/10.2307/2337340	journal	December 1995
The Identification of a Particular Nonlinear Time Series System Brillinger, David R. Biometrika, Vol. 64, Issue 3, p. 509 https://doi.org/10.2307/2345326	journal	December 1977
Sensitivity Analysis and Emulation for Functional Data using Bayesian Adaptive Splines Francom,, Devin; Sanso, Bruno; Kupresanin, Ana Statistica Sinica https://doi.org/10.5705/ss.202016.0130	journal	January 2017
Inferring Atmospheric Release Characteristics in a Large Computer Experiment using Bayesian Adaptive Splines Francom, Devin; Sansó, Bruno; Bulaevskaya, Vera Figshare https://doi.org/10.6084/m9.figshare.7639697.v1	collection	January 2019

Similar Records

Classification And Multiple Regression Through Projection Pursuit

Technical Report · Thu Oct 31 23:00:00 EST 1985 · OSTI ID:1447844

Multiclass Classification Using Bayesian Multivariate Adaptive Regression Splines

Journal Article · Mon May 12 20:00:00 EDT 2025 · Baynesian Analysis · OSTI ID:2570745

Projection pursuit adaptation on polynomial chaos expansions

Journal Article · Wed Dec 20 19:00:00 EST 2023 · Computer Methods in Applied Mechanics and Engineering · OSTI ID:2421877

Related Subjects

97 MATHEMATICS AND COMPUTING
bayesian nonparametric statistics
computer experiments
emulator
spline basis
uncertainty quantification

Bayesian projection pursuit regression

Citation Formats

References (35)

Similar Records

Related Subjects