Fast increased fidelity samplers for approximate Bayesian Gaussian process regression

Moran, Kelly R.; Wheeler, Matthew W.

doi:10.1111/rssb.12494

Fast increased fidelity samplers for approximate Bayesian Gaussian process regression

Journal Article · Tue Apr 19 00:00:00 EDT 2022 · Journal of the Royal Statistical Society: Series B (Statistical Methodology)

DOI:https://doi.org/10.1111/rssb.12494· OSTI ID:1876804

^[1]; ^[2]

Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
US National Institutes of Health (NIH), Research Triangle Park, NC (United States).National Institute of Environmental Health Sciences

Gaussian processes (GPs) are common components in Bayesian non-parametric models having a rich methodological literature and strong theoretical grounding. The use of exact GPs in Bayesian models is limited to problems containing several thousand observations due to their prohibitive computational demands. We develop a posterior sampling algorithm using H-matrix approximations that scales at O(n log² n). We show that this approximation’s Kullback-Leibler divergence to the true posterior can be made arbitrarily small. Though multidimensional GPs could be used with our algorithm, d-dimensional surfaces are modeled as tensor products of univariate GPs to minimize the cost of matrix construction and maximize computational efficiency. We illustrate the performance of this fast increased fidelity approximate GP, FIFA-GP, using both simulated and non-synthetic data sets

View Accepted Manuscript (DOE)

Research Organization:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Organization:: National Institute of Environmental Health Sciences (NIEHS); USDOE Laboratory Directed Research and Development (LDRD) Program

Grant/Contract Number:: 89233218CNA000001; FG02-97ER25308

OSTI ID:: 1876804

Report Number(s):: LA-UR-21-23822

Journal Information:: Journal of the Royal Statistical Society: Series B (Statistical Methodology), Journal Name: Journal of the Royal Statistical Society: Series B (Statistical Methodology) Journal Issue: 4 Vol. 84; ISSN 1369-7412

Publisher:: Royal Statistical Society - WileyCopyright Statement

Country of Publication:: United States

Language:: English

References (39)

Small ball probabilities for smooth Gaussian fields and tensor products of compact operators: Smooth Gaussian fields and tensor products of compact operators Karol', Andrei I.; Nazarov, Alexander I. Mathematische Nachrichten, Vol. 287, Issue 5-6 https://doi.org/10.1002/mana.201100010	journal	October 2013
Practical Bayesian modeling and inference for massive spatial data sets on modest computing environments ^† Zhang, Lu; Datta, Abhirup; Banerjee, Sudipto Statistical Analysis and Data Mining: The ASA Data Science Journal, Vol. 12, Issue 3 https://doi.org/10.1002/sam.11413	journal	April 2019
Splines, knots, and penalties Eilers, Paul H. C.; Marx, Brian D. WIREs Computational Statistics, Vol. 2, Issue 6 https://doi.org/10.1002/wics.125	journal	September 2010
On nearest-neighbor Gaussian process models for massive spatial data: Nearest-neighbor Gaussian process models Datta, Abhirup; Banerjee, Sudipto; Finley, Andrew O. Wiley Interdisciplinary Reviews: Computational Statistics, Vol. 8, Issue 5 https://doi.org/10.1002/wics.1383	journal	August 2016
Matrix Analysis Bhatia, Rajendra Graduate Texts in Mathematics https://doi.org/10.1007/978-1-4612-0653-8	book	November 1996
Das asymptotische Verteilungsgesetz der Eigenwerte linearer partieller Differentialgleichungen (mit einer Anwendung auf die Theorie der Hohlraumstrahlung) Weyl, Hermann Mathematische Annalen, Vol. 71, Issue 4 https://doi.org/10.1007/BF01456804	journal	December 1912
Algorithms for smoothing data on the sphere with tensor product splines Dierckx, P. Computing, Vol. 32, Issue 4 https://doi.org/10.1007/BF02243776	journal	December 1984
Data-sparse Approximation by Adaptive ℋ2-Matrices Hackbusch, W.; Börm, S. Computing, Vol. 69, Issue 1, p. 1-35 https://doi.org/10.1007/s00607-002-1450-4	journal	September 2002
Construction and Arithmetics of H -Matrices Grasedyck, Lars; Hackbusch, Wolfgang Computing, Vol. 70, Issue 4 https://doi.org/10.1007/s00607-003-0019-1	journal	August 2003
A Sparse Matrix Arithmetic Based on $\Cal H$ -Matrices. Part I: Introduction to ${\Cal H}$ -Matrices Hackbusch, W. Computing, Vol. 62, Issue 2 https://doi.org/10.1007/s006070050015	journal	April 1999
An $$\mathcal O (N \log N)$$ O ( N log N ) Fast Direct Solver for Partial Hierarchically Semi-Separable Matrices: With Application to Radial Basis Function Interpolation Ambikasaran, Sivaram; Darve, Eric Journal of Scientific Computing, Vol. 57, Issue 3 https://doi.org/10.1007/s10915-013-9714-z	journal	April 2013
Hierarchical-block conditioning approximations for high-dimensional multivariate normal probabilities Cao, Jian; Genton, Marc G.; Keyes, David E. Statistics and Computing, Vol. 29, Issue 3 https://doi.org/10.1007/s11222-018-9825-3	journal	July 2018
Surface fitting using convex tensor-product splines Jüttler, Bert Journal of Computational and Applied Mathematics, Vol. 84, Issue 1 https://doi.org/10.1016/S0377-0427(97)00100-3	journal	October 1997
Fast computation of uncertainty quantification measures in the geostatistical approach to solve inverse problems Saibaba, Arvind K.; Kitanidis, Peter K. Advances in Water Resources, Vol. 82 https://doi.org/10.1016/j.advwatres.2015.04.012	journal	August 2015
Likelihood approximation with hierarchical matrices for large spatial datasets Litvinenko, Alexander; Sun, Ying; Genton, Marc G. Computational Statistics & Data Analysis, Vol. 137 https://doi.org/10.1016/j.csda.2019.02.002	journal	September 2019
Posterior consistency of logistic Gaussian process priors in density estimation Tokdar, Surya T.; Ghosh, Jayanta K. Journal of Statistical Planning and Inference, Vol. 137, Issue 1 https://doi.org/10.1016/j.jspi.2005.09.005	journal	January 2007
Nonparametric binary regression using a Gaussian process prior Choudhuri, Nidhan; Ghosal, Subhashis; Roy, Anindya Statistical Methodology, Vol. 4, Issue 2 https://doi.org/10.1016/j.stamet.2006.07.003	journal	April 2007
Matrix Analysis Horn, Roger A.; Johnson, Charles R. https://doi.org/10.1017/CBO9781139020411	book	April 2013
Efficient methods for large-scale linear inversion using a geostatistical approach: EFFICIENT GEOSTATISTICAL APPROACH Saibaba, Arvind K.; Kitanidis, Peter K. Water Resources Research, Vol. 48, Issue 5 https://doi.org/10.1029/2011WR011778	journal	May 2012
Hierarchical Nearest-Neighbor Gaussian Process Models for Large Geostatistical Datasets Datta, Abhirup; Banerjee, Sudipto; Finley, Andrew O. Journal of the American Statistical Association, Vol. 111, Issue 514 https://doi.org/10.1080/01621459.2015.1044091	journal	April 2016
Highly Scalable Bayesian Geostatistical Modeling via Meshed Gaussian Processes on Partitioned Domains Peruzzi, Michele; Banerjee, Sudipto; Finley, Andrew O. Journal of the American Statistical Association, Vol. 117, Issue 538 https://doi.org/10.1080/01621459.2020.1833889	journal	November 2020
Hierarchical Decompositions for the Computation of High-Dimensional Multivariate Normal Probabilities Genton, Marc G.; Keyes, David E.; Turkiyyah, George Journal of Computational and Graphical Statistics, Vol. 27, Issue 2 https://doi.org/10.1080/10618600.2017.1375936	journal	April 2018
Scalable Gaussian Process Computations Using Hierarchical Matrices Geoga, Christopher J.; Anitescu, Mihai; Stein, Michael L. Journal of Computational and Graphical Statistics, Vol. 29, Issue 2 https://doi.org/10.1080/10618600.2019.1652616	journal	September 2019
The horseshoe estimator for sparse signals Carvalho, C. M.; Polson, N. G.; Scott, J. G. Biometrika, Vol. 97, Issue 2 https://doi.org/10.1093/biomet/asq017	journal	April 2010
Efficient Gaussian process regression for large datasets Banerjee, A.; Dunson, D. B.; Tokdar, S. T. Biometrika, Vol. 100, Issue 1 https://doi.org/10.1093/biomet/ass068	journal	December 2012
Nonparametric Bayes modelling of count processes Canale, A.; Dunson, D. B. Biometrika, Vol. 100, Issue 4 https://doi.org/10.1093/biomet/ast037	journal	October 2013
Improved Approximation Algorithms for Large Matrices via Random Projections Sarlos, Tamas 2006 47th Annual IEEE Conference on Foundations of Computer Science, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06) https://doi.org/10.1109/FOCS.2006.37	conference	October 2006
A Simple Sampler for the Horseshoe Estimator Makalic, Enes; Schmidt, Daniel F. IEEE Signal Processing Letters, Vol. 23, Issue 1 https://doi.org/10.1109/LSP.2015.2503725	journal	January 2016
On building fast kd-Trees for Ray Tracing, and on doing that in O(N log N) Wald, Ingo; Havran, Vlastimil 2006 IEEE Symposium on Interactive Ray Tracing https://doi.org/10.1109/RT.2006.280216	conference	September 2006
Fast Direct Methods for Gaussian Processes Ambikasaran, Sivaram; Foreman-Mackey, Daniel; Greengard, Leslie IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38, Issue 2 https://doi.org/10.1109/TPAMI.2015.2448083	journal	February 2016
Bayesian additive adaptive basis tensor product models for modeling high dimensional surfaces: an application to high‐throughput toxicity testing Wheeler, Matthew W. Biometrics, Vol. 75, Issue 1 https://doi.org/10.1111/biom.12942	journal	August 2018
Spatial factor modeling: A Bayesian matrix‐normal approach for misaligned data Zhang, Lu; Banerjee, Sudipto Biometrics, Vol. 78, Issue 2 https://doi.org/10.1111/biom.13452	journal	March 2021
Gaussian predictive process models for large spatial data sets Banerjee, Sudipto; Gelfand, Alan E.; Finley, Andrew O. Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 70, Issue 4 https://doi.org/10.1111/j.1467-9868.2008.00663.x	journal	September 2008
A full scale approximation of covariance functions for large spatial data sets: Approximation of Covariance Functions for Large Data Sets Sang, Huiyan; Huang, Jianhua Z. Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 74, Issue 1 https://doi.org/10.1111/j.1467-9868.2011.01007.x	journal	October 2011
Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions Halko, N.; Martinsson, P. G.; Tropp, J. A. SIAM Review, Vol. 53, Issue 2 https://doi.org/10.1137/090771806	journal	January 2011
Bayesian Treed Gaussian Process Models With an Application to Computer Modeling Gramacy, Robert B.; Lee, Herbert K. H. Journal of the American Statistical Association, Vol. 103, Issue 483 https://doi.org/10.1198/016214508000000689	journal	September 2008
Adaptive estimation of multivariate functions using conditionally Gaussian tensor-product spline priors de Jonge, R.; van Zanten, J. H. Electronic Journal of Statistics, Vol. 6, Issue none https://doi.org/10.1214/12-EJS735	journal	January 2012
HODLRlib: A Library for Hierarchical Matrices Ambikasaran, Sivaram; Singh, Karan; Sankaran, Shyam Journal of Open Source Software, Vol. 4, Issue 34 https://doi.org/10.21105/joss.01167	journal	February 2019
Additive Covariance kernels for high-dimensional Gaussian Process modeling Durrande, Nicolas; Ginsbourger, David; Roustant, Olivier Annales de la Faculté des sciences de Toulouse : Mathématiques, Vol. 21, Issue 3 https://doi.org/10.5802/afst.1342	journal	January 2012

Similar Records

Bayesian learning of orthogonal embeddings for multi-fidelity Gaussian Processes

Journal Article · Wed Sep 08 20:00:00 EDT 2021 · Computer Methods in Applied Mechanics and Engineering · OSTI ID:1976970

Approximate Bayesian Model Inversion for PDEs with Heterogeneous and State-Dependent Coefficients

Journal Article · Tue Oct 15 00:00:00 EDT 2019 · Journal of Computational Physics · OSTI ID:1572471

Physics makes the difference: Bayesian optimization and active learning via augmented Gaussian process

Journal Article · Sun Feb 06 19:00:00 EST 2022 · Machine Learning: Science and Technology · OSTI ID:1843878

Related Subjects

97 MATHEMATICS AND COMPUTING
Bayesian Gaussian process
Matrix compression
Scalability
Surface estimation
Tensor product

Fast increased fidelity samplers for approximate Bayesian Gaussian process regression

Citation Formats

References (39)

Similar Records

Related Subjects