Foundational Principles for Large-Scale Inference: Illustrations Through Correlation Mining

Hero, Alfred O.; Rajaratnam, Bala

doi:10.1109/JPROC.2015.2494178

Title: Foundational Principles for Large-Scale Inference: Illustrations Through Correlation Mining

Full Record
Other Related Research

Abstract

When can reliable inference be drawn in the ‘‘Big Data’’ context? This article presents a framework for answering this fundamental question in the context of correlation mining, with implications for general large-scale inference. In large-scale data applications like genomics, connectomics, and eco-informatics, the data set is often variable rich but sample starved: a regime where the number n of acquired samples (statistical replicates) is far fewer than the number p of observed variables (genes, neurons, voxels, or chemical constituents). Much of recent work has focused on understanding the computational complexity of proposed methods for ‘‘Big Data.’’ Sample complexity, however, has received relatively less attention, especially in the setting when the sample size n is fixed, and the dimension p grows without bound. To address this gap, we develop a unified statistical framework that explicitly quantifies the sample complexity of various inferential tasks. Sampling regimes can be divided into several categories: 1) the classical asymptotic regime where the variable dimension is fixed and the sample size goes to infinity; 2) the mixed asymptotic regime where both variable dimension and sample size go to infinity at comparable rates; and 3) the purely high-dimensional asymptotic regime where the variable dimension goes to infinitymore »« less

Authors:

Hero, Alfred O. ^[1]; Rajaratnam, Bala ^[2]

Univ. of Michigan, Ann Arbor, MI (United States)
Stanford Univ., CA (United States)

Publication Date:: 2015-12-09

Research Org.:: Univ. of Michigan, Ann Arbor, MI (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA); National Institutes of Health (NIH); National Science Foundation (NSF); US Army Research Office (ARO)

OSTI Identifier:: 1367662

Grant/Contract Number:: NA0002534; FA9550-13-1-0043; W911NF-11-1-0391; W911NF-12-1-0443; 2P01CA087634-06A2; DMS-0906392; DMS-CMG1025465; AGS-1003823; DMS-1106642; DMS-CAREER-1352656; DARPA-YFAN66001-111-4131

Resource Type:: Accepted Manuscript

Journal Name:: Proceedings of the IEEE

Additional Journal Information:: Journal Volume: 104; Journal Issue: 1; Journal ID: ISSN 0018-9219

Publisher:: Institute of Electrical and Electronics Engineers

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Asymptotic regimes; big data; correlation estima-tion; correlation mining; correlation screening; correlation selection; graphical models; large-scale inference; purely high dimensional; sample complexity; triple asymptotic framework; unifying learning theory

Citation Formats


                    Hero, Alfred O., and Rajaratnam, Bala. Foundational Principles for Large-Scale Inference: Illustrations Through Correlation Mining.  United States: N. p., 2015. 
Web.  doi:10.1109/JPROC.2015.2494178.

Copy to clipboard


                    Hero, Alfred O., & Rajaratnam, Bala. Foundational Principles for Large-Scale Inference: Illustrations Through Correlation Mining.  United States.  https://doi.org/10.1109/JPROC.2015.2494178

Copy to clipboard


                    Hero, Alfred O., and Rajaratnam, Bala. Wed .  
"Foundational Principles for Large-Scale Inference: Illustrations Through Correlation Mining".  United States.  https://doi.org/10.1109/JPROC.2015.2494178.  https://www.osti.gov/servlets/purl/1367662.

Copy to clipboard


                    
@article{osti_1367662,

  title        = {Foundational Principles for Large-Scale Inference: Illustrations Through Correlation Mining},

  author       = {Hero, Alfred O. and Rajaratnam, Bala},

  abstractNote = {When can reliable inference be drawn in the ‘‘Big Data’’ context? This article presents a framework for answering this fundamental question in the context of correlation mining, with implications for general large-scale inference. In large-scale data applications like genomics, connectomics, and eco-informatics, the data set is often variable rich but sample starved: a regime where the number n of acquired samples (statistical replicates) is far fewer than the number p of observed variables (genes, neurons, voxels, or chemical constituents). Much of recent work has focused on understanding the computational complexity of proposed methods for ‘‘Big Data.’’ Sample complexity, however, has received relatively less attention, especially in the setting when the sample size n is fixed, and the dimension p grows without bound. To address this gap, we develop a unified statistical framework that explicitly quantifies the sample complexity of various inferential tasks. Sampling regimes can be divided into several categories: 1) the classical asymptotic regime where the variable dimension is fixed and the sample size goes to infinity; 2) the mixed asymptotic regime where both variable dimension and sample size go to infinity at comparable rates; and 3) the purely high-dimensional asymptotic regime where the variable dimension goes to infinity and the sample size is fixed. Each regime has its niche but only the latter regime applies to exa-scale data dimension. We illustrate this high-dimensional framework for the problem of correlation mining, where it is the matrix of pairwise and partial correlations among the variables that are of interest. Correlation mining arises in numerous applications and subsumes the regression context as a special case. We demonstrate various regimes of correlation mining based on the unifying perspective of high-dimensional learning rates and sample complexity for different structured covariance models and different inference tasks.},

  doi          = {10.1109/JPROC.2015.2494178},

  journal      = {Proceedings of the IEEE},

  number       = 1,

  volume       = 104,

  place        = {United States},

  year         = {2015},

  month        = {12}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1109/JPROC.2015.2494178

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 13 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

A convex pseudolikelihood framework for high dimensional partial correlation estimation with convergence guarantees

Journal Article Khare, Kshitij ; Oh, Sang-Yun ; Rajaratnam, Bala - Journal of the Royal Statistical Society: Series B (Statistical Methodology)

Sparse high dimensional graphical model selection is a topic of much interest in modern day statistics. A popular approach is to apply l1-penalties to either parametric likelihoods, or regularized regression/pseudolikelihoods, with the latter having the distinct advantage that they do not explicitly assume Gaussianity. As none of the popular methods proposed for solving pseudolikelihood-based objective functions have provable convergence guarantees, it is not clear whether corresponding estimators exist or are even computable, or if they actually yield correct partial correlation graphs. We propose a new pseudolikelihood-based graphical model selection method that aims to overcome some of the shortcomings of currentmore »« less
Cited by 57
https://doi.org/10.1111/rssb.12088

Full Text Available
Statistical theory of nuclear reactions and the Gaussian orthogonal ensemble

Journal Article Weidenmueller, H A - Ann. Phys. (N.Y.); (United States)

Using methods developed in field theory and statical mechanics, especially in the context of the Anderson model as generalised by Wegner, a novel approach to the statistical theory of nuclear reactions is developed. A finite set of N bound states, coupled to each other by an ensemble of Gaussian orthogonal matrices, is considered and coupled to a set of channels via fixed coupling matrix elements. The ensemble average and the variance of the elements of the nuclear scattering matrix are evaluated, using the method of a generating function combined with the replica trick, followed by the Hubbard-Stratonovich transformation and amore »« less
https://doi.org/10.1016/0003-4916(84)90241-0
Bayesian sparse learning with preconditioned stochastic gradient MCMC and its applications

Journal Article Wang, Yating ; Deng, Wei ; Lin, Guang - Journal of Computational Physics

Deep neural networks have been successfully employed in an extensive variety of research areas, including solving partial differential equations. Despite its significant success, there are some challenges in effectively training DNN, such as avoiding overfitting in over-parameterized DNNs and accelerating the optimization in DNNs with pathological curvature. Here, we propose a Bayesian type sparse deep learning algorithm. The algorithm utilizes a set of spike-and-slab priors for the parameters in the deep neural network. The hierarchical Bayesian mixture will be trained using an adaptive empirical method. That is, one will alternatively sample from the posterior using preconditioned stochastic gradient Langevin Dynamicsmore »« less
https://doi.org/10.1016/j.jcp.2021.110134

Full Text Available
Quickest Detection for Changes in Maximal kNN Coherence of Random Matrices

Journal Article Banerjee, Taposh ; Firouzi, Hamed ; Hero, III, Alfred O. - IEEE Transactions on Signal Processing

This paper addresses the problem of quickest detection of a change in the maximal coherence between columns of a n p random matrix based on a sequence of matrix observations having a single unknown change point. The random matrix is assumed to have identically distributed rows and the maximal coherence is defined as the largest of the (₂p) correlation coefficients associated with any row. Likewise the κ nearest neighbor (kNN) coherence is defined as the κ-th largest of these correlation coefficients. The forms of the preand postchange distributions of the observed matrices are assumed to belong to the family ofmore »« less
Cited by 6
https://doi.org/10.1109/TSP.2018.2855644

Full Text Available
Correlator bank detection of gravitational wave chirps--False-alarm probability, template density, and thresholds: Behind and beyond the minimal-match issue

Journal Article Croce, R P ; Demma, Th ; Pierro, V ; ... - Physical Review. D, Particles Fields

The general problem of computing the false-alarm probability vs the detection-threshold relationship for a bank of correlators is addressed, in the context of maximum-likelihood detection of gravitational waves in additive stationary Gaussian noise. Specific reference is made to chirps from coalescing binary systems. Accurate (lower-bound) approximants for the cumulative distribution of the whole-bank supremum are deduced from a class of Bonferroni-type inequalities. The asymptotic properties of the cumulative distribution are obtained, in the limit where the number of correlators goes to infinity. The validity of numerical simulations made on small-size banks is extended to banks of any size, via amore »« less
https://doi.org/10.1103/PhysRevD.70.122001

Similar Records