Faster Johnson–Lindenstrauss transforms via Kronecker products

Jin, Ruhui; Kolda, Tamara G.; Ward, Rachel

doi:10.1093/imaiai/iaaa028

Faster Johnson–Lindenstrauss transforms via Kronecker products

Journal Article · Fri Oct 23 00:00:00 EDT 2020 · Information and Inference (Online)

DOI:https://doi.org/10.1093/imaiai/iaaa028· OSTI ID:1738933

Jin, Ruhui ^[1]; Kolda, Tamara G. ^[2]; Ward, Rachel ^[1]

Univ. of Texas, Austin, TX (United States)
Sandia National Lab. (SNL-CA), Livermore, CA (United States)

The Kronecker product is an important matrix operation with a wide range of applications in signal processing, graph theory, quantum computing and deep learning. In this work, we introduce a generalization of the fast Johnson–Lindenstrauss projection for embedding vectors with Kronecker product structure, the Kronecker fast Johnson–Lindenstrauss transform (KFJLT). The KFJLT reduces the embedding cost by an exponential factor of the standard fast Johnson–Lindenstrauss transform’s cost when applied to vectors with Kronecker structure, by avoiding explicitly forming the full Kronecker products. Here, we prove that this computational gain comes with only a small price in embedding power: consider a finite set of $$p$$ points in a tensor product of $$d$$ constituent Euclidean spaces $$\bigotimes _{k=d}^{1}{\mathbb{R}}^{n_k}$$, and let $$N = \prod _{k=1}^{d}n_k$$. With high probability, a random KFJLT matrix of dimension $$m \times N$$ embeds the set of points up to multiplicative distortion $$(1\pm \varepsilon )$$ provided $$m \gtrsim \varepsilon ^{-2} \, \log ^{2d - 1} (p) \, \log N$$. We conclude by describing a direct application of the KFJLT to the efficient solution of large-scale Kronecker-structured least squares problems for fitting the CP tensor decomposition.

View Accepted Manuscript (DOE)

Research Organization:: Sandia National Laboratories (SNL-CA), Livermore, CA (United States)

Sponsoring Organization:: National Science Foundation (NSF); US Air Force Office of Scientific Research (AFOSR); USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)

Grant/Contract Number:: AC04-94AL85000; NA0003525

OSTI ID:: 1738933

Report Number(s):: SAND--2020-13781J; 692879

Journal Information:: Information and Inference (Online), Journal Name: Information and Inference (Online) Journal Issue: 4 Vol. 10; ISSN 2049-8772

Publisher:: Oxford University PressCopyright Statement

Country of Publication:: United States

Language:: English

References (24)

Restricted Isometry of Fourier Matrices and List Decodability of Random Linear Codes Cheraghchi, Mahdi; Guruswami, Venkatesan; Velingker, Ameya SIAM Journal on Computing, Vol. 42, Issue 5 https://doi.org/10.1137/120896773	journal	January 2013
Quantum Wavelet Transforms: Fast Algorithms and Complete Circuits Fijany, Amir; Williams, Colin P. arXiv https://doi.org/10.48550/arxiv.quant-ph/9809004	preprint	January 1998
On sparse reconstruction from Fourier and Gaussian measurements Rudelson, Mark; Vershynin, Roman Communications on Pure and Applied Mathematics, Vol. 61, Issue 8 https://doi.org/10.1002/cpa.20227	journal	January 2008
Suprema of Chaos Processes and the Restricted Isometry Property Krahmer, Felix; Mendelson, Shahar; Rauhut, Holger Communications on Pure and Applied Mathematics, Vol. 67, Issue 11 https://doi.org/10.1002/cpa.21504	journal	January 2014
An elementary proof of a theorem of Johnson and Lindenstrauss Dasgupta, Sanjoy; Gupta, Anupam Random Structures and Algorithms, Vol. 22, Issue 1 https://doi.org/10.1002/rsa.10073	journal	November 2002
A Simple Proof of the Restricted Isometry Property for Random Matrices Baraniuk, Richard; Davenport, Mark; DeVore, Ronald Constructive Approximation, Vol. 28, Issue 3, p. 253-263 https://doi.org/10.1007/s00365-007-9003-x	journal	January 2008
Finding frequent items in data streams Charikar, Moses; Chen, Kevin; Farach-Colton, Martin Theoretical Computer Science, Vol. 312, Issue 1 https://doi.org/10.1016/S0304-3975(03)00400-6	journal	January 2004
Randomized interpolative decomposition of separated representations Biagioni, David J.; Beylkin, Daniel; Beylkin, Gregory Journal of Computational Physics, Vol. 281 https://doi.org/10.1016/j.jcp.2014.10.009	journal	January 2015
Guarantees for the Kronecker fast Johnson–Lindenstrauss transform using a coherence and sampling argument Malik, Osman Asif; Becker, Stephen Linear Algebra and its Applications, Vol. 602 https://doi.org/10.1016/j.laa.2020.05.004	journal	October 2020
Probability Inequalities for Sums of Bounded Random Variables Hoeffding, Wassily Journal of the American Statistical Association, Vol. 58, Issue 301 https://doi.org/10.1080/01621459.1963.10500830	journal	March 1963
Optimality of the Johnson-Lindenstrauss Lemma Larsen, Kasper Green; Nelson, Jelani 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) https://doi.org/10.1109/FOCS.2017.64	conference	October 2017
Kronecker Compressive Sensing Duarte, M. F.; Baraniuk, R. G. IEEE Transactions on Image Processing, Vol. 21, Issue 2, p. 494-504 https://doi.org/10.1109/TIP.2011.2165289	journal	February 2012
Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information Candes, E.J.; Romberg, J.; Tao, T. IEEE Transactions on Information Theory, Vol. 52, Issue 2, p. 489-509 https://doi.org/10.1109/TIT.2005.862083	journal	February 2006
Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? Candes, Emmanuel J.; Tao, Terence IEEE Transactions on Information Theory, Vol. 52, Issue 12, p. 5406-5425 https://doi.org/10.1109/TIT.2006.885507	journal	January 2006
Sparse Sampling for Inverse Problems With Tensors Ortiz-Jimenez, Guillermo; Coutino, Mario; Chepuri, Sundeep Prabhakar IEEE Transactions on Signal Processing, Vol. 67, Issue 12 https://doi.org/10.1109/TSP.2019.2914879	journal	June 2019
The Fast Johnson–Lindenstrauss Transform and Approximate Nearest Neighbors Ailon, Nir; Chazelle, Bernard SIAM Journal on Computing, Vol. 39, Issue 1 https://doi.org/10.1137/060673096	journal	January 2009
Fast Fourier Transforms for Nonequispaced Data Dutt, A.; Rokhlin, V. SIAM Journal on Scientific Computing, Vol. 14, Issue 6 https://doi.org/10.1137/0914081	journal	November 1993
The Restricted Isometry Property of Subsampled Fourier Matrices Haviv, Ishay; Regev, Oded Proceedings of the Twenty-Seventh Annual ACM-SIAM Symposium on Discrete Algorithms https://doi.org/10.1137/1.9781611974331.ch22	conference	January 2016
New and Improved Johnson–Lindenstrauss Embeddings via the Restricted Isometry Property Krahmer, Felix; Ward, Rachel SIAM Journal on Mathematical Analysis, Vol. 43, Issue 3 https://doi.org/10.1137/100810447	journal	January 2011
A Practical Randomized CP Tensor Decomposition Battaglino, Casey; Ballard, Grey; Kolda, Tamara G. SIAM Journal on Matrix Analysis and Applications, Vol. 39, Issue 2 https://doi.org/10.1137/17M1112303	journal	January 2018
An Almost Optimal Unrestricted Fast Johnson-Lindenstrauss Transform Ailon, Nir; Liberty, Edo ACM Transactions on Algorithms, Vol. 9, Issue 3 https://doi.org/10.1145/2483699.2483701	journal	June 2013
Fast and scalable polynomial kernels via explicit feature maps Pham, Ninh; Pagh, Rasmus KDD' 13: The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining https://doi.org/10.1145/2487575.2487591	conference	August 2013
A Bound on Tail Probabilities for Quadratic Forms in Independent Random Variables Hanson, D. L.; Wright, F. T. The Annals of Mathematical Statistics, Vol. 42, Issue 3 https://doi.org/10.1214/aoms/1177693335	journal	June 1971
Concentration inequalities using the entropy method Massart, Pascal; Lugosi, G�bor; Boucheron, St�phane The Annals of Probability, Vol. 31, Issue 3 https://doi.org/10.1214/aop/1055425791	journal	July 2003

Cited By (2)

Structured Random Sketching for PDE Inverse Problems Chen, Ke; Li, Qin; Newton, Kit SIAM Journal on Matrix Analysis and Applications, Vol. 41, Issue 4 https://doi.org/10.1137/20m1310497	journal	January 2020
Practical Leverage-Based Sampling for Low-Rank Tensor Decomposition Larsen, Brett W.; Kolda, Tamara G. arXiv https://doi.org/10.48550/arxiv.2006.16438	preprint	January 2020

Similar Records

An Analysis of the Johnson-Lindenstrauss Lemma with the Bivariate Gamma Distribution

Technical Report · Tue Jan 24 23:00:00 EST 2023 · OSTI ID:1959476

A Class of Sparse Johnson–Lindenstrauss Transforms and Analysis of their Extreme Singular Values

Journal Article · Mon Feb 17 19:00:00 EST 2025 · SIAM Journal on Matrix Analysis and Applications · OSTI ID:2563384

EXTENT: A portable programming environment for designing and implementing high-performance block recursive algorithms

Conference · Fri Dec 30 23:00:00 EST 1994 · OSTI ID:87610

Related Subjects

Kronecker structure
97 MATHEMATICS AND COMPUTING
Johnson-Lindenstrauss embedding
concentration inequality
fast Johnson-Lindenstrauss transform (FJLT)
restricted isometry property

Faster Johnson–Lindenstrauss transforms via Kronecker products

Citation Formats

References (24)

Cited By (2)

Similar Records

Related Subjects