Randomized Algorithms for Symmetric Nonnegative Matrix Factorization

Hayashi, Koby; Aksoy, Sinan G.; Ballard, Grey; Park, Haesun

doi:10.1137/24m1638355

Title: Randomized Algorithms for Symmetric Nonnegative Matrix Factorization

Journal Article · 27 February 2025 · SIAM Journal on Matrix Analysis and Applications

DOI: https://doi.org/10.1137/24m1638355 · OSTI ID:2526199

^[1];

^[2];

^[3]; Park, Haesun ^[1]

Georgia Institute of Technology, Atlanta, GA (United States)
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
Wake Forest Univ., Winston-Salem, NC (United States)

Symmetric Nonnegative Matrix Factorization (SymNMF) is a technique in data analysis and machine learning that approximates a matrix with a product of a nonnegative, low-rank matrix and it transpose. To design faster and more scalable algorithms for SymNMF we develop two randomized algorithms for its computation. The first method uses randomized matrix sketching to compute an initial low-rank approximation to the input matrix and proceeds to uses this as a low-rank input to rapidly compute a SymNMF. The second methods uses randomized leverage score sampling to approximately solve constrained least squares problems. Many successful methods for SymNMF rely on (approximately) solving sequences of constrained least squares problems. Here, we prove theoretically that leverage score sampling can approximately solve constrained least squares problems to e-accuracy. Finally we demonstrate both methods work in practice by applying them to graph clustering tasks on large real world data sets. These experiments show that our methods approximately maintain solution quality and achieve significant speed ups for both large dense and large sparse problems.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE

Grant/Contract Number:: AC05-76RL01830; SC0020347

OSTI ID:: 2526199

Report Number(s):: PNNL-SA--193926

Journal Information:: SIAM Journal on Matrix Analysis and Applications, Journal Name: SIAM Journal on Matrix Analysis and Applications Journal Issue: 1 Vol. 46; ISSN 0895-4798

Publisher:: Society for Industrial and Applied Mathematics (SIAM)Copyright Statement

Country of Publication:: United States

Language:: English

References (31)

Stability of the solution of definite quadratic programs Daniel, James W. Mathematical Programming, Vol. 5, Issue 1 https://doi.org/10.1007/BF01580110	journal	December 1973
Faster least squares approximation Drineas, Petros; Mahoney, Michael W.; Muthukrishnan, S. Numerische Mathematik, Vol. 117, Issue 2 https://doi.org/10.1007/s00211-010-0331-6	journal	October 2010
Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework Kim, Jingu; He, Yunlong; Park, Haesun Journal of Global Optimization, Vol. 58, Issue 2 https://doi.org/10.1007/s10898-013-0035-4	journal	March 2013
SymNMF: nonnegative low-rank approximation of a similarity matrix for graph clustering Kuang, Da; Yun, Sangwoon; Park, Haesun Journal of Global Optimization, Vol. 62, Issue 3 https://doi.org/10.1007/s10898-014-0247-2	journal	November 2014
DC-NMF: nonnegative matrix factorization based on divide-and-conquer for fast clustering and topic modeling Du, Rundong; Kuang, Da; Drake, Barry Journal of Global Optimization, Vol. 68, Issue 4 https://doi.org/10.1007/s10898-017-0515-z	journal	April 2017
Hybrid clustering based on content and connection structure using joint nonnegative matrix factorization Du, Rundong; Drake, Barry; Park, Haesun Journal of Global Optimization, Vol. 74, Issue 4 https://doi.org/10.1007/s10898-017-0578-x	journal	October 2017
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis Rousseeuw, Peter J. Journal of Computational and Applied Mathematics, Vol. 20 https://doi.org/10.1016/0377-0427(87)90125-7	journal	November 1987
Random projections for the nonnegative least-squares problem Boutsidis, Christos; Drineas, Petros Linear Algebra and its Applications, Vol. 431, Issue 5-7 https://doi.org/10.1016/j.laa.2009.03.026	journal	August 2009
SVD based initialization: A head start for nonnegative matrix factorization Boutsidis, C.; Gallopoulos, E. Pattern Recognition, Vol. 41, Issue 4 https://doi.org/10.1016/j.patcog.2007.09.010	journal	April 2008
Randomized nonnegative matrix factorization Erichson, N. Benjamin; Mendible, Ariana; Wihlborn, Sophie Pattern Recognition Letters, Vol. 104 https://doi.org/10.1016/j.patrec.2018.01.007	journal	March 2018
Randomized numerical linear algebra: Foundations and algorithms Martinsson, Per-Gunnar; Tropp, Joel A. Acta Numerica, Vol. 29 https://doi.org/10.1017/S0962492920000021	journal	May 2020
Learning the parts of objects by non-negative matrix factorization Lee, Daniel D.; Seung, H. Sebastian Nature, Vol. 401, Issue 6755 https://doi.org/10.1038/44565	journal	October 1999
Randomized CP tensor decomposition Erichson, N. Benjamin; Manohar, Krithika; Brunton, Steven L. Machine Learning: Science and Technology, Vol. 1, Issue 2 https://doi.org/10.1088/2632-2153/ab8240	journal	May 2020
MPI-FAUN: An MPI-Based Framework for Alternating-Updating Nonnegative Matrix Factorization Kannan, Ramakrishnan; Ballard, Grey; Park, Haesun IEEE Transactions on Knowledge and Data Engineering, Vol. 30, Issue 3 https://doi.org/10.1109/TKDE.2017.2767592	journal	March 2018
Fast Nonnegative Matrix/Tensor Factorization Based on Low-Rank Approximation Zhou, Guoxu; Cichocki, Andrzej; Xie, Shengli IEEE Transactions on Signal Processing, Vol. 60, Issue 6 https://doi.org/10.1109/TSP.2012.2190410	journal	June 2012
Compressed Nonnegative Matrix Factorization Is Fast and Accurate Tepper, Mariano; Sapiro, Guillermo IEEE Transactions on Signal Processing, Vol. 64, Issue 9 https://doi.org/10.1109/TSP.2016.2516971	journal	May 2016
Efficient and Non-Convex Coordinate Descent for Symmetric Nonnegative Matrix Factorization Vandaele, Arnaud; Gillis, Nicolas; Lei, Qi IEEE Transactions on Signal Processing, Vol. 64, Issue 21 https://doi.org/10.1109/TSP.2016.2591510	journal	November 2016
Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions Halko, N.; Martinsson, P. G.; Tropp, J. A. SIAM Review, Vol. 53, Issue 2 https://doi.org/10.1137/090771806	journal	January 2011
Efficient Nonnegative Matrix Factorization with Random Projections Wang, Fei; Li, Ping Proceedings of the 2010 SIAM International Conference on Data Mining https://doi.org/10.1137/1.9781611972801.25	conference	April 2010
Nonnegative Matrix Factorization Gillis, Nicolas https://doi.org/10.1137/1.9781611976410	book	January 2020
Fast Nonnegative Matrix Factorization: An Active-Set-Like Method and Comparisons Kim, Jingu; Park, Haesun SIAM Journal on Scientific Computing, Vol. 33, Issue 6 https://doi.org/10.1137/110821172	journal	January 2011
Subspace Iteration Randomization and Singular Value Problems Gu, M. SIAM Journal on Scientific Computing, Vol. 37, Issue 3 https://doi.org/10.1137/130938700	journal	January 2015
A Practical Randomized CP Tensor Decomposition Battaglino, Casey; Ballard, Grey; Kolda, Tamara G. SIAM Journal on Matrix Analysis and Applications, Vol. 39, Issue 2 https://doi.org/10.1137/17M1112303	journal	January 2018
Practical Leverage-Based Sampling for Low-Rank Tensor Decomposition Larsen, Brett W.; Kolda, Tamara G. SIAM Journal on Matrix Analysis and Applications, Vol. 43, Issue 3, p. 1488-1517 https://doi.org/10.1137/21M1441754	journal	August 2022
Fast Monte Carlo Algorithms for Matrices I: Approximating Matrix Multiplication Drineas, Petros; Kannan, Ravi; Mahoney, Michael W. SIAM Journal on Computing, Vol. 36, Issue 1 https://doi.org/10.1137/S0097539704442684	journal	January 2006
Planc Eswar, Srinivas; Hayashi, Koby; Ballard, Grey ACM Transactions on Mathematical Software, Vol. 47, Issue 3 https://doi.org/10.1145/3432185	journal	June 2021
Accelerated Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization Gillis, Nicolas; Glineur, François Neural Computation, Vol. 24, Issue 4 https://doi.org/10.1162/NECO_a_00256	journal	April 2012
Projected Gradient Methods for Nonnegative Matrix Factorization Lin, Chih-Jen Neural Computation, Vol. 19, Issue 10 https://doi.org/10.1162/neco.2007.19.10.2756	journal	October 2007
Hierarchical community detection via rank-2 symmetric nonnegative matrix factorization Du, Rundong; Kuang, Da; Drake, Barry Computational Social Networks, Vol. 4, Issue 1 https://doi.org/10.1186/s40649-017-0043-5	journal	September 2017
Mega Whang, Joyce Jiyoung; Du, Rundong; Jung, Sangwon Proceedings of the VLDB Endowment, Vol. 13, Issue 5 https://doi.org/10.14778/3377369.3377378	journal	January 2020
Fast Local Algorithms for Large Scale Nonnegative Matrix and Tensor Factorizations Cichocki, Andrzej; Phan, Anh-Huy IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, Vol. E92-A, Issue 3 https://doi.org/10.1587/transfun.E92.A.708	journal	January 2009

Similar Records

Distributed-Memory Parallel Symmetric Nonnegative Matrix Factorization

Conference · 2020 · OSTI ID:1798617

Eswar, Srinivas; Hayashi, Koby; Ballard, Grey; +3 more

Algorithm for linear least squares problems with equality and nonnegativity constraints

Technical Report · 1978 · OSTI ID:6461744

Haskell, K. H.; Hanson, R. J.

Tensor Decompositions for Count Data that Leverage Stochastic and Deterministic Optimization

Technical Report · 2023 · OSTI ID:2430475

Myers, Jeremy Moulton; Dunlavy, Daniel Michael

Related Subjects

97 MATHEMATICS AND COMPUTING
leverage scores
matrix sketching
nonnegative matrix factorization
randomized linear algebra

Title: Randomized Algorithms for Symmetric Nonnegative Matrix Factorization

Citation Formats

References (31)

Similar Records

Related Subjects