A Study of Clustering Techniques and Hierarchical Matrix Formats for Kernel Ridge Regression

Rebrova, Elizaveta; Chavez, Gustavo; Liu, Yang; Ghysels, Pieter; Li, Xiaoye Sherry

doi:10.1109/IPDPSW.2018.00140

Title: A Study of Clustering Techniques and Hierarchical Matrix Formats for Kernel Ridge Regression

Journal Article · Mon Aug 06 00:00:00 EDT 2018 · Proceedings - IEEE International Parallel and Distributed Processing Symposium (IPDPS)

DOI:https://doi.org/10.1109/IPDPSW.2018.00140· OSTI ID:1563957

Rebrova, Elizaveta ^[1]; Chavez, Gustavo ^[2]; Liu, Yang ^[2]; Ghysels, Pieter ^[2]; Li, Xiaoye Sherry ^[2]

Univ. of Michigan, Ann Arbor, MI (United States). Dept. of Mathematics
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division

We present memory-efficient and scalable algorithms for kernel methods used in machine learning. Using hierarchical matrix approximations for the kernel matrix the memory requirements, the number of floating point operations, and the execution time are drastically reduced compared to standard dense linear algebra routines. We consider both the general H matrix hierarchical format as well as Hierarchically Semi-Separable (HSS) matrices. Furthermore, we investigate the impact of several preprocessing and clustering techniques on the hierarchical matrix compression. Effective clustering of the input leads to a ten-fold increase in efficiency of the compression. The algorithms are implemented using the STRUMPACK solver library. These results confirm that - with correct tuning of the hyperparameters - classification using kernel ridge regression with the compressed matrix does not lose prediction accuracy compared to the exact - not compressed - kernel matrix and that our approach can be extended to O(1M) datasets, for which computation with the full kernel matrix becomes prohibitively expensive. We present numerical experiments in a distributed memory environment up to 1,024 processors of the NERSC's Cori supercomputer using well-known datasets to the machine learning community that range from dimension 8 up to 784.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1563957

Journal Information:: Proceedings - IEEE International Parallel and Distributed Processing Symposium (IPDPS), Vol. 2018; ISSN 1530-2075

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 9 works

Citation information provided by
Web of Science

References (18)

A Fast $ULV$ Decomposition Solver for Hierarchically Semiseparable Representations Chandrasekaran, S.; Gu, M.; Pals, T. SIAM Journal on Matrix Analysis and Applications, Vol. 28, Issue 3 https://doi.org/10.1137/S0895479803436652	journal	January 2006
A Robust Parallel Preconditioner for Indefinite Systems Using Hierarchical Matrices and Randomized Sampling Ghysels, Pieter; Xiaoye, Sherry Li; Gorman, Christopher 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2017.21	conference	May 2017
OpenTuner: an extensible framework for program autotuning Ansel, Jason; Kamil, Shoaib; Veeramachaneni, Kalyan Proceedings of the 23rd international conference on Parallel architectures and compilation - PACT '14 https://doi.org/10.1145/2628071.2628092	conference	January 2014
Far-field compression for fast kernel summation methods in high dimensions March, William B.; Biros, George Applied and Computational Harmonic Analysis, Vol. 43, Issue 1 https://doi.org/10.1016/j.acha.2015.09.007	journal	July 2017
ASKIT: Approximate Skeletonization Kernel-Independent Treecode in High Dimensions March, William B.; Xiao, Bo; Biros, George SIAM Journal on Scientific Computing, Vol. 37, Issue 2 https://doi.org/10.1137/140989546	journal	January 2015
A Fast Randomized Algorithm for Computing a Hierarchically Semiseparable Representation of a Matrix Martinsson, P. G. SIAM Journal on Matrix Analysis and Applications, Vol. 32, Issue 4 https://doi.org/10.1137/100786617	journal	October 2011
A Distributed-Memory Package for Dense Hierarchically Semi-Separable Matrix Computations Using Randomization Rouet, François-Henry; Li, Xiaoye S.; Ghysels, Pieter ACM Transactions on Mathematical Software, Vol. 42, Issue 4 https://doi.org/10.1145/2930660	journal	June 2016
Improved Approximation Algorithms for Large Matrices via Random Projections Sarlos, Tamas 2006 47th Annual IEEE Conference on Foundations of Computer Science, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06) https://doi.org/10.1109/FOCS.2006.37	conference	October 2006
Clustered low rank approximation of graphs in information science applications Savas, Berkant; Dhillon, Inderjit S. Proceedings of the 2011 SIAM International Conference on Data Mining https://doi.org/10.1137/1.9781611972818.15	conference	December 2013
On the Compression of Low Rank Matrices Cheng, H.; Gimbutas, Z.; Martinsson, P. G. SIAM Journal on Scientific Computing, Vol. 26, Issue 4 https://doi.org/10.1137/030602678	journal	January 2005
INV-ASKIT: A Parallel Fast Direct Solver for Kernel Matrices Yu, Chenhan D.; March, William B.; Xiao, Bo 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2016.12	conference	May 2016
A fast algorithm for particle simulations Greengard, L.; Rokhlin, V. Journal of Computational Physics, Vol. 73, Issue 2 https://doi.org/10.1016/0021-9991(87)90140-9	journal	December 1987
A theory of pseudoskeleton approximations Goreinov, S. A.; Tyrtyshnikov, E. E.; Zamarashkin, N. L. Linear Algebra and its Applications, Vol. 261, Issue 1-3 https://doi.org/10.1016/S0024-3795(96)00301-1	journal	August 1997
An N log N Parallel Fast Direct Solver for Kernel Matrices Yu, Chenhan D.; March, William B.; Biros, George 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2017.10	conference	May 2017
Kernel methods in machine learning Hofmann, Thomas; Schölkopf, Bernhard; Smola, Alexander J. The Annals of Statistics, Vol. 36, Issue 3 https://doi.org/10.1214/009053607000000677	journal	June 2008
Performance optimization for the k-nearest neighbors kernel on x86 architectures Yu, Chenhan D.; Huang, Jianyu; Austin, Woody SC15: The International Conference for High Performance Computing, Networking, Storage and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/2807591.2807601	conference	November 2015
A hierarchical O(N log N) force-calculation algorithm Barnes, Josh; Hut, Piet Nature, Vol. 324, Issue 6096 https://doi.org/10.1038/324446a0	journal	December 1986
A Butterfly-Based Direct Integral-Equation Solver Using Hierarchical LU Factorization for Analyzing Scattering From Electrically Large Conducting Objects Guo, Han; Liu, Yang; Hu, Jun IEEE Transactions on Antennas and Propagation, Vol. 65, Issue 9 https://doi.org/10.1109/TAP.2017.2727511	journal	September 2017

Cited By (1)

Preparing sparse solvers for exascale computing Anzt, Hartwig; Boman, Erik; Falgout, Rob Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 378, Issue 2166 https://doi.org/10.1098/rsta.2019.0053	journal	January 2020