Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

A Study of Clustering Techniques and Hierarchical Matrix Formats for Kernel Ridge Regression

Journal Article · · Proceedings - IEEE International Parallel and Distributed Processing Symposium (IPDPS)
 [1];  [2];  [2];  [2];  [2]
  1. Univ. of Michigan, Ann Arbor, MI (United States). Dept. of Mathematics
  2. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
+ Show Author Affiliations
We present memory-efficient and scalable algorithms for kernel methods used in machine learning. Using hierarchical matrix approximations for the kernel matrix the memory requirements, the number of floating point operations, and the execution time are drastically reduced compared to standard dense linear algebra routines. We consider both the general H matrix hierarchical format as well as Hierarchically Semi-Separable (HSS) matrices. Furthermore, we investigate the impact of several preprocessing and clustering techniques on the hierarchical matrix compression. Effective clustering of the input leads to a ten-fold increase in efficiency of the compression. The algorithms are implemented using the STRUMPACK solver library. These results confirm that - with correct tuning of the hyperparameters - classification using kernel ridge regression with the compressed matrix does not lose prediction accuracy compared to the exact - not compressed - kernel matrix and that our approach can be extended to O(1M) datasets, for which computation with the full kernel matrix becomes prohibitively expensive. We present numerical experiments in a distributed memory environment up to 1,024 processors of the NERSC's Cori supercomputer using well-known datasets to the machine learning community that range from dimension 8 up to 784.
Research Organization:
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
Sponsoring Organization:
USDOE Office of Science (SC)
Grant/Contract Number:
AC02-05CH11231
OSTI ID:
1563957
Journal Information:
Proceedings - IEEE International Parallel and Distributed Processing Symposium (IPDPS), Journal Name: Proceedings - IEEE International Parallel and Distributed Processing Symposium (IPDPS) Vol. 2018; ISSN 1530-2075
Publisher:
IEEECopyright Statement
Country of Publication:
United States
Language:
English

References (18)

A fast algorithm for particle simulations journal December 1987
A theory of pseudoskeleton approximations journal August 1997
Far-field compression for fast kernel summation methods in high dimensions journal July 2017
A hierarchical O(N log N) force-calculation algorithm journal December 1986
Improved Approximation Algorithms for Large Matrices via Random Projections
  • Sarlos, Tamas
  • 2006 47th Annual IEEE Conference on Foundations of Computer Science, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06) https://doi.org/10.1109/FOCS.2006.37
conference October 2006
INV-ASKIT: A Parallel Fast Direct Solver for Kernel Matrices conference May 2016
An N log N Parallel Fast Direct Solver for Kernel Matrices conference May 2017
A Robust Parallel Preconditioner for Indefinite Systems Using Hierarchical Matrices and Randomized Sampling conference May 2017
A Butterfly-Based Direct Integral-Equation Solver Using Hierarchical LU Factorization for Analyzing Scattering From Electrically Large Conducting Objects journal September 2017
On the Compression of Low Rank Matrices journal January 2005
Clustered low rank approximation of graphs in information science applications conference December 2013
A Fast Randomized Algorithm for Computing a Hierarchically Semiseparable Representation of a Matrix journal October 2011
ASKIT: Approximate Skeletonization Kernel-Independent Treecode in High Dimensions journal January 2015
A Fast $ULV$ Decomposition Solver for Hierarchically Semiseparable Representations journal January 2006
OpenTuner: an extensible framework for program autotuning
  • Ansel, Jason; Kamil, Shoaib; Veeramachaneni, Kalyan
  • Proceedings of the 23rd international conference on Parallel architectures and compilation - PACT '14 https://doi.org/10.1145/2628071.2628092
conference January 2014
Performance optimization for the k-nearest neighbors kernel on x86 architectures
  • Yu, Chenhan D.; Huang, Jianyu; Austin, Woody
  • SC15: The International Conference for High Performance Computing, Networking, Storage and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/2807591.2807601
conference November 2015
A Distributed-Memory Package for Dense Hierarchically Semi-Separable Matrix Computations Using Randomization journal June 2016
Kernel methods in machine learning journal June 2008

Cited By (1)

Preparing sparse solvers for exascale computing
  • Anzt, Hartwig; Boman, Erik; Falgout, Rob
  • Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 378, Issue 2166 https://doi.org/10.1098/rsta.2019.0053
journal January 2020

Figures / Tables (11)

Figure 1(p. 3)figure Figure 1
Table 1(p. 3)table Table 1
Figure 2(p. 5)figure Figure 2
Figure 3(p. 5)figure Figure 3
Figure 4(p. 6)figure Figure 4
Figure 5(p. 8)figure Figure 5
Table 2(p. 8)table Table 2
Figure 6(p. 9)figure Figure 6
Figure 7(p. 9)figure Figure 7
Table 4(p. 9)table Table 4
Figure 8(p. 10)figure Figure 8

Similar Records

Efficient scalable algorithms for hierarchically semiseparable matrices
Journal Article · Wed Sep 14 00:00:00 EDT 2011 · SIAM J. Scientific Computing · OSTI ID:1052181
STRUMPACK -- STRUctured Matrices PACKage
Software · Mon Dec 01 00:00:00 EST 2014 · OSTI ID:1328126
Approximate l-fold cross-validation with Least Squares SVM and Kernel Ridge Regression
Conference · Mon Dec 31 23:00:00 EST 2012 · OSTI ID:1111451

Related Subjects

97 MATHEMATICS AND COMPUTING