ParChain: a framework for parallel hierarchical agglomerative clustering using nearest-neighbor chain

Yu, Shangdi; Wang, Yiqiu; Gu, Yan; Dhulipala, Laxman; Shun, Julian

doi:10.14778/3489496.3489509

ParChain: a framework for parallel hierarchical agglomerative clustering using nearest-neighbor chain

Journal Article · Fri Oct 01 00:00:00 EDT 2021 · Proceedings of the VLDB Endowment

DOI:https://doi.org/10.14778/3489496.3489509· OSTI ID:1980994

Yu, Shangdi ^[1]; Wang, Yiqiu ^[2]; Gu, Yan ^[3]; Dhulipala, Laxman ^[2]; Shun, Julian ^[2]

MIT CSAIL; OSTI
MIT CSAIL
UC Riverside

This paper studies the hierarchical clustering problem, where the goal is to produce a dendrogram that represents clusters at varying scales of a data set. We propose the ParChain framework for designing parallel hierarchical agglomerative clustering (HAC) algorithms, and using the framework we obtain novel parallel algorithms for the complete linkage, average linkage, and Ward's linkage criteria. Compared to most previous parallel HAC algorithms, which require quadratic memory, our new algorithms require only linear memory, and are scalable to large data sets. ParChain is based on our parallelization of the nearest-neighbor chain algorithm, and enables multiple clusters to be merged on every round. We introduce two key optimizations that are critical for efficiency: a range query optimization that reduces the number of distance computations required when finding nearest neighbors of clusters, and a caching optimization that stores a subset of previously computed distances, which are likely to be reused.

Experimentally, we show that our highly-optimized implementations using 48 cores with two-way hyper-threading achieve 5.8--110.1x speedup over state-of-the-art parallel HAC algorithms and achieve 13.75--54.23x self-relative speedup. Compared to state-of-the-art algorithms, our algorithms require up to 237.3x less space. Our algorithms are able to scale to data set sizes with tens of millions of points, which existing algorithms are not able to handle.

Research Organization:: Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

DOE Contract Number:: SC0018947

OSTI ID:: 1980994

Journal Information:: Proceedings of the VLDB Endowment, Journal Name: Proceedings of the VLDB Endowment Journal Issue: 2 Vol. 15; ISSN 2150-8097

Publisher:: Association for Computing Machinery (ACM)

Country of Publication:: United States

Language:: English

References (43)

Scalable Hierarchical Clustering with Tree Grafting Monath, Nicholas; Kobren, Ari; Krishnamurthy, Akshay Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining https://doi.org/10.1145/3292500.3330929	conference	July 2019
Reducing contention through priority updates Shun, Julian; Blelloch, Guy E.; Fineman, Jeremy T. Proceedings of the twenty-fifth annual ACM symposium on Parallelism in algorithms and architectures https://doi.org/10.1145/2486159.2486189	conference	July 2013
Parallel algorithms for hierarchical clustering Olson, Clark F. Parallel Computing, Vol. 21, Issue 8 https://doi.org/10.1016/0167-8191(95)00017-I	journal	August 1995
Efficient algorithms for accurate hierarchical clustering of huge datasets: tackling the entire protein space Loewenstein, Y.; Portugaly, E.; Fromer, M. Bioinformatics, Vol. 24, Issue 13 https://doi.org/10.1093/bioinformatics/btn174	journal	June 2008
Parallel algorithms for hierarchical clustering and cluster validity Li, X. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 12, Issue 11 https://doi.org/10.1109/34.61708	journal	January 1990
A Comprehensive Survey of Clustering Algorithms Xu, Dongkuan; Tian, Yingjie Annals of Data Science, Vol. 2, Issue 2 https://doi.org/10.1007/s40745-015-0040-1	journal	June 2015
Performance guarantees for hierarchical clustering Dasgupta, Sanjoy; Long, Philip M. Journal of Computer and System Sciences, Vol. 70, Issue 4 https://doi.org/10.1016/j.jcss.2004.10.006	journal	June 2005
Fast Parallel Algorithms for Euclidean Minimum Spanning Tree and Hierarchical Spatial Clustering Wang, Yiqiu; Yu, Shangdi; Gu, Yan Proceedings of the 2021 International Conference on Management of Data https://doi.org/10.1145/3448016.3457296	conference	June 2021
A General Theory of Classificatory Sorting Strategies: 1. Hierarchical Systems Lance, G. N.; Williams, W. T. The Computer Journal, Vol. 9, Issue 4 https://doi.org/10.1093/comjnl/9.4.373	journal	February 1967
Hierarchical Grouping to Optimize an Objective Function Ward, Joe H. Journal of the American Statistical Association, Vol. 58, Issue 301 https://doi.org/10.1080/01621459.1963.10500845	journal	March 1963
The Cilk++ concurrency platform Leiserson, Charles E. The Journal of Supercomputing, Vol. 51, Issue 3 https://doi.org/10.1007/s11227-010-0405-3	journal	March 2010
Multi-Threaded Hierarchical Clustering by Parallel Nearest-Neighbor Chaining Jeon, Yongkweon; Yoon, Sungroh IEEE Transactions on Parallel and Distributed Systems, Vol. 26, Issue 9 https://doi.org/10.1109/TPDS.2014.2355205	journal	September 2015
Optimal implementations of UPGMA and other common clustering algorithms Gronau, Ilan; Moran, Shlomo Information Processing Letters, Vol. 104, Issue 6 https://doi.org/10.1016/j.ipl.2007.07.002	journal	December 2007
Efficient Clustering and Matching for Object Class Recognition Leibe, B.; Mikolajczyk, K.; Schiele, B. Procedings of the British Machine Vision Conference 2006 https://doi.org/10.5244/C.20.81	conference	January 2006
An efficient algorithm for a complete link method Defays, D. The Computer Journal, Vol. 20, Issue 4 https://doi.org/10.1093/comjnl/20.4.364	journal	April 1977
Correlation, hierarchies, and networks in financial markets Tumminello, Michele; Lillo, Fabrizio; Mantegna, Rosario N. Journal of Economic Behavior & Organization, Vol. 75, Issue 1 https://doi.org/10.1016/j.jebo.2010.01.004	journal	July 2010
Parallel clustering algorithms Li, Xiaobo; Fang, Zhixi Parallel Computing, Vol. 11, Issue 3 https://doi.org/10.1016/0167-8191(89)90036-7	journal	August 1989
DHC: A Distributed Hierarchical Clustering Algorithm for Large Datasets Zhang, Wei; Zhang, Gongxuan; Chen, Xiaohui Journal of Circuits, Systems and Computers, Vol. 28, Issue 04 https://doi.org/10.1142/S0218126619500658	journal	March 2019
ConnectIt Dhulipala, Laxman; Hong, Changwan; Shun, Julian Proceedings of the VLDB Endowment, Vol. 14, Issue 4 https://doi.org/10.14778/3436905.3436923	journal	December 2020
A novel parallelization approach for hierarchical clustering Du, Z.; Lin, F. Parallel Computing, Vol. 31, Issue 5 https://doi.org/10.1016/j.parco.2005.01.001	journal	May 2005
Methods for multidimensional event classification: a case study using images from a Cherenkov gamma-ray telescope Bock, R. K.; Chilingarian, A.; Gaug, M. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 516, Issue 2-3 https://doi.org/10.1016/j.nima.2003.08.157	journal	January 2004
Twister Tries Cochez, Michael; Mou, Hao Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data https://doi.org/10.1145/2723372.2751521	conference	May 2015
fast_protein_cluster: parallel and optimized clustering of large-scale protein modeling data Hung, Ling-Hong; Samudrala, Ram Bioinformatics, Vol. 30, Issue 12 https://doi.org/10.1093/bioinformatics/btu098	journal	February 2014
Fast reciprocal nearest neighbors clustering López-Sastre, Roberto J.; Oñoro-Rubio, Daniel; Gil-Jiménez, Pedro Signal Processing, Vol. 92, Issue 1 https://doi.org/10.1016/j.sigpro.2011.07.023	journal	January 2012
Valgrind: a framework for heavyweight dynamic binary instrumentation Nethercote, Nicholas; Seward, Julian ACM SIGPLAN Notices, Vol. 42, Issue 6 https://doi.org/10.1145/1273442.1250746	journal	June 2007
ESPRIT: estimating species richness using large collections of 16S rRNA pyrosequences Sun, Yijun; Cai, Yunpeng; Liu, Li Nucleic Acids Research, Vol. 37, Issue 10 https://doi.org/10.1093/nar/gkp285	journal	May 2009
Efficient parallel hierarchical clustering algorithms Rajasekaran, S. IEEE Transactions on Parallel and Distributed Systems, Vol. 16, Issue 6 https://doi.org/10.1109/TPDS.2005.72	journal	June 2005
Hierarchical Clustering Cohen-addad, Vincent; Kanade, Varun; Mallmann-trenn, Frederik Journal of the ACM, Vol. 66, Issue 4 https://doi.org/10.1145/3321386	journal	June 2019
Algorithms for hierarchical clustering: an overview, II Murtagh, Fionn; Contreras, Pedro WIREs Data Mining and Knowledge Discovery, Vol. 7, Issue 6 https://doi.org/10.1002/widm.1219	journal	September 2017
A cost function for similarity-based hierarchical clustering Dasgupta, Sanjoy Proceedings of the forty-eighth annual ACM symposium on Theory of Computing https://doi.org/10.1145/2897518.2897527	conference	June 2016
Online decorrelation of humidity and temperature in chemical sensors for continuous monitoring Huerta, Ramon; Mosqueiro, Thiago; Fonollosa, Jordi Chemometrics and Intelligent Laboratory Systems, Vol. 157 https://doi.org/10.1016/j.chemolab.2016.07.004	journal	October 2016
Fast euclidean minimum spanning tree March, William B.; Ram, Parikshit; Gray, Alexander G. Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining https://doi.org/10.1145/1835804.1835882	conference	July 2010
SparseHC: A Memory-efficient Online Hierarchical Clustering Algorithm Nguyen, Thuy-Diem; Schmidt, Bertil; Kwoh, Chee-Keong Procedia Computer Science, Vol. 29 https://doi.org/10.1016/j.procs.2014.05.001	journal	January 2014
Data clustering: a review Jain, A. K.; Murty, M. N.; Flynn, P. J. ACM Computing Surveys, Vol. 31, Issue 3, p. 264-323 https://doi.org/10.1145/331499.331504	journal	September 1999
Learning transportation mode from raw gps data for geographic applications on the web Zheng, Yu; Liu, Like; Wang, Longhao Proceedings of the 17th international conference on World Wide Web https://doi.org/10.1145/1367497.1367532	conference	April 2008
Cluster analysis and display of genome-wide expression patterns Eisen, M. B.; Spellman, P. T.; Brown, P. O. Proceedings of the National Academy of Sciences, Vol. 95, Issue 25 https://doi.org/10.1073/pnas.95.25.14863	journal	December 1998
Optimal algorithms for complete linkage clustering in d dimensions Krznaric, Drago; Levcopoulos, Christos Theoretical Computer Science, Vol. 286, Issue 1 https://doi.org/10.1016/S0304-3975(01)00239-0	journal	September 2002
A Survey of Recent Advances in Hierarchical Clustering Algorithms Murtagh, F. The Computer Journal, Vol. 26, Issue 4 https://doi.org/10.1093/comjnl/26.4.354	journal	November 1983
Fast approximate hierarchical clustering using similarity heuristics Kull, Meelis; Vilo, Jaak BioData Mining, Vol. 1, Issue 1 https://doi.org/10.1186/1756-0381-1-9	journal	September 2008
Efficient hierarchical clustering of large high dimensional datasets Gilpin, Sean; Qian, Buyue; Davidson, Ian Proceedings of the 22nd ACM international conference on Conference on information & knowledge management - CIKM '13 https://doi.org/10.1145/2505515.2505527	conference	January 2013
fastcluster : Fast Hierarchical, Agglomerative Clustering Routines for R and Python Müllner, Daniel Journal of Statistical Software, Vol. 53, Issue 9 https://doi.org/10.18637/jss.v053.i09	journal	January 2013
Reservoir computing compensates slow response of chemosensor arrays exposed to fast varying gas concentrations in continuous monitoring Fonollosa, Jordi; Sheik, Sadique; Huerta, Ramón Sensors and Actuators B: Chemical, Vol. 215 https://doi.org/10.1016/j.snb.2015.03.028	journal	August 2015
SciPy 1.0: fundamental algorithms for scientific computing in Python Virtanen, Pauli; Gommers, Ralf; Oliphant, Travis E. Nature Methods https://doi.org/10.1038/s41592-019-0686-2	journal	February 2020

Similar Records

Delaunay walk for fast nearest neighbor: accelerating correspondence matching for ICP

Journal Article · Mon Feb 14 23:00:00 EST 2022 · Machine Vision and Applications · OSTI ID:1981469

PANDORA: A Parallel Dendrogram Construction Algorithm for Single Linkage Clustering on GPU

Conference · Thu Aug 01 00:00:00 EDT 2024 · OSTI ID:2438688

Theoretically and practically efficient parallel nucleus decomposition

Journal Article · Mon Nov 01 00:00:00 EDT 2021 · Proceedings of the VLDB Endowment · OSTI ID:1980995

Related Subjects

Computer Science

ParChain: a framework for parallel hierarchical agglomerative clustering using nearest-neighbor chain

Citation Formats

References (43)

Similar Records

Related Subjects