ParChain: a framework for parallel hierarchical agglomerative clustering using nearest-neighbor chain

Yu, Shangdi; Wang, Yiqiu; Gu, Yan; Dhulipala, Laxman; Shun, Julian

doi:10.14778/3489496.3489509

ParChain: a framework for parallel hierarchical agglomerative clustering using nearest-neighbor chain

Journal Article · 01 October 2021 · Proceedings of the VLDB Endowment

DOI:https://doi.org/10.14778/3489496.3489509· OSTI ID:1980994

Yu, Shangdi ^[1]; Wang, Yiqiu ^[1]; Gu, Yan ^[2]; Dhulipala, Laxman ^[1]; Shun, Julian ^[1]

MIT CSAIL
UC Riverside

This paper studies the hierarchical clustering problem, where the goal is to produce a dendrogram that represents clusters at varying scales of a data set. We propose the ParChain framework for designing parallel hierarchical agglomerative clustering (HAC) algorithms, and using the framework we obtain novel parallel algorithms for the complete linkage, average linkage, and Ward's linkage criteria. Compared to most previous parallel HAC algorithms, which require quadratic memory, our new algorithms require only linear memory, and are scalable to large data sets. ParChain is based on our parallelization of the nearest-neighbor chain algorithm, and enables multiple clusters to be merged on every round. We introduce two key optimizations that are critical for efficiency: a range query optimization that reduces the number of distance computations required when finding nearest neighbors of clusters, and a caching optimization that stores a subset of previously computed distances, which are likely to be reused.

Experimentally, we show that our highly-optimized implementations using 48 cores with two-way hyper-threading achieve 5.8--110.1x speedup over state-of-the-art parallel HAC algorithms and achieve 13.75--54.23x self-relative speedup. Compared to state-of-the-art algorithms, our algorithms require up to 237.3x less space. Our algorithms are able to scale to data set sizes with tens of millions of points, which existing algorithms are not able to handle.

Research Organization:: Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

DOE Contract Number:: SC0018947

OSTI ID:: 1980994

Journal Information:: Proceedings of the VLDB Endowment, Vol. 15, Issue 2; ISSN 2150-8097

Publisher:: Association for Computing Machinery (ACM)

Country of Publication:: United States

Language:: English

References (43)

Methods for multidimensional event classification: a case study using images from a Cherenkov gamma-ray telescope No authors listed Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment https://doi.org/10.1016/j.nima.2003.08.157		January 2004
Twister Tries No authors listed Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data https://doi.org/10.1145/2723372.2751521		May 2015
Hierarchical Clustering No authors listed Journal of the ACM https://doi.org/10.1145/3321386		June 2019
A cost function for similarity-based hierarchical clustering No authors listed Proceedings of the forty-eighth annual ACM symposium on Theory of Computing https://doi.org/10.1145/2897518.2897527		June 2016
Performance guarantees for hierarchical clustering No authors listed Journal of Computer and System Sciences https://doi.org/10.1016/j.jcss.2004.10.006		June 2005
An efficient algorithm for a complete link method No authors listed The Computer Journal https://doi.org/10.1093/comjnl/20.4.364		April 1977
ConnectIt No authors listed Proceedings of the VLDB Endowment https://doi.org/10.14778/3436905.3436923		December 2020
A novel parallelization approach for hierarchical clustering No authors listed Parallel Computing https://doi.org/10.1016/j.parco.2005.01.001		May 2005
Cluster analysis and display of genome-wide expression patterns No authors listed Proceedings of the National Academy of Sciences https://doi.org/10.1073/pnas.95.25.14863		December 1998
Reservoir computing compensates slow response of chemosensor arrays exposed to fast varying gas concentrations in continuous monitoring No authors listed Sensors and Actuators B: Chemical https://doi.org/10.1016/j.snb.2015.03.028		August 2015
Efficient hierarchical clustering of large high dimensional datasets No authors listed Proceedings of the 22nd ACM international conference on Conference on information & knowledge management - CIKM '13 https://doi.org/10.1145/2505515.2505527		January 2013
Optimal implementations of UPGMA and other common clustering algorithms No authors listed Information Processing Letters https://doi.org/10.1016/j.ipl.2007.07.002		December 2007
Online decorrelation of humidity and temperature in chemical sensors for continuous monitoring No authors listed Chemometrics and Intelligent Laboratory Systems https://doi.org/10.1016/j.chemolab.2016.07.004		October 2016
fast_protein_cluster: parallel and optimized clustering of large-scale protein modeling data No authors listed Bioinformatics https://doi.org/10.1093/bioinformatics/btu098		February 2014
Data clustering: a review No authors listed ACM Computing Surveys https://doi.org/10.1145/331499.331504		September 1999
Multi-Threaded Hierarchical Clustering by Parallel Nearest-Neighbor Chaining No authors listed IEEE Transactions on Parallel and Distributed Systems https://doi.org/10.1109/TPDS.2014.2355205		September 2015
Optimal algorithms for complete linkage clustering in d dimensions No authors listed Theoretical Computer Science https://doi.org/10.1016/S0304-3975(01)00239-0		September 2002
Fast approximate hierarchical clustering using similarity heuristics No authors listed BioData Mining https://doi.org/10.1186/1756-0381-1-9		September 2008
A General Theory of Classificatory Sorting Strategies: 1. Hierarchical Systems No authors listed The Computer Journal https://doi.org/10.1093/comjnl/9.4.373		February 1967
Efficient Clustering and Matching for Object Class Recognition No authors listed Procedings of the British Machine Vision Conference 2006 https://doi.org/10.5244/C.20.81		January 2006
The Cilk++ concurrency platform No authors listed The Journal of Supercomputing https://doi.org/10.1007/s11227-010-0405-3		March 2010
Parallel algorithms for hierarchical clustering and cluster validity No authors listed IEEE Transactions on Pattern Analysis and Machine Intelligence https://doi.org/10.1109/34.61708		January 1990
Parallel clustering algorithms No authors listed Parallel Computing https://doi.org/10.1016/0167-8191(89)90036-7		August 1989
Efficient algorithms for accurate hierarchical clustering of huge datasets: tackling the entire protein space No authors listed Bioinformatics https://doi.org/10.1093/bioinformatics/btn174		June 2008
Fast reciprocal nearest neighbors clustering No authors listed Signal Processing https://doi.org/10.1016/j.sigpro.2011.07.023		January 2012
Fast euclidean minimum spanning tree No authors listed Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining https://doi.org/10.1145/1835804.1835882		July 2010
Scalable Hierarchical Clustering with Tree Grafting No authors listed Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining https://doi.org/10.1145/3292500.3330929		July 2019
fastcluster : Fast Hierarchical, Agglomerative Clustering Routines for R and Python No authors listed Journal of Statistical Software https://doi.org/10.18637/jss.v053.i09		January 2013
A Survey of Recent Advances in Hierarchical Clustering Algorithms No authors listed The Computer Journal https://doi.org/10.1093/comjnl/26.4.354		November 1983
Algorithms for hierarchical clustering: an overview, II No authors listed WIREs Data Mining and Knowledge Discovery https://doi.org/10.1002/widm.1219		September 2017
Valgrind: a framework for heavyweight dynamic binary instrumentation No authors listed ACM SIGPLAN Notices https://doi.org/10.1145/1273442.1250746		June 2007
SparseHC: A Memory-efficient Online Hierarchical Clustering Algorithm No authors listed Procedia Computer Science https://doi.org/10.1016/j.procs.2014.05.001		January 2014
Parallel algorithms for hierarchical clustering No authors listed Parallel Computing https://doi.org/10.1016/0167-8191(95)00017-I		August 1995
Efficient parallel hierarchical clustering algorithms No authors listed IEEE Transactions on Parallel and Distributed Systems https://doi.org/10.1109/TPDS.2005.72		June 2005
Reducing contention through priority updates No authors listed Proceedings of the twenty-fifth annual ACM symposium on Parallelism in algorithms and architectures https://doi.org/10.1145/2486159.2486189		July 2013
ESPRIT: estimating species richness using large collections of 16S rRNA pyrosequences No authors listed Nucleic Acids Research https://doi.org/10.1093/nar/gkp285		May 2009
Correlation, hierarchies, and networks in financial markets No authors listed Journal of Economic Behavior & Organization https://doi.org/10.1016/j.jebo.2010.01.004		July 2010
SciPy 1.0: fundamental algorithms for scientific computing in Python No authors listed Nature Methods https://doi.org/10.1038/s41592-019-0686-2		February 2020
Fast Parallel Algorithms for Euclidean Minimum Spanning Tree and Hierarchical Spatial Clustering No authors listed Proceedings of the 2021 International Conference on Management of Data https://doi.org/10.1145/3448016.3457296		June 2021
Hierarchical Grouping to Optimize an Objective Function No authors listed Journal of the American Statistical Association https://doi.org/10.1080/01621459.1963.10500845		March 1963
A Comprehensive Survey of Clustering Algorithms No authors listed Annals of Data Science https://doi.org/10.1007/s40745-015-0040-1		June 2015
DHC: A Distributed Hierarchical Clustering Algorithm for Large Datasets No authors listed Journal of Circuits, Systems and Computers https://doi.org/10.1142/S0218126619500658		March 2019
Learning transportation mode from raw gps data for geographic applications on the web No authors listed Proceedings of the 17th international conference on World Wide Web https://doi.org/10.1145/1367497.1367532		April 2008

Similar Records

ConnectIt: a framework for static and incremental parallel graph connectivity algorithms

Journal Article · 2020 · Proceedings of the VLDB Endowment · OSTI ID:1852729

Dhulipala, Laxman; Hong, Changwan; Shun, Julian

PANDORA: A Parallel Dendrogram Construction Algorithm for Single Linkage Clustering on GPU

Conference · 2024 · OSTI ID:2438688

Sao, Piyush; Prokopenko, Andrey; Lebrun-Grandie, Damien

Delaunay walk for fast nearest neighbor: accelerating correspondence matching for ICP

Journal Article · 2022 · Machine Vision and Applications · OSTI ID:1981469

Anderson, James D.; Raettig, Ryan M.; Larson, Josh; +3 more

Related Subjects

Computer Science

ParChain: a framework for parallel hierarchical agglomerative clustering using nearest-neighbor chain

Citation Formats

References (43)

Similar Records

Related Subjects