HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks

Azad, Ariful; Pavlopoulos, Georgios A.; Ouzounis, Christos A.; Kyrpides, Nikos C.; Buluc, Aydin

doi:10.1093/nar/gkx1313

Title: HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks

Journal Article · Fri Jan 05 00:00:00 EST 2018 · Nucleic Acids Research

DOI:https://doi.org/10.1093/nar/gkx1313· OSTI ID:1439241

Azad, Ariful ^[1];

^[2]; Ouzounis, Christos A. ^[3]; Kyrpides, Nikos C. ^[2]; Buluc, Aydin ^[4]

Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
USDOE Joint Genome Institute (JGI), Walnut Creek, CA (United States)
Centre for Research & Technology Hellas, Thessalonica (Greece). Biological Computation & Process Lab. Chemical Process & Energy Resources Inst.
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division; Univ. of California, Berkeley, CA (United States). Dept. of Electrical Engineering and Computer Sciences

Biological networks capture structural or functional properties of relevant entities such as molecules, proteins or genes. Characteristic examples are gene expression networks or protein–protein interaction networks, which hold information about functional affinities or structural similarities. Such networks have been expanding in size due to increasing scale and abundance of biological data. While various clustering algorithms have been proposed to find highly connected regions, Markov Clustering (MCL) has been one of the most successful approaches to cluster sequence similarity or expression networks. Despite its popularity, MCL’s scalability to cluster large datasets still remains a bottleneck due to high running times and memory demands. In this paper, we present High-performance MCL (HipMCL), a parallel implementation of the original MCL algorithm that can run on distributed-memory computers. We show that HipMCL can efficiently utilize 2000 compute nodes and cluster a network of ~70 million nodes with ~68 billion edges in ~2.4 h. By exploiting distributed-memory environments, HipMCL clusters large-scale networks several orders of magnitude faster than MCL and enables clustering of even bigger networks. Finally, HipMCL is based on MPI and OpenMP and is freely available under a modified BSD license.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1439241

Journal Information:: Nucleic Acids Research, Vol. 46, Issue 6; ISSN 0305-1048

Publisher:: Oxford University PressCopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 62 works

Citation information provided by
Web of Science

References (52)

A survey of visualization tools for biological network analysis Pavlopoulos, Georgios A.; Wegener, Anna-Lynn; Schneider, Reinhard BioData Mining, Vol. 1, Issue 1 https://doi.org/10.1186/1756-0381-1-12	journal	November 2008
Uncovering the overlapping community structure of complex networks in nature and society Palla, Gergely; Derényi, Imre; Farkas, Illés Nature, Vol. 435, Issue 7043 https://doi.org/10.1038/nature03607	journal	June 2005
SNAP: A General-Purpose Network Analysis and Graph-Mining Library Leskovec, Jure; Sosič, Rok ACM Transactions on Intelligent Systems and Technology, Vol. 8, Issue 1 https://doi.org/10.1145/2898361	journal	July 2016
Fast unfolding of communities in large networks Blondel, Vincent D.; Guillaume, Jean-Loup; Lambiotte, Renaud Journal of Statistical Mechanics: Theory and Experiment, Vol. 2008, Issue 10 https://doi.org/10.1088/1742-5468/2008/10/P10008	journal	October 2008
Time bounds for selection Blum, Manuel; Floyd, Robert W.; Pratt, Vaughan Journal of Computer and System Sciences, Vol. 7, Issue 4 https://doi.org/10.1016/S0022-0000(73)80033-9	journal	August 1973
Clustering by Passing Messages Between Data Points Frey, B. J.; Dueck, D. Science, Vol. 315, Issue 5814 https://doi.org/10.1126/science.1136800	journal	February 2007
Detection of functional modules from protein interaction networks Pereira-Leal, Jose B.; Enright, Anton J.; Ouzounis, Christos A. Proteins: Structure, Function, and Bioinformatics, Vol. 54, Issue 1 https://doi.org/10.1002/prot.10505	journal	September 2003
SUMMA: scalable universal matrix multiplication algorithm Van De Geijn, R. A.; Watts, J. Concurrency: Practice and Experience, Vol. 9, Issue 4 https://doi.org/10.1002/(SICI)1096-9128(199704)9:4<255::AID-CPE250>3.0.CO;2-2	journal	April 1997
Domain enhanced lookup time accelerated BLAST Boratyn, Grzegorz M.; Schäffer, Alejandro A.; Agarwala, Richa Biology Direct, Vol. 7, Issue 1 https://doi.org/10.1186/1745-6150-7-12	journal	January 2012
SWORD—a highly efficient protein database search Vaser, Robert; Pavlović, Dario; Šikić, Mile Bioinformatics, Vol. 32, Issue 17 https://doi.org/10.1093/bioinformatics/btw445	journal	September 2016
CoGenT++: an extensive and extensible data environment for computational genomics Goldovsky, L.; Janssen, P.; Ahren, D. Bioinformatics, Vol. 21, Issue 19 https://doi.org/10.1093/bioinformatics/bti579	journal	October 2005
Parallel Sparse Matrix-Matrix Multiplication and Indexing: Implementation and Experiments Buluç, Aydin; Gilbert, John R. SIAM Journal on Scientific Computing, Vol. 34, Issue 4 https://doi.org/10.1137/110848244	journal	January 2012
Graph Clustering Via a Discrete Uncoupling Process Van Dongen, Stijn SIAM Journal on Matrix Analysis and Applications, Vol. 30, Issue 1 https://doi.org/10.1137/040608635	journal	January 2008
Markov clustering versus affinity propagation for the partitioning of protein interaction graphs Vlasblom, James; Wodak, Shoshana J. BMC Bioinformatics, Vol. 10, Issue 1 https://doi.org/10.1186/1471-2105-10-99	journal	March 2009
jClust: a clustering and visualization toolbox Pavlopoulos, G. A.; Moschopoulos, C. N.; Hooper, S. D. Bioinformatics, Vol. 25, Issue 15 https://doi.org/10.1093/bioinformatics/btp330	journal	May 2009
Microbiome Data Science: Understanding Our Microbial Planet Kyrpides, Nikos C.; Eloe-Fadrosh, Emiley A.; Ivanova, Natalia N. Trends in Microbiology, Vol. 24, Issue 6 https://doi.org/10.1016/j.tim.2016.02.011	journal	June 2016
BSW: FPGA-accelerated BLAST-Wrapped Smith-Waterman aligner Lam, Bryant C.; Pascoe, Carlo; Schaecher, Scott 2013 International Conference on ReConFigurable Computing and FPGAs (ReConFig) https://doi.org/10.1109/ReConFig.2013.6732273	conference	December 2013
Adaptive seeds tame genomic sequence comparison Kielbasa, S. M.; Wan, R.; Sato, K. Genome Research, Vol. 21, Issue 3 https://doi.org/10.1101/gr.113985.110	journal	January 2011
Parallel Reproducible Summation Demmel, James; Nguyen, Hong Diep IEEE Transactions on Computers, Vol. 64, Issue 7 https://doi.org/10.1109/TC.2014.2345391	journal	July 2015
Classification schemes for protein structure and function Ouzounis, Christos A.; Coulson, Richard M. R.; Enright, Anton J. Nature Reviews Genetics, Vol. 4, Issue 7 https://doi.org/10.1038/nrg1113	journal	July 2003
Comparing the performance of biomedical clustering methods Wiwie, Christian; Baumbach, Jan; Röttger, Richard Nature Methods, Vol. 12, Issue 11 https://doi.org/10.1038/nmeth.3583	journal	September 2015
Empirical Comparison of Visualization Tools for Larger-Scale Network Analysis Pavlopoulos, Georgios A.; Paez-Espino, David; Kyrpides, Nikos C. Advances in Bioinformatics, Vol. 2017 https://doi.org/10.1155/2017/1278932	journal	January 2017
SPICi: a fast clustering algorithm for large biological networks Jiang, Peng; Singh, Mona Bioinformatics, Vol. 26, Issue 8 https://doi.org/10.1093/bioinformatics/btq078	journal	February 2010
Visualizing genome and systems biology: technologies, tools, implementation techniques and trends, past, present and future Pavlopoulos, Georgios A.; Malliarakis, Dimitris; Papanikolaou, Nikolas GigaScience, Vol. 4, Issue 1 https://doi.org/10.1186/s13742-015-0077-2	journal	August 2015
Cytoscape: A Software Environment for Integrated Models of Biomolecular Interaction Networks Shannon, P. Genome Research, Vol. 13, Issue 11 https://doi.org/10.1101/gr.1239303	journal	November 2003
clusterMaker: a multi-algorithm clustering plugin for Cytoscape Morris, John H.; Apeltsin, Leonard; Newman, Aaron M. BMC Bioinformatics, Vol. 12, Issue 1 https://doi.org/10.1186/1471-2105-12-436	journal	November 2011
Network biology: understanding the cell's functional organization Barabási, Albert-László; Oltvai, Zoltán N. Nature Reviews Genetics, Vol. 5, Issue 2 https://doi.org/10.1038/nrg1272	journal	February 2004
Evaluation of clustering algorithms for protein-protein interaction networks Brohée, Sylvain; van Helden, Jacques BMC Bioinformatics, Vol. 7, Issue 1 https://doi.org/10.1186/1471-2105-7-488	journal	November 2006
NAP: The Network Analysis Profiler, a web tool for easier topological analysis and comparison of medium-scale biological networks Theodosiou, Theodosios; Efstathiou, Georgios; Papanikolaou, Nikolas BMC Research Notes, Vol. 10, Issue 1 https://doi.org/10.1186/s13104-017-2607-8	journal	July 2017
Using graph theory to analyze biological networks Pavlopoulos, Georgios A.; Secrier, Maria; Moschopoulos, Charalampos N. BioData Mining, Vol. 4, Issue 1 https://doi.org/10.1186/1756-0381-4-10	journal	April 2011
Superparamagnetic Clustering of Data Blatt, Marcelo; Wiseman, Shai; Domany, Eytan Physical Review Letters, Vol. 76, Issue 18 https://doi.org/10.1103/PhysRevLett.76.3251	journal	April 1996
An efficient algorithm for large-scale detection of protein families Enright, A. J. Nucleic Acids Research, Vol. 30, Issue 7 https://doi.org/10.1093/nar/30.7.1575	journal	April 2002
Medusa: A tool for exploring and clustering biological networks Pavlopoulos, Georgios A.; Hooper, Sean D.; Sifrim, Alejandro BMC Research Notes, Vol. 4, Issue 1 https://doi.org/10.1186/1756-0500-4-384	journal	October 2011
A Genomic Perspective on Protein Families Tatusov, R. L. Science, Vol. 278, Issue 5338 https://doi.org/10.1126/science.278.5338.631	journal	October 1997
Protein complex prediction via cost-based clustering King, A. D.; Przulj, N.; Jurisica, I. Bioinformatics, Vol. 20, Issue 17 https://doi.org/10.1093/bioinformatics/bth351	journal	June 2004
IMG/M: integrated genome and metagenome comparative data analysis system Chen, I-Min A.; Markowitz, Victor M.; Chu, Ken Nucleic Acids Research, Vol. 45, Issue D1 https://doi.org/10.1093/nar/gkw929	journal	October 2016
CALU: A Communication Optimal LU Factorization Algorithm Grigori, Laura; Demmel, James W.; Xiang, Hua SIAM Journal on Matrix Analysis and Applications, Vol. 32, Issue 4 https://doi.org/10.1137/100788926	journal	October 2011
Efficient algorithms for accurate hierarchical clustering of huge datasets: tackling the entire protein space Loewenstein, Y.; Portugaly, E.; Fromer, M. Bioinformatics, Vol. 24, Issue 13 https://doi.org/10.1093/bioinformatics/btn174	journal	June 2008
The Combinatorial BLAS: design, implementation, and applications Buluç, Aydın; Gilbert, John R. The International Journal of High Performance Computing Applications, Vol. 25, Issue 4 https://doi.org/10.1177/1094342011403516	journal	May 2011
Construction, Visualisation, and Clustering of Transcription Networks from Microarray Expression Data Freeman, Tom C.; Goldovsky, Leon; Brosch, Markus PLoS Computational Biology, Vol. 3, Issue 10 https://doi.org/10.1371/journal.pcbi.0030206	journal	January 2007
Which clustering algorithm is better for predicting protein complexes? Moschopoulos, Charalampos N.; Pavlopoulos, Georgios A.; Iacucci, Ernesto BMC Research Notes, Vol. 4, Issue 1 https://doi.org/10.1186/1756-0500-4-549	journal	December 2011
Dynamics of collective action to conserve a large common-pool resource Andersson, David; Bratsberg, Sigrid; Ringsmuth, Andrew K. Scientific Reports, Vol. 11, Issue 1 https://doi.org/10.1038/s41598-021-87109-x	journal	April 2021
Superparamagnetic clustering of data Wiseman, Shai; Blatt, Marcelo; Domany, Eytan Physical Review E, Vol. 57, Issue 4 https://doi.org/10.1103/physreve.57.3767	journal	April 1998
Visualizing genome and systems biology: technologies, tools, implementation techniques and trends, past, present and future. Pavlopoulos, Georgios A.; Malliarakis, Dimitris; Papanikolaou, Nikolas Apollo - University of Cambridge Repository https://doi.org/10.17863/cam.17912	text	January 2015
Construction, visualisation, and clustering of transcription networks from microarray expression data. Freeman, Tom C.; Goldovsky, Leon; Brosch, Markus Apollo - University of Cambridge Repository https://doi.org/10.17863/cam.17926	text	January 2007
Parallel Sparse Matrix-Matrix Multiplication and Indexing: Implementation and Experiments Buluc, Aydin; Gilbert, John arXiv https://doi.org/10.48550/arxiv.1109.3739	text	January 2011
SNAP: A General Purpose Network Analysis and Graph Mining Library Leskovec, Jure; Sosic, Rok arXiv https://doi.org/10.48550/arxiv.1606.07550	preprint	January 2016
Basic local alignment search tool Altschul, Stephen F.; Gish, Warren; Miller, Webb Journal of Molecular Biology, Vol. 215, Issue 3, p. 403-410 https://doi.org/10.1016/s0022-2836(05)80360-2	journal	October 1990
H-BLAST: a fast protein sequence alignment toolkit on heterogeneous computers with GPUs Ye, Weicai; Chen, Ying; Zhang, Yongdong Bioinformatics, Vol. 33, Issue 8 https://doi.org/10.1093/bioinformatics/btw769	journal	January 2017
New Connectivity and MSF Algorithms for Shuffle-Exchange Network and PRAM IEEE Transactions on Computers, Vol. C-36, Issue 10 https://doi.org/10.1109/tc.1987.1676869	journal	October 1987
Fast Parallel Markov Clustering in Bioinformatics Using Massively Parallel Computing on GPU with CUDA and ELLPACK-R Sparse Format Bustamam, A.; Burrage, K.; Hamilton, N. A. IEEE/ACM Transactions on Computational Biology and Bioinformatics, Vol. 9, Issue 3 https://doi.org/10.1109/tcbb.2011.68	journal	May 2012
Survey of Clustering Algorithms Xu, R.; WunschII, D. IEEE Transactions on Neural Networks, Vol. 16, Issue 3 https://doi.org/10.1109/tnn.2005.845141	journal	May 2005

Cited By (11)

Register-Aware Optimizations for Parallel Sparse Matrix–Matrix Multiplication Liu, Junhong; He, Xin; Liu, Weifeng International Journal of Parallel Programming, Vol. 47, Issue 3 https://doi.org/10.1007/s10766-018-0604-8	journal	January 2019
A novel parallel Markov clustering method in biological interaction network analysis under multi-GPU computing environment Fu, You; Zhou, Wei The Journal of Supercomputing, Vol. 76, Issue 10 https://doi.org/10.1007/s11227-020-03193-2	journal	February 2020
The parallelism motifs of genomic data analysis Yelick, Katherine; Buluç, Aydın; Awan, Muaaz Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 378, Issue 2166 https://doi.org/10.1098/rsta.2019.0394	journal	January 2020
RAFTS3G: an efficient and versatile clustering software to analyses in large protein datasets de Lima Nichio, Bruno Thiago; de Oliveira, Aryel Marlus Repula; de Pierri, Camilla Reginatto BMC Bioinformatics, Vol. 20, Issue 1 https://doi.org/10.1186/s12859-019-2973-4	journal	July 2019
Self-analysis of repeat proteins reveals evolutionarily conserved patterns Merski, Matthew; Młynarczyk, Krzysztof; Ludwiczak, Jan BMC Bioinformatics, Vol. 21, Issue 1 https://doi.org/10.1186/s12859-020-3493-y	journal	May 2020
JGI QC impact on assembly, binning, phylogenomics, and functional analysis Whitham, Jason North Carolina State Univ., Raleigh, NC (United States) https://doi.org/10.25982/62657.1515/1779219	dataset	January 2021
Impact of BBDuk metagenomic read trimming and decontamination Whitham, Jason North Carolina State Univ., Raleigh, NC (United States) https://doi.org/10.25982/77705.1341/1779218	dataset	January 2021
Developing computational biology at meridian 23° E, and a little eastwards Ouzounis, Christos A. Journal of Biological Research-Thessaloniki, Vol. 25, Issue 1 https://doi.org/10.1186/s40709-018-0091-5	journal	November 2018
Systematical Identification of Breast Cancer-Related Circular RNA Modules for Deciphering circRNA Functions Based on the Non-Negative Matrix Factorization Algorithm Wang, Shuyuan; Xia, Peng; Zhang, Li International Journal of Molecular Sciences, Vol. 20, Issue 4 https://doi.org/10.3390/ijms20040919	journal	February 2019
Optimizing High Performance Markov Clustering for Pre-Exascale Architectures Selvitopi, Oguz; Hussain, Md Taufique; Azad, Ariful arXiv https://doi.org/10.48550/arxiv.2002.10083	text	January 2020
Communication-Avoiding and Memory-Constrained Sparse Matrix-Matrix Multiplication at Extreme Scale Hussain, Md Taufique; Selvitopi, Oguz; Buluç, Aydin arXiv https://doi.org/10.48550/arxiv.2010.08526	preprint	January 2020

Figures / Tables (7)

Similar Records

Optimizing High Performance Markov Clustering for Pre-Exascale Architectures

Journal Article · Fri May 01 00:00:00 EDT 2020 · Proceedings - IEEE International Parallel and Distributed Processing Symposium (IPDPS) · OSTI ID:1439241

Selvitopi, Oguz; Hussain, Md Taufique; Azad, Ariful; +1 more

Multi-node and Multi-core Performance Studies of a Monte Carlo code RMC

Journal Article · Wed Jun 15 00:00:00 EDT 2016 · Transactions of the American Nuclear Society · OSTI ID:1439241

Yang, Feng; Yu, GangLin; Liang, JinGang; +1 more

Roofline Analysis in the Intel® Advisor to Deliver Optimized Performance for applications on Intel® Xeon Phi™ Processor

Conference · Tue May 23 00:00:00 EDT 2017 · OSTI ID:1439241

Koskela, Tuomas S.; Lobet, Mathieu; Deslippe, Jack; +1 more

Related Subjects

97 MATHEMATICS AND COMPUTING
59 BASIC BIOLOGICAL SCIENCES
computational methods
genomics