HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks

doi:10.1093/nar/gkx1313

This content will become publicly available on January 5, 2019

Title: HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks

Biological networks capture structural or functional properties of relevant entities such as molecules, proteins or genes. Characteristic examples are gene expression networks or protein–protein interaction networks, which hold information about functional affinities or structural similarities. Such networks have been expanding in size due to increasing scale and abundance of biological data. While various clustering algorithms have been proposed to find highly connected regions, Markov Clustering (MCL) has been one of the most successful approaches to cluster sequence similarity or expression networks. Despite its popularity, MCL’s scalability to cluster large datasets still remains a bottleneck due to high running times and memory demands. In this paper, we present High-performance MCL (HipMCL), a parallel implementation of the original MCL algorithm that can run on distributed-memory computers. We show that HipMCL can efficiently utilize 2000 compute nodes and cluster a network of ~70 million nodes with ~68 billion edges in ~2.4 h. By exploiting distributed-memory environments, HipMCL clusters large-scale networks several orders of magnitude faster than MCL and enables clustering of even bigger networks. Finally, HipMCL is based on MPI and OpenMP and is freely available under a modified BSD license.

Authors:

Azad, Ariful ^[1] ; ; Ouzounis, Christos A. ^[3] ; Kyrpides, Nikos C. ^[2] ; Buluc, Aydin ^[4]

Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
USDOE Joint Genome Institute (JGI), Walnut Creek, CA (United States)
Centre for Research & Technology Hellas, Thessalonica (Greece). Biological Computation & Process Lab. Chemical Process & Energy Resources Inst.
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division; Univ. of California, Berkeley, CA (United States). Dept. of Electrical Engineering and Computer Sciences

Publication Date:: 2018-01-05

Grant/Contract Number:: AC02-05CH11231

Type:: Accepted Manuscript

Journal Name:: Nucleic Acids Research

Additional Journal Information:: Journal Volume: 46; Journal Issue: 6; Journal ID: ISSN 0305-1048

Publisher:: Oxford University Press

Research Org:: Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); USDOE Joint Genome Institute (JGI), Walnut Creek, CA (United States)

Sponsoring Org:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); USDOE National Nuclear Security Administration (NNSA)

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; 59 BASIC BIOLOGICAL SCIENCES; computational methods; genomics

OSTI Identifier:: 1439241


                    Azad, Ariful, Pavlopoulos, Georgios A., Ouzounis, Christos A., Kyrpides, Nikos C., and Buluc, Aydin. HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks.  United States: N. p.,  
        Web.  doi:10.1093/nar/gkx1313.

Copy to clipboard


                    Azad, Ariful, Pavlopoulos, Georgios A., Ouzounis, Christos A., Kyrpides, Nikos C., & Buluc, Aydin. HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks.  United States.  doi:10.1093/nar/gkx1313.

Copy to clipboard


                    Azad, Ariful, Pavlopoulos, Georgios A., Ouzounis, Christos A., Kyrpides, Nikos C., and Buluc, Aydin. 2018.  
        "HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks".  United States.  
        doi:10.1093/nar/gkx1313.

Copy to clipboard


                    
@article{osti_1439241,

  title        = {HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks},

  author       = {Azad, Ariful and Pavlopoulos, Georgios A. and Ouzounis, Christos A. and Kyrpides, Nikos C. and Buluc, Aydin},

  abstractNote = {Biological networks capture structural or functional properties of relevant entities such as molecules, proteins or genes. Characteristic examples are gene expression networks or protein–protein interaction networks, which hold information about functional affinities or structural similarities. Such networks have been expanding in size due to increasing scale and abundance of biological data. While various clustering algorithms have been proposed to find highly connected regions, Markov Clustering (MCL) has been one of the most successful approaches to cluster sequence similarity or expression networks. Despite its popularity, MCL’s scalability to cluster large datasets still remains a bottleneck due to high running times and memory demands. In this paper, we present High-performance MCL (HipMCL), a parallel implementation of the original MCL algorithm that can run on distributed-memory computers. We show that HipMCL can efficiently utilize 2000 compute nodes and cluster a network of ~70 million nodes with ~68 billion edges in ~2.4 h. By exploiting distributed-memory environments, HipMCL clusters large-scale networks several orders of magnitude faster than MCL and enables clustering of even bigger networks. Finally, HipMCL is based on MPI and OpenMP and is freely available under a modified BSD license.},

  doi          = {10.1093/nar/gkx1313},

  journal      = {Nucleic Acids Research},

  number       = 6,

  volume       = 46,

  place        = {United States},

  year         = {2018},

  month        = {1}

}

Copy to clipboard

Free Publicly Accessible Full Text
This content will become publicly available on January 5, 2019
Publisher's Version of Record
10.1093/nar/gkx1313
https://doi.org/10.1093/nar/gkx1313

Works referenced in this record:

Basic local alignment search tool
journal, October 1990

Altschul, Stephen F.; Gish, Warren; Miller, Webb
Journal of Molecular Biology, Vol. 215, Issue 3, p. 403-410
DOI: 10.1016/S0022-2836(05)80360-2

Similar Records

Similar records in DOE PAGES and OSTI.GOV collections:

Large-scale optimization-based non-negative computational framework for diffusion equations: Parallel implementation and performance studies

Journal Article Chang, Justin ; Karra, Satish ; Nakshatrala, Kalyana B. July 2016 - Journal of Scientific Computing

It is well-known that the standard Galerkin formulation, which is often the formulation of choice under the finite element method for solving self-adjoint diffusion equations, does not meet maximum principles and the non-negative constraint for anisotropic diffusion equations. Recently, optimization-based methodologies that satisfy maximum principles and the non-negative constraint for steady-state and transient diffusion-type equations have been proposed. To date, these methodologies have been tested only on small-scale academic problems. The purpose of this paper is to systematically study the performance of the non-negative methodology in the context of high performance computing (HPC). PETSc and TAO libraries are, respectively, usedmore »« less
Cited by 1Full Text Available
Large-scale 3D geoelectromagnetic modeling using parallel adaptive high-order finite element method

Journal Article Grayver, Alexander V. ; Kolev, Tzanio V. November 2015 - Geophysics

Here, we have investigated the use of the adaptive high-order finite-element method (FEM) for geoelectromagnetic modeling. Because high-order FEM is challenging from the numerical and computational points of view, most published finite-element studies in geoelectromagnetics use the lowest order formulation. Solution of the resulting large system of linear equations poses the main practical challenge. We have developed a fully parallel and distributed robust and scalable linear solver based on the optimal block-diagonal and auxiliary space preconditioners. The solver was found to be efficient for high finite element orders, unstructured and nonconforming locally refined meshes, a wide range of frequencies, largemore »« less
Cited by 14Full Text Available
SQDFT: Spectral Quadrature method for large-scale parallel $O (N)$ Kohn–Sham calculations at high temperature

Journal Article Suryanarayana, Phanish ; Pratapa, Phanisri P. ; Sharma, Abhiraj ; ... December 2017 - Computer Physics Communications

We present SQDFT: a large-scale parallel implementation of the Spectral Quadrature (SQ) method formore »« less
Cited by 6Full Text Available
A novel approach to estimate emissions from large transportation networks: Hierarchical clustering-based link-driving-schedules for EPA-MOVES using dynamic time warping measures

Journal Article Aziz, H. M. Abdul ; Ukkusuri, Satish V. June 2017 - International Journal of Sustainable Transportation

We present that EPA-MOVES (Motor Vehicle Emission Simulator) is often integrated with traffic simulators to assess emission levels of large-scale urban networks with signalized intersections. High variations in speed profiles exist in the context of congested urban networks with signalized intersections. The traditional average-speed-based emission estimation technique with EPA-MOVES provides faster execution while underestimates the emissions in most cases because of ignoring the speed variation at congested networks with signalized intersections. In contrast, the atomic second-by-second speed profile (i.e., the trajectory of each vehicle)-based technique provides accurate emissions at the cost of excessive computational power and time. We addressed thismore »« less
Full Text Available
A High-Performance Sintered Iron Electrode for Rechargeable Alkaline Batteries to Enable Large-Scale Energy Storage

Journal Article Yang, Chenguang ; Manohar, Aswin K. ; Narayanan, S. R. January 2017 - Journal of the Electrochemical Society

Iron-based alkaline rechargeable batteries such as iron-air and nickel-iron batteries are particularly attractive for large-scale energy storage because these batteries can be relatively inexpensive, environment- friendly, and also safe. Therefore, our study has focused on achieving the essential electrical performance and cycling properties needed for the widespread use of iron-based alkaline batteries in stationary and distributed energy storage applications.We have demonstrated for the first time, an advanced sintered iron electrode capable of 3500 cycles of repeated charge and discharge at the 1-hour rate and 100% depth of discharge in each cycle, and an average Coulombic efficiency of over 97%. Suchmore »« less
Cited by 5Full Text Available

Access journal articles and accepted manuscripts resulting from DOE research funding

Title: HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks

Access journal articles and accepted manuscripts
resulting from DOE research funding