HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks

doi:10.1093/nar/gkx1313

Title: HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks

Abstract

Biological networks capture structural or functional properties of relevant entities such as molecules, proteins or genes. Characteristic examples are gene expression networks or protein–protein interaction networks, which hold information about functional affinities or structural similarities. Such networks have been expanding in size due to increasing scale and abundance of biological data. While various clustering algorithms have been proposed to find highly connected regions, Markov Clustering (MCL) has been one of the most successful approaches to cluster sequence similarity or expression networks. Despite its popularity, MCL’s scalability to cluster large datasets still remains a bottleneck due to high running times and memory demands. In this paper, we present High-performance MCL (HipMCL), a parallel implementation of the original MCL algorithm that can run on distributed-memory computers. We show that HipMCL can efficiently utilize 2000 compute nodes and cluster a network of ~70 million nodes with ~68 billion edges in ~2.4 h. By exploiting distributed-memory environments, HipMCL clusters large-scale networks several orders of magnitude faster than MCL and enables clustering of even bigger networks. Finally, HipMCL is based on MPI and OpenMP and is freely available under a modified BSD license.

Authors:

Azad, Ariful ^[1];

^[2]; Ouzounis, Christos A. ^[3]; Kyrpides, Nikos C. ^[2]; Buluc, Aydin ^[4]

Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
USDOE Joint Genome Institute (JGI), Walnut Creek, CA (United States)
Centre for Research & Technology Hellas, Thessalonica (Greece). Biological Computation & Process Lab. Chemical Process & Energy Resources Inst.
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division; Univ. of California, Berkeley, CA (United States). Dept. of Electrical Engineering and Computer Sciences

Publication Date:: 2018-01-05

Research Org.:: Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); USDOE Joint Genome Institute (JGI), Walnut Creek, CA (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); USDOE National Nuclear Security Administration (NNSA)

OSTI Identifier:: 1439241

Grant/Contract Number:: AC02-05CH11231

Resource Type:: Accepted Manuscript

Journal Name:: Nucleic Acids Research

Additional Journal Information:: Journal Volume: 46; Journal Issue: 6; Journal ID: ISSN 0305-1048

Publisher:: Oxford University Press

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; 59 BASIC BIOLOGICAL SCIENCES; computational methods; genomics

Citation Formats


                    Azad, Ariful, Pavlopoulos, Georgios A., Ouzounis, Christos A., Kyrpides, Nikos C., and Buluc, Aydin. HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks.  United States: N. p., 2018. 
        Web.  doi:10.1093/nar/gkx1313.

Copy to clipboard


                    Azad, Ariful, Pavlopoulos, Georgios A., Ouzounis, Christos A., Kyrpides, Nikos C., & Buluc, Aydin. HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks.  United States.  doi:10.1093/nar/gkx1313.

Copy to clipboard


                    Azad, Ariful, Pavlopoulos, Georgios A., Ouzounis, Christos A., Kyrpides, Nikos C., and Buluc, Aydin. Fri .  
        "HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks".  United States.  doi:10.1093/nar/gkx1313.  https://www.osti.gov/servlets/purl/1439241.

Copy to clipboard


                    
@article{osti_1439241,

  title        = {HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks},

  author       = {Azad, Ariful and Pavlopoulos, Georgios A. and Ouzounis, Christos A. and Kyrpides, Nikos C. and Buluc, Aydin},

  abstractNote = {Biological networks capture structural or functional properties of relevant entities such as molecules, proteins or genes. Characteristic examples are gene expression networks or protein–protein interaction networks, which hold information about functional affinities or structural similarities. Such networks have been expanding in size due to increasing scale and abundance of biological data. While various clustering algorithms have been proposed to find highly connected regions, Markov Clustering (MCL) has been one of the most successful approaches to cluster sequence similarity or expression networks. Despite its popularity, MCL’s scalability to cluster large datasets still remains a bottleneck due to high running times and memory demands. In this paper, we present High-performance MCL (HipMCL), a parallel implementation of the original MCL algorithm that can run on distributed-memory computers. We show that HipMCL can efficiently utilize 2000 compute nodes and cluster a network of ~70 million nodes with ~68 billion edges in ~2.4 h. By exploiting distributed-memory environments, HipMCL clusters large-scale networks several orders of magnitude faster than MCL and enables clustering of even bigger networks. Finally, HipMCL is based on MPI and OpenMP and is freely available under a modified BSD license.},

  doi          = {10.1093/nar/gkx1313},

  journal      = {Nucleic Acids Research},

  number       = 6,

  volume       = 46,

  place        = {United States},

  year         = {2018},

  month        = {1}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

DOI: 10.1093/nar/gkx1313

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 5 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Basic local alignment search tool
journal, October 1990

Altschul, Stephen F.; Gish, Warren; Miller, Webb
Journal of Molecular Biology, Vol. 215, Issue 3, p. 403-410
DOI: 10.1016/S0022-2836(05)80360-2

Similar Records in DOE PAGES and OSTI.GOV collections:

Large-scale optimization-based non-negative computational framework for diffusion equations: Parallel implementation and performance studies

Journal Article Chang, Justin ; Karra, Satish ; Nakshatrala, Kalyana B. - Journal of Scientific Computing

It is well-known that the standard Galerkin formulation, which is often the formulation of choice under the finite element method for solving self-adjoint diffusion equations, does not meet maximum principles and the non-negative constraint for anisotropic diffusion equations. Recently, optimization-based methodologies that satisfy maximum principles and the non-negative constraint for steady-state and transient diffusion-type equations have been proposed. To date, these methodologies have been tested only on small-scale academic problems. The purpose of this paper is to systematically study the performance of the non-negative methodology in the context of high performance computing (HPC). PETSc and TAO libraries are, respectively, usedmore »« less
Cited by 1
DOI: 10.1007/s10915-016-0250-5

Full Text Available
Modeling Large-Scale Slim Fly Networks Using Parallel Discrete-Event Simulation

Journal Article Wolfe, Noah ; Mubarak, Misbah ; Carothers, Christopher D. ; ... - ACM Transactions on Modeling and Computer Simulation

As supercomputers approach exascale performance, the increased number of processors translates to an increased demand on the underlying network interconnect. We present that the slim fly network topology, a new low-diameter, low-latency, and low-cost interconnection network, is gaining interest as one possible solution for next-generation supercomputing interconnect systems. In this article, we present a high-fidelity slim fly packet-level model leveraging the Rensselaer Optimistic Simulation System (ROSS) and Co-Design of Exascale Storage (CODES) frameworks. We validate the model with published work before scaling the network size up to an unprecedented 1 million compute nodes and confirming that the slim fly observesmore »« less
Cited by 1
DOI: 10.1145/3203406

Full Text Available
A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States

Journal Article Wang, Jiali ; Wang, Cheng ; Rao, Vishwas ; ... - Geoscientific Model Development (Online)

The Weather Research and Forecasting Hydrological (WRF-Hydro) system is a state-of-the-art numerical model that models the entire hydrological cycle based on physical principles. As with other hydrological models, WRF-Hydro parameterizes many physical processes. Hence, WRF-Hydro needs to be calibrated to optimize its output with respect to observations for the application region. When applied to a relatively large domain, both WRF-Hydro simulations and calibrations require intensive computing resources and are best performed on multimode, multicore high-performance computing (HPC) systems. Typically, each physics-based model requires a calibration process that works specifically with that model and is not transferrable to a different processmore »« less
DOI: 10.5194/gmd-12-3523-2019

Full Text Available
Large-scale 3D geoelectromagnetic modeling using parallel adaptive high-order finite element method

Journal Article Grayver, Alexander V. ; Kolev, Tzanio V. - Geophysics

Here, we have investigated the use of the adaptive high-order finite-element method (FEM) for geoelectromagnetic modeling. Because high-order FEM is challenging from the numerical and computational points of view, most published finite-element studies in geoelectromagnetics use the lowest order formulation. Solution of the resulting large system of linear equations poses the main practical challenge. We have developed a fully parallel and distributed robust and scalable linear solver based on the optimal block-diagonal and auxiliary space preconditioners. The solver was found to be efficient for high finite element orders, unstructured and nonconforming locally refined meshes, a wide range of frequencies, largemore »« less
Cited by 14
DOI: 10.1190/geo2015-0013.1

Full Text Available
SQDFT: Spectral Quadrature method for large-scale parallel $O (N)$ Kohn–Sham calculations at high temperature

Journal Article Suryanarayana, Phanish ; Pratapa, Phanisri P. ; Sharma, Abhiraj ; ... - Computer Physics Communications

We present SQDFT: a large-scale parallel implementation of the Spectral Quadrature (SQ) method formore »« less
Cited by 7
DOI: 10.1016/j.cpc.2017.12.003

Full Text Available

Similar Records

Title: HipMCL: a high-performance parallel implementation of the Markov clustering algorithm for large-scale networks

Abstract

Citation Formats

Basic local alignment search tool journal, October 1990

Basic local alignment search tool
journal, October 1990