skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Distributed Louvain Algorithm for Graph Community Detection

Abstract

In most real-world networks, the nodes/vertices tend to be organized into tightly-knit modules known as communities or clusters, such that nodes within a community are more likely to be “related” to one another than they are to the rest of the network. The goodness of partitioning into communities is typically measured using a well known measure called modularity. However, modularity optimization is an NP-complete problem. In 2008, Blondel, et al. introduced a multi-phase, iterative heuristic for modularity optimization, called the Louvain method. Owing to its speed and ability to yield high quality communities, the Louvain method continues to be one of the most widely used tools for serial community detection. In this paper, we present the design of a distributed memory implementation of the Louvain algorithm for parallel community detection. Our approach begins with an arbitrarily partitioned distributed graph input, and employs several heuristics to speedup the computation of the different steps of the Louvain algorithm. We evaluate our implementation and its different variants using real-world networks from various application domains (including internet, biology, social networks). Our MPI+OpenMP implementation yields about 7x speedup (on 4K processes) for soc-friendster network (1.8B edges) over a state-of-the-art shared-memory multicore implementation (on 64 threads),more » without compromising output quality. Furthermore, our distributed implementation was able to process a larger graph (uk-2007; 3.3B edges) in 32 seconds on 1K cores (64 nodes) of NERSC Cori, when the state-of-the-art shared-memory implementation failed to run due to insufficient memory on a single Cori node containing 128 GB.« less

Authors:
 [1]; ORCiD logo [2];  [2];  [3];  [3];  [2];  [2];  [4]
  1. UNIVERSITY PROGRAMS
  2. BATTELLE (PACIFIC NW LAB)
  3. WASHINGTON STATE UNIV
  4. Washington State University
Publication Date:
Research Org.:
Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1526306
Report Number(s):
PNNL-SA-130211
DOE Contract Number:  
AC05-76RL01830
Resource Type:
Conference
Resource Relation:
Conference: IEEE International Parallel & Distributed Processing Symposium (IPDPS 2018), May 21-25, 2018, Vancouver, BC
Country of Publication:
United States
Language:
English
Subject:
High performance computing, graph algorithms, Distributed computing, MPI

Citation Formats

Ghosh, Sayan, Halappanavar, Mahantesh, Tumeo, Antonino, Kalyanaraman, Anantharaman, Lu, Hao, Chavarría-Miranda, Daniel G., Khan, Md Ariful H., and Gebremedhin, Assefaw. Distributed Louvain Algorithm for Graph Community Detection. United States: N. p., 2018. Web. doi:10.1109/IPDPS.2018.00098.
Ghosh, Sayan, Halappanavar, Mahantesh, Tumeo, Antonino, Kalyanaraman, Anantharaman, Lu, Hao, Chavarría-Miranda, Daniel G., Khan, Md Ariful H., & Gebremedhin, Assefaw. Distributed Louvain Algorithm for Graph Community Detection. United States. doi:10.1109/IPDPS.2018.00098.
Ghosh, Sayan, Halappanavar, Mahantesh, Tumeo, Antonino, Kalyanaraman, Anantharaman, Lu, Hao, Chavarría-Miranda, Daniel G., Khan, Md Ariful H., and Gebremedhin, Assefaw. Tue . "Distributed Louvain Algorithm for Graph Community Detection". United States. doi:10.1109/IPDPS.2018.00098.
@article{osti_1526306,
title = {Distributed Louvain Algorithm for Graph Community Detection},
author = {Ghosh, Sayan and Halappanavar, Mahantesh and Tumeo, Antonino and Kalyanaraman, Anantharaman and Lu, Hao and Chavarría-Miranda, Daniel G. and Khan, Md Ariful H. and Gebremedhin, Assefaw},
abstractNote = {In most real-world networks, the nodes/vertices tend to be organized into tightly-knit modules known as communities or clusters, such that nodes within a community are more likely to be “related” to one another than they are to the rest of the network. The goodness of partitioning into communities is typically measured using a well known measure called modularity. However, modularity optimization is an NP-complete problem. In 2008, Blondel, et al. introduced a multi-phase, iterative heuristic for modularity optimization, called the Louvain method. Owing to its speed and ability to yield high quality communities, the Louvain method continues to be one of the most widely used tools for serial community detection. In this paper, we present the design of a distributed memory implementation of the Louvain algorithm for parallel community detection. Our approach begins with an arbitrarily partitioned distributed graph input, and employs several heuristics to speedup the computation of the different steps of the Louvain algorithm. We evaluate our implementation and its different variants using real-world networks from various application domains (including internet, biology, social networks). Our MPI+OpenMP implementation yields about 7x speedup (on 4K processes) for soc-friendster network (1.8B edges) over a state-of-the-art shared-memory multicore implementation (on 64 threads), without compromising output quality. Furthermore, our distributed implementation was able to process a larger graph (uk-2007; 3.3B edges) in 32 seconds on 1K cores (64 nodes) of NERSC Cori, when the state-of-the-art shared-memory implementation failed to run due to insufficient memory on a single Cori node containing 128 GB.},
doi = {10.1109/IPDPS.2018.00098},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {5}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: