SHMEMGraph: Efficient and Balanced Graph Processing Using One-Sided Communication

doi:https://doi.org/10.1109/CCGRID.2018.00078

Title: SHMEMGraph: Efficient and Balanced Graph Processing Using One-Sided Communication

Full Record
Other Related Research

Abstract

State-of-the-art synchronous graph processing frameworks face both inefficiency and imbalance issues that cause their performance to be suboptimal. These issues include the inefficiency of communication and the imbalanced graph computation/communication costs in an iteration. We propose to replace their conventional two-sided communication model with the one-sided counterpart. Accordingly, we design SHMEMGraph, an efficient and balanced graph processing framework that is formulated across a global memory space and takes advantage of the flexibility and efficiency of one-sided communication for graph processing. Through an efficient one-sided communication channel, SHMEMGraph utilizes the high-performance operations with RDMA while minimizing the resource contention within a computer node. In addition, SHMEMGraph synthesizes a number of optimizations to address both computation imbalance and communication imbalance. By using a graph of 1 billion edges, our evaluation shows that compared to the state-of-the-art Gemini framework, SHMEMGraph achieves an average improvement of 35.5% in terms of job completion time for five representative graph algorithms.

Authors:

Fu, Huansong ^[1];

^[2]; Salman, Shaeke ^[1];

^[2]; Yu, Weikuan ^[1]

Florida State University, Tallahassee
ORNL

Publication Date:: 2018-05-01

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1468157

DOE Contract Number:: AC05-00OR22725

Resource Type:: Conference

Resource Relation:: Conference: IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing - Washington D.C, District of Columbia, United States of America - 5/1/2018 12:00:00 PM-5/4/2018 12:00:00 PM

Country of Publication:: United States

Language:: English

Citation Formats


                    Fu, Huansong, Gorentla Venkata, Manjunath, Salman, Shaeke, Imam, Neena, and Yu, Weikuan. SHMEMGraph: Efficient and Balanced Graph Processing Using One-Sided Communication.  United States: N. p., 2018. 
        Web.  doi:10.1109/CCGRID.2018.00078.

Copy to clipboard


                    Fu, Huansong, Gorentla Venkata, Manjunath, Salman, Shaeke, Imam, Neena, & Yu, Weikuan. SHMEMGraph: Efficient and Balanced Graph Processing Using One-Sided Communication.  United States.  https://doi.org/10.1109/CCGRID.2018.00078

Copy to clipboard


                    Fu, Huansong, Gorentla Venkata, Manjunath, Salman, Shaeke, Imam, Neena, and Yu, Weikuan. Tue .  
        "SHMEMGraph: Efficient and Balanced Graph Processing Using One-Sided Communication".  United States.  https://doi.org/10.1109/CCGRID.2018.00078.  https://www.osti.gov/servlets/purl/1468157.

Copy to clipboard


                    
@article{osti_1468157,

  title        = {SHMEMGraph: Efficient and Balanced Graph Processing Using One-Sided Communication},

  author       = {Fu, Huansong and Gorentla Venkata, Manjunath and Salman, Shaeke and Imam, Neena and Yu, Weikuan},

  abstractNote = {State-of-the-art synchronous graph processing frameworks face both inefficiency and imbalance issues that cause their performance to be suboptimal. These issues include the inefficiency of communication and the imbalanced graph computation/communication costs in an iteration. We propose to replace their conventional two-sided communication model with the one-sided counterpart. Accordingly, we design SHMEMGraph, an efficient and balanced graph processing framework that is formulated across a global memory space and takes advantage of the flexibility and efficiency of one-sided communication for graph processing. Through an efficient one-sided communication channel, SHMEMGraph utilizes the high-performance operations with RDMA while minimizing the resource contention within a computer node. In addition, SHMEMGraph synthesizes a number of optimizations to address both computation imbalance and communication imbalance. By using a graph of 1 billion edges, our evaluation shows that compared to the state-of-the-art Gemini framework, SHMEMGraph achieves an average improvement of 35.5% in terms of job completion time for five representative graph algorithms.},

  doi          = {10.1109/CCGRID.2018.00078},

  url          = {https://www.osti.gov/biblio/1468157},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2018},

  month        = {5}

}

Copy to clipboard

Conference:

View Conference (3.13 MB)

https://doi.org/10.1109/CCGRID.2018.00078

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

An asynchronous traversal engine for graph-based rich metadata management

Journal Article Dai, Dong ; Carns, Philip ; Ross, Robert B. ; ... - Parallel Computing

Rich metadata in high-performance computing (HPC) systems contains extended information about users, jobs, data files, and their relationships. Property graphs are a promising data model to represent heterogeneous rich metadata flexibly. Specifically, a property graph can use vertices to represent different entities and edges to record the relationships between vertices with unique annotations. The high-volume HPC use case, with millions of entities and relationships, naturally requires an out-of-core distributed property graph database, which must support live updates (to ingest production information in real time), low-latency point queries (for frequent metadata operations such as permission checking), and large-scale traversals (for provenancemore »« less
https://doi.org/10.1016/j.parco.2016.06.002

Full Text Available
Final Report for Project DE-FC02-06ER25755 [Pmodels2]

Technical Report Panda, Dhabaleswar ; Sadayappan, P.

In this report, we describe the research accomplished by the OSU team under the Pmodels2 project. The team has worked on various angles: designing high performance MPI implementations on modern networking technologies (Mellanox InfiniBand (including the new ConnectX2 architecture and Quad Data Rate), QLogic InfiniPath, the emerging 10GigE/iWARP and RDMA over Converged Enhanced Ethernet (RoCE) and Obsidian IB-WAN), studying MPI scalability issues for multi-thousand node clusters using XRC transport, scalable job start-up, dynamic process management support, efficient one-sided communication, protocol offloading and designing scalable collective communication libraries for emerging multi-core architectures. New designs conforming to the Argonne’s Nemesis interface havemore »« less
https://doi.org/10.2172/1122948

Full Text Available
Applying graph partitioning methods in measurement-based dynamic load balancing

Technical Report Bhatele, Abhinav ; Fourestier, Sebastien ; Menon, Harshitha ; ...

Load imbalance leads to an increasing waste of resources as an application is scaled to more and more processors. Achieving the best parallel efficiency for a program requires optimal load balancing which is a NP-hard problem. However, finding near-optimal solutions to this problem for complex computational science and engineering applications is becoming increasingly important. Charm++, a migratable objects based programming model, provides a measurement-based dynamic load balancing framework. This framework instruments and then migrates over-decomposed objects to balance computational load and communication at runtime. This paper explores the use of graph partitioning algorithms, traditionally used for partitioning physical domains/meshes, formore »« less
https://doi.org/10.2172/1114706

Full Text Available
Decomposition of Large Scale Semantic Graphsvia an Efficient Communities Algorithm

Technical Report Yao, Y

Semantic graphs have become key components in analyzing complex systems such as the Internet, or biological and social networks. These types of graphs generally consist of sparsely connected clusters or 'communities' whose nodes are more densely connected to each other than to other nodes in the graph. The identification of these communities is invaluable in facilitating the visualization, understanding, and analysis of large graphs by producing subgraphs of related data whose interrelationships can be readily characterized. Unfortunately, the ability of LLNL to effectively analyze the terabytes of multisource data at its disposal has remained elusive, since existing decomposition algorithms becomemore »« less
https://doi.org/10.2172/926011

Full Text Available
Improved Accuracy for Automated Communication Pattern Characterization Using Communication Graphs and Aggressive Search Space Pruning

Conference Roth, Philip

An understanding of a parallel application’s communication behavior is useful for a range of activities including debugging and optimization, job scheduling, target system selection, and system design. Because it can be challenging to understand communication behavior, especially for those who lack expertise or who are not familiar with the application, I and two colleagues recently developed an automated, search-based approach for recognizing and parameterizing application communication behavior using a library of common communication patterns. This initial approach was effective for characterizing the behavior of many workloads, but I identified some combinations of communication patterns for which the method was inefficientmore »« less
https://doi.org/10.1007/978-3-030-17872-7_3

Full Text Available

Similar Records