MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs

Conference · Thu Nov 07 04:00:00 EST 2024

DOI:https://doi.org/10.1109/CLUSTER59578.2024.00013· OSTI ID:2479155

Sarkar, Aishwarya ^[1]; Ghosh, Sayan ^[1]; Tallent, Nathan R. ^[1]; Jannesari, Ali ^[2]

---

1. BATTELLE (PACIFIC NW LAB)
2. Iowa State University

Graph Neural Networks (GNN) are indispensable in learning from graph-structured data, yet their rising computational costs, especially on massively connected graphs, pose significant challenges in terms of execution performance. To tackle this, distributed-memory solutions such as partitioning the graph to concurrently train multiple replicas of GNNs are in practice. However, approaches requiring a partitioned graph usually suffer from communication overhead and load imbalance, even under optimal partitioning and communication strategies due to irregularities in the neighborhood minibatch sampling. This paper proposes practical trade-offs for improving the sampling and communication overheads for representation learn- ing on distributed graphs (using popular GraphSAGE architecture) by developing a parameterized prefetch and eviction scheme on top of the state-of-the-art Amazon DistDGL distributed GNN framework, demonstrating about 15–40% improvement in end-to-end training performance on the NERSC Perlmutter supercomputer for various OGB datasets.

• Research Organization: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-76RL01830
• OSTI ID: 2479155
• Report Number(s): PNNL-SA-200893
• Country of Publication: United States
• Language: English

Related Subjects

Machine Leanring
high performance comptuing
graph neural networks