Overlapping clusters for distributed computation.

Conference · Mon Nov 01 04:00:00 EDT 2010

OSTI ID:1030279

Mirrokni, Vahab ^[1]; Andersen, Reid ^[2]; Gleich, David F

---

1. Google Research, New York, NY
2. Microsoft Corporation, Redmond, WA

Scalable, distributed algorithms must address communication problems. We investigate overlapping clusters, or vertex partitions that intersect, for graph computations. This setup stores more of the graph than required but then affords the ease of implementation of vertex partitioned algorithms. Our hope is that this technique allows us to reduce communication in a computation on a distributed graph. The motivation above draws on recent work in communication avoiding algorithms. Mohiyuddin et al. (SC09) design a matrix-powers kernel that gives rise to an overlapping partition. Fritzsche et al. (CSC2009) develop an overlapping clustering for a Schwarz method. Both techniques extend an initial partitioning with overlap. Our procedure generates overlap directly. Indeed, Schwarz methods are commonly used to capitalize on overlap. Elsewhere, overlapping communities (Ahn et al, Nature 2009; Mishra et al. WAW2007) are now a popular model of structure in social networks. These have long been studied in statistics (Cole and Wishart, CompJ 1970). We present two types of results: (i) an estimated swapping probability {rho}{infinity}; and (ii) the communication volume of a parallel PageRank solution (link-following {alpha} = 0.85) using an additive Schwarz method. The volume ratio is the amount of extra storage for the overlap (2 means we store the graph twice). Below, as the ratio increases, the swapping probability and PageRank communication volume decreases.

• Research Organization: Sandia National Laboratories
• Sponsoring Organization: USDOE
• DOE Contract Number: AC04-94AL85000
• OSTI ID: 1030279
• Report Number(s): SAND2010-7728C
• Country of Publication: United States
• Language: English

Related Subjects

99 GENERAL AND MISCELLANEOUS
ADDITIVES
ALGORITHMS
COMMUNICATIONS
COMMUNITIES
DESIGN
IMPLEMENTATION
KERNELS
PROBABILITY
STATISTICS
STORAGE