Streaming data analytics via message passing with application to graph algorithms

Plimpton, Steven J.; Shead, Tim

doi:10.1016/j.jpdc.2014.04.001

Streaming data analytics via message passing with application to graph algorithms

Journal Article · 06 May 2014 · Journal of Parallel and Distributed Computing

DOI:https://doi.org/10.1016/j.jpdc.2014.04.001· OSTI ID:1062811

Plimpton, Steven J. ^[1]; Shead, Tim ^[1]

Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

The need to process streaming data, which arrives continuously at high-volume in real-time, arises in a variety of contexts including data produced by experiments, collections of environmental or network sensors, and running simulations. Streaming data can also be formulated as queries or transactions which operate on a large dynamic data store, e.g. a distributed database. We describe a lightweight, portable framework named PHISH which enables a set of independent processes to compute on a stream of data in a distributed-memory parallel manner. Datums are routed between processes in patterns defined by the application. PHISH can run on top of either message-passing via MPI or sockets via ZMQ. The former means streaming computations can be run on any parallel machine which supports MPI; the latter allows them to run on a heterogeneous, geographically dispersed network of machines. We illustrate how PHISH can support streaming MapReduce operations, and describe streaming versions of three algorithms for large, sparse graph analytics: triangle enumeration, subgraph isomorphism matching, and connected component finding. Lastly, we also provide benchmark timings for MPI versus socket performance of several kernel operations useful in streaming algorithms.

View Accepted Manuscript (DOE)

Research Organization:: Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: AC04-94AL85000

OSTI ID:: 1062811

Report Number(s):: SAND2012--9495J; PII: S0743731514000884

Journal Information:: Journal of Parallel and Distributed Computing, Journal Name: Journal of Parallel and Distributed Computing Journal Issue: 8 Vol. 74; ISSN 0743-7315

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (14)

Inexact subgraph isomorphism in MapReduce Plantenga, Todd Journal of Parallel and Distributed Computing, Vol. 73, Issue 2 https://doi.org/10.1016/j.jpdc.2012.10.005	journal	February 2013
MapReduce in MPI for Large-scale graph algorithms Plimpton, Steven J.; Devine, Karen D. Parallel Computing, Vol. 37, Issue 9 https://doi.org/10.1016/j.parco.2011.02.004	journal	September 2011
S4: Distributed Stream Computing Platform Neumeyer, Leonardo; Robbins, Bruce; Nair, Anish 2010 IEEE International Conference on Data Mining Workshops (ICDMW) https://doi.org/10.1109/icdmw.2010.172	conference	December 2010
Graph Twiddling in a MapReduce World Cohen, J. Computing in Science & Engineering, Vol. 11, Issue 4 https://doi.org/10.1109/mcse.2009.120	journal	July 2009
A unified toolkit for information and scientific visualization Wylie, Brian; Baumes, Jeffrey SPIE Proceedings https://doi.org/10.1117/12.805589	conference	January 2009
Advances in dataflow programming languages Johnston, Wesley M.; Hanna, J. R. Paul; Millar, Richard J. ACM Computing Surveys, Vol. 36, Issue 1 https://doi.org/10.1145/1013208.1013209	journal	March 2004
A platform for scalable one-pass analytics using MapReduce Li, Boduo; Mazur, Edward; Diao, Yanlei Proceedings of the 2011 ACM SIGMOD International Conference on Management of data https://doi.org/10.1145/1989323.1989426	conference	June 2011
Maintaining connected components for infinite graph streams Berry, Jonathan; Oster, Matthew; Phillips, Cynthia A. Proceedings of the 2nd International Workshop on Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications https://doi.org/10.1145/2501221.2501234	conference	August 2013
R-MAT: A Recursive Model for Graph Mining Chakrabarti, Deepayan; Zhan, Yiping; Faloutsos, Christos Figshare https://doi.org/10.1184/r1/6609113	text	January 2018
MapReduce: Simplified Data Processing on Large Cluster No authors listed International Journal of Research and Engineering, Vol. 5, Issue 5 https://doi.org/10.21276/ijre.2018.5.5.4	journal	April 2018
S4: Distributed Stream Computing Platform Neumeyer, Leonardo; Robbins, Bruce; Nair, Anish 2010 IEEE International Conference on Data Mining Workshops (ICDMW) https://doi.org/10.1109/ICDMW.2010.172	conference	December 2010
Graph Twiddling in a MapReduce World Cohen, J. Computing in Science & Engineering, Vol. 11, Issue 4 https://doi.org/10.1109/MCSE.2009.120	journal	July 2009
R-MAT: A Recursive Model for Graph Mining Chakrabarti, Deepayan; Zhan, Yiping; Faloutsos, Christos Proceedings of the 2004 SIAM International Conference on Data Mining https://doi.org/10.1137/1.9781611972740.43	conference	December 2013
MapReduce: simplified data processing on large clusters Dean, Jeffrey; Ghemawat, Sanjay; Mehta, Brijesh Communications of the ACM, Vol. 51, Issue 1 https://doi.org/10.1145/1327452.1327492	journal	January 2008

Similar Records

Parallel Harness for Informatic Stream Hashing

Software · 2012 · OSTI ID:1231551

Parallel Harness for Informatic Stream Hashing

Software · 2012 · OSTI ID:code-6196

MapReduce MPI library (MR-MPI) v. 1.0

Software · 2009 · OSTI ID:1253283

Related Subjects

97 MATHEMATICS AND COMPUTING
Graph algorithms
MPI
MapReduce
Message passing
Sockets
Streaming data

Streaming data analytics via message passing with application to graph algorithms

Citation Formats

References (14)

Similar Records

Related Subjects