A Parallel Graph Environment for Real-World Data Analytics Workflows

Conference · Mon Mar 25 04:00:00 EDT 2019

DOI:https://doi.org/10.23919/DATE.2019.8715196· OSTI ID:1591771

Castellana, Vito G. ^[1]; Drocco, Maurizio ^[1]; Feo, John T. ^[1]; Firoz, Jesun ^[2]; Kanewala, Thejaka A. ^[2]; Lumsdaine, Andrew ^[1]; Manzano Franco, Joseph B. ^[1]; Marquez, Andres ^[1]; Minutoli, Marco ^[1]; Suetterlein, Joshua D. ^[1]; Tumeo, Antonino ^[1]; Zalewski, Marcin J. ^[1]

---

1. BATTELLE (PACIFIC NW LAB)
2. Indiana University-Bloomington

Economic competitiveness and national security depend increasingly on the insightful analysis of large data sets. The diversity of real-world data sources and analytic workflows impose challenging hardware and software requirements for parallel graph platforms. The irregular nature of graph methods is not supported well by the deep memory hierarchies of conventional distributed systems, requiring new processor and runtime system designs to tolerate memory and synchronization latencies. Moreover, the efficiency of relational table operations and matrix computations are not attainable when data is stored in common graph data structures. In this paper, we present HAGGLE, a high-performance, scalable data analytics platform. The platform’s hybrid data model supports a variety of distributed, thread-safe data structures, parallel programming constructs, and persistent and streaming data. An abstract runtime layer enables us to map the stack to conventional, distributed computer systems with accelerators. The runtime uses multithreading, active messages, and data aggregation to hide memory and synchronization latencies on large-scale systems.

• Research Organization: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-76RL01830
• OSTI ID: 1591771
• Report Number(s): PNNL-SA-140268
• Country of Publication: United States
• Language: English

Related Subjects

Graph Analytics