In-depth analysis on parallel processing patterns for high-performance Dataframes

Journal Article · Thu Jul 13 00:00:00 EDT 2023 · Future Generations Computer Systems

DOI:https://doi.org/10.1016/j.future.2023.07.007· OSTI ID:3000839

Perera, Niranda ^[1]; Sarker, Arup Kumar ^[2]; Staylor, Mills ^[2]; von Laszewski, Gregor ^[2]; Shan, Kaiying ^[2]; Kamburugamuve, Supun ^[1]; Widanage, Chathura ^[1]; Abeykoon, Vibhatha ^[1]; Kanewela, Thejaka Amila ^[1]; Fox, Geoffrey ^[2]

---

1. Indiana Univ., Bloomington, IN (United States)
2. Univ. of Virginia, Charlottesville, VA (United States)

The Data Science domain has expanded monumentally in both research and industry communities during the past decade, predominantly owing to the Big Data revolution. Artificial Intelligence (AI) and Machine Learning (ML) are bringing more complexities to data engineering applications, which are now integrated into data processing pipelines to process terabytes of data. Typically, a significant amount of time is spent on data preprocessing in these pipelines, and hence improving its efficiency directly impacts the overall pipeline performance. The community has recently embraced the concept of Dataframes as the de-facto data structure for data representation and manipulation. However, the most widely used serial Dataframes today (R, pandas) experience performance limitations while working on even moderately large data sets. We believe that there is plenty of room for improvement by taking a look at this problem from a high-performance computing point of view. In a prior publication, we presented a set of parallel processing patterns for distributed dataframe operators and the reference runtime implementation, Cylon. In this paper, we are expanding on the initial concept by introducing a cost model for evaluating the said patterns. Furthermore, we evaluate the performance of Cylon on the ORNL Summit supercomputer.

• Research Organization: Univ. of Virginia, Charlottesville, VA (United States)
• Sponsoring Organization: USDOE Office of Science (SC)
• Grant/Contract Number: SC0023452
• OSTI ID: 3000839
• Report Number(s): 2307.01394
• Journal Information: Future Generations Computer Systems, Journal Name: Future Generations Computer Systems Vol. 149; ISSN 0167-739X
• Publisher: Elsevier
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Data engineering
Dataframes
Distributed Memory Parallel
High-performance computing
MPI
Relational algebra