Towards scalable dataframe systems

Petersohn, Devin; Macke, Stephen; Xin, Doris; Ma, William; Lee, Doris; Mo, Xiangxi; Gonzalez, Joseph E.; Hellerstein, Joseph M.; Joseph, Anthony D.; Parameswaran, Aditya

doi:10.14778/3407790.3407807

Towards scalable dataframe systems

Journal Article · Mon Sep 14 00:00:00 EDT 2020 · Proceedings of the VLDB Endowment

DOI:https://doi.org/10.14778/3407790.3407807· OSTI ID:1803099

Petersohn, Devin ^[1]; Macke, Stephen ^[2]; Xin, Doris ^[2]; Ma, William ^[2]; Lee, Doris ^[2]; Mo, Xiangxi ^[2]; Gonzalez, Joseph E. ^[2]; Hellerstein, Joseph M. ^[2]; Joseph, Anthony D. ^[2]; Parameswaran, Aditya ^[2]

Univ. of California, Berkeley, CA (United States); OSTI
Univ. of California, Berkeley, CA (United States)

Dataframes are a popular abstraction to represent, prepare, and analyze data. Despite the remarkable success of dataframe libraries in R and Python, dataframes face performance issues even on moderately large datasets. Moreover, there is significant ambiguity regarding dataframe semantics. In this paper we lay out a vision and roadmap for scalable dataframe systems. To demonstrate the potential in this area, we report on our experience building Modin, a scaled-up implementation of the most widely-used and complex dataframe API today, Python's pandas. With pandas as a reference, we propose a simple data model and algebra for dataframes to ground discussion in the field. Given this foundation, we lay out an agenda of open research opportunities where the distinct features of dataframes will require extending the state of the art in many dimensions of data management. We discuss the implications of signature dataframe features including flexible schemas, ordering, row/column equivalence, and data/metadata fluidity, as well as the piecemeal, trial-and-error-based approach to interacting with dataframes.

View Accepted Manuscript (DOE)

Research Organization:: Univ. of California, Oakland, CA (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE Office of Science (SC)

Grant/Contract Number:: SC0016934

OSTI ID:: 1803099

Journal Information:: Proceedings of the VLDB Endowment, Journal Name: Proceedings of the VLDB Endowment Journal Issue: 12 Vol. 13; ISSN 2150-8097

Publisher:: Association for Computing Machinery (ACM)Copyright Statement

Country of Publication:: United States

Language:: English

References (31)

R: A Language for Data Analysis and Graphics Ihaka, Ross; Gentleman, Robert Journal of Computational and Graphical Statistics, Vol. 5, Issue 3 https://doi.org/10.1080/10618600.1996.10474713	journal	September 1996
Partial results in database systems Lang, Willis; Nehme, Rimma V.; Robinson, Eric Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data https://doi.org/10.1145/2588555.2612176	conference	June 2014
Efficient and extensible algorithms for multi query optimization Roy, Prasan; Seshadri, S.; Sudarshan, S. ACM SIGMOD Record, Vol. 29, Issue 2 https://doi.org/10.1145/335191.335419	journal	May 2000
Multiple-query optimization Sellis, Timos K. ACM Transactions on Database Systems, Vol. 13, Issue 1 https://doi.org/10.1145/42201.42203	journal	March 1988
DataSpread Bendre, Mangesh; Sun, Bofan; Zhang, Ding Proceedings of the VLDB Endowment, Vol. 8, Issue 12 https://doi.org/10.14778/2824032.2824121	journal	August 2015
Optimization of nested XQuery expressions with orderby clauses Wang, Song; Rundensteiner, Elke A.; Mani, Murali Data & Knowledge Engineering, Vol. 60, Issue 2 https://doi.org/10.1016/j.datak.2006.03.004	journal	February 2007
Visualization-aware sampling for very large databases Park, Yongjoo; Cafarella, Michael; Mozafari, Barzan 2016 IEEE 32nd International Conference on Data Engineering (ICDE) https://doi.org/10.1109/ICDE.2016.7498287	conference	May 2016
QPipe Harizopoulos, Stavros; Shkapenyuk, Vladislav; Ailamaki, Anastassia Proceedings of the 2005 ACM SIGMOD international conference on Management of data https://doi.org/10.1145/1066157.1066201	conference	June 2005
A formal characterization of PIVOT/UNPIVOT Wyss, Catharine M.; Robertson, Edward L. Proceedings of the 14th ACM international conference on Information and knowledge management https://doi.org/10.1145/1099554.1099709	conference	October 2005
Skimmer Singh, Manish; Nandi, Arnab; Jagadish, H. V. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data https://doi.org/10.1145/2213836.2213858	conference	May 2012
BlinkDB Agarwal, Sameer; Mozafari, Barzan; Panda, Aurojit Proceedings of the 8th ACM European Conference on Computer Systems https://doi.org/10.1145/2465351.2465355	conference	April 2013
Online aggregation Hellerstein, Joseph M.; Haas, Peter J.; Wang, Helen J. ACM SIGMOD Record, Vol. 26, Issue 2 https://doi.org/10.1145/253262.253291	journal	June 1997
The analytical bootstrap Zeng, Kai; Gao, Shi; Mozafari, Barzan Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data https://doi.org/10.1145/2588555.2588579	conference	June 2014
Fully Functional Static and Dynamic Succinct Trees Navarro, Gonzalo; Sadakane, Kunihiko ACM Transactions on Algorithms, Vol. 10, Issue 3 https://doi.org/10.1145/2601073	journal	May 2014
Spark SQL: Relational Data Processing in Spark Armbrust, Michael; Ghodsi, Ali; Zaharia, Matei Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data - SIGMOD '15 https://doi.org/10.1145/2723372.2742797	conference	January 2015
Query unnesting in object-oriented databases Fegaras, Leonidas ACM SIGMOD Record, Vol. 27, Issue 2 https://doi.org/10.1145/276305.276310	journal	June 1998
Sample + Seek Ding, Bolin; Huang, Silu; Chaudhuri, Surajit Proceedings of the 2016 International Conference on Management of Data https://doi.org/10.1145/2882903.2915249	conference	June 2016
PFunk-H Alabi, Daniel; Wu, Eugene Proceedings of the Workshop on Human-In-the-Loop Data Analytics https://doi.org/10.1145/2939502.2939512	conference	June 2016
LaraDB Hutchison, Dylan; Howe, Bill; Suciu, Dan Proceedings of the 4th ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond https://doi.org/10.1145/3070607.3070608	conference	May 2017
Exploration and Explanation in Computational Notebooks Rule, Adam; Tabard, Aurélien; Hollan, James D. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems https://doi.org/10.1145/3173574.3173606	conference	April 2018
VerdictDB Park, Yongjoo; Mozafari, Barzan; Sorenson, Joseph Proceedings of the 2018 International Conference on Management of Data https://doi.org/10.1145/3183713.3196905	conference	May 2018
Optimally Leveraging Density and Locality for Exploratory Browsing and Sampling Kim, Albert; Xu, Liqi; Siddiqui, Tarique Proceedings of the Workshop on Human-In-the-Loop Data Analytics https://doi.org/10.1145/3209900.3209903	conference	June 2018
DuckDB Raasveldt, Mark; Mühleisen, Hannes Proceedings of the 2019 International Conference on Management of Data https://doi.org/10.1145/3299869.3320212	conference	June 2019
From Ad-Hoc Data Analytics to DataOps Munappy, Aiswarya Raj; Mattos, David Issa; Bosch, Jan Proceedings of the International Conference on Software and System Processes https://doi.org/10.1145/3379177.3388909	conference	June 2020
Efficient and extensible algorithms for multi query optimization Roy, Prasan; Seshadri, S.; Sudarshan, S. Proceedings of the 2000 ACM SIGMOD international conference on Management of data https://doi.org/10.1145/342009.335419	conference	May 2000
Rate-based query optimization for streaming information sources Viglas, Stratis D.; Naughton, Jeffrey F. Proceedings of the 2002 ACM SIGMOD international conference on Management of data https://doi.org/10.1145/564691.564697	conference	June 2002
Access path selection in a relational database management system Selinger, P. Griffiths; Astrahan, M. M.; Chamberlin, D. D. Proceedings of the 1979 ACM SIGMOD international conference on Management of data - SIGMOD '79 https://doi.org/10.1145/582095.582099	conference	January 1979
SharedDB Giannikis, Georgios; Alonso, Gustavo; Kossmann, Donald Proceedings of the VLDB Endowment, Vol. 5, Issue 6 https://doi.org/10.14778/2168651.2168654	journal	February 2012
Rapid sampling for visualizations with ordering guarantees Kim, Albert; Blais, Eric; Parameswaran, Aditya Proceedings of the VLDB Endowment, Vol. 8, Issue 5 https://doi.org/10.14778/2735479.2735485	journal	January 2015
Adaptive sampling for rapidly matching histograms Macke, Stephen; Zhang, Yiming; Huang, Silu Proceedings of the VLDB Endowment, Vol. 11, Issue 10 https://doi.org/10.14778/3231751.3231753	journal	June 2018
Helix Xin, Doris; Macke, Stephen; Ma, Litian Proceedings of the VLDB Endowment, Vol. 12, Issue 4 https://doi.org/10.14778/3297753.3297763	journal	December 2018

Similar Records

Adapter Python IO (Adapter) v1.0

Software · Mon Apr 19 20:00:00 EDT 2021 · OSTI ID:code-56244

pandas-sacct v1.0.0

Software · Wed Sep 30 20:00:00 EDT 2020 · OSTI ID:code-51920

In-depth analysis on parallel processing patterns for high-performance Dataframes

Journal Article · Wed Jul 12 20:00:00 EDT 2023 · Future Generations Computer Systems · OSTI ID:3000839

Related Subjects

97 MATHEMATICS AND COMPUTING
Computer science

Towards scalable dataframe systems

Citation Formats

References (31)

Similar Records

Related Subjects