Novel functional and distributed approaches to data analysis available in ROOT

Amadio, G.; Blomer, J.; Canal, P.; Ganis, G.; Guiraud, E.; Mato Vila, P.; Moneta, L.; Piparo, D.; Tejedor, E.; Valls Pla, X.

doi:10.1088/1742-6596/1085/4/042008

Title: Novel functional and distributed approaches to data analysis available in ROOT

Abstract

The bright future of particle physics at the Energy and Intensity frontiers poses exciting challenges to the scientific software community. The traditional strategies for processing and analyzing data are evolving in order to (i) offer higher-level programming model approaches and (ii) exploit parallelism to cope with the ever increasing complexity and size of the datasets. This contribution describes how the ROOT framework, a cornerstone of software stacks dedicated to particle physics, is preparing to provide adequate solutions for the analysis of large amount of scientific data on parallel architectures. The functional approach to parallel data analysis provided with the ROOT TDataFrame interface is then characterized. The design choices behind this new interface are described also comparing with other widely adopted tools such as Pandas and Apache Spark. The programming model is illustrated highlighting the reduction of boilerplate code, composability of the actions and data transformations as well as the capabilities of dealing with different data sources such as ROOT, JSON, CSV or databases. Details are given about how the functional approach allows transparent implicit parallelization of the chain of operations specified by the user. The progress done in the field of distributed analysis is examined. In particular, the power ofmore »« less

Authors:

Amadio, G. ^[1]; Blomer, J. ^[1];

^[2]; Ganis, G. ^[1]; Guiraud, E. ^[3]; Mato Vila, P. ^[1]; Moneta, L. ^[1]; Piparo, D. ^[1]; Tejedor, E. ^[1]; Valls Pla, X. ^[4]

European Organization for Nuclear Research (CERN), Geneva (Switzerland)
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
European Organization for Nuclear Research (CERN), Geneva (Switzerland); Univ. of Oldenburg, Oldenburg (Germany)
European Organization for Nuclear Research (CERN), Geneva (Switzerland); Univ. Jaume I, Castellon (Spain)

Publication Date:: Thu Oct 18 00:00:00 EDT 2018

Research Org.:: Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), High Energy Physics (HEP)

OSTI Identifier:: 1523435

Report Number(s):: FERMILAB-CONF-18-750-CD
Journal ID: ISSN 1742-6588; 1699879

Grant/Contract Number:: AC02-07CH11359

Resource Type:: Accepted Manuscript

Journal Name:: Journal of Physics. Conference Series

Additional Journal Information:: Journal Volume: 1085; Journal Issue: 4; Conference: 18th International Workshop on Advanced Computing and Analysis Techniques in Physics Research, Seattle, WA (United States), 21-25 Aug 2017; Journal ID: ISSN 1742-6588

Publisher:: IOP Publishing

Country of Publication:: United States

Language:: English

Subject:: 72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS

Citation Formats


                    Amadio, G., Blomer, J., Canal, P., Ganis, G., Guiraud, E., Mato Vila, P., Moneta, L., Piparo, D., Tejedor, E., and Valls Pla, X. Novel functional and distributed approaches to data analysis available in ROOT.  United States: N. p., 2018. 
Web.  doi:10.1088/1742-6596/1085/4/042008.

Copy to clipboard


                    Amadio, G., Blomer, J., Canal, P., Ganis, G., Guiraud, E., Mato Vila, P., Moneta, L., Piparo, D., Tejedor, E., & Valls Pla, X. Novel functional and distributed approaches to data analysis available in ROOT.  United States.  https://doi.org/10.1088/1742-6596/1085/4/042008

Copy to clipboard


                    Amadio, G., Blomer, J., Canal, P., Ganis, G., Guiraud, E., Mato Vila, P., Moneta, L., Piparo, D., Tejedor, E., and Valls Pla, X. Thu .  
"Novel functional and distributed approaches to data analysis available in ROOT".  United States.  https://doi.org/10.1088/1742-6596/1085/4/042008.  https://www.osti.gov/servlets/purl/1523435.

Copy to clipboard


                    
@article{osti_1523435,

  title        = {Novel functional and distributed approaches to data analysis available in ROOT},

  author       = {Amadio, G. and Blomer, J. and Canal, P. and Ganis, G. and Guiraud, E. and Mato Vila, P. and Moneta, L. and Piparo, D. and Tejedor, E. and Valls Pla, X.},

  abstractNote = {The bright future of particle physics at the Energy and Intensity frontiers poses exciting challenges to the scientific software community. The traditional strategies for processing and analyzing data are evolving in order to (i) offer higher-level programming model approaches and (ii) exploit parallelism to cope with the ever increasing complexity and size of the datasets. This contribution describes how the ROOT framework, a cornerstone of software stacks dedicated to particle physics, is preparing to provide adequate solutions for the analysis of large amount of scientific data on parallel architectures. The functional approach to parallel data analysis provided with the ROOT TDataFrame interface is then characterized. The design choices behind this new interface are described also comparing with other widely adopted tools such as Pandas and Apache Spark. The programming model is illustrated highlighting the reduction of boilerplate code, composability of the actions and data transformations as well as the capabilities of dealing with different data sources such as ROOT, JSON, CSV or databases. Details are given about how the functional approach allows transparent implicit parallelization of the chain of operations specified by the user. The progress done in the field of distributed analysis is examined. In particular, the power of the integration of ROOT with Apache Spark via the PyROOT interface is shown. In addition, the building blocks for the expression of parallelism in ROOT are briefly characterized together with the structural changes applied in the building and testing infrastructure which were necessary to put them in production.},

  doi          = {10.1088/1742-6596/1085/4/042008},

  journal      = {Journal of Physics. Conference Series},

  number       = 4,

  volume       = 1085,

  place        = {United States},

  year         = {Thu Oct 18 00:00:00 EDT 2018},

  month        = {Thu Oct 18 00:00:00 EDT 2018}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1088/1742-6596/1085/4/042008

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 1 work

Citation information provided by
Web of Science

Figures / Tables:

Figure 1: Complex control ﬂows, which for example incarnates a series of cuts on collision events, can be concisely and expressively represented with TDataFrame.

All figures and tables (3 total)

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

SWAN: A service for interactive analysis in the cloud
journal, January 2018

Piparo, Danilo; Tejedor, Enric; Mato, Pere
Future Generation Computer Systems, Vol. 78
DOI: 10.1016/j.future.2016.11.035

Figures / Tables found in this record:

Figure 1 (p. 4)

Figure 2 (p. 5)

Figure 3 (p. 5)

Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.

Similar Records in DOE PAGES and OSTI.GOV collections:

Center for Technology for Advanced Scientific Componet Software (TASCS)

Technical Report Govindaraju, Madhusudhan

Advanced Scientific Computing Research Computer Science FY 2010Report Center for Technology for Advanced Scientific Component Software: Distributed CCA State University of New York, Binghamton, NY, 13902 Summary The overall objective of Binghamton's involvement is to work on enhancements of the CCA environment, motivated by the applications and research initiatives discussed in the proposal. This year we are working on re-focusing our design and development efforts to develop proof-of-concept implementations that have the potential to significantly impact scientific components. We worked on developing parallel implementations for non-hydrostatic code and worked on a model coupling interface for biogeochemical computations coded in MATLAB.more »« less
https://doi.org/10.2172/1092881

Full Text Available
Classification of River Catchments in the Contiguous United States: Code, Dataset, Similarity Patterns, and Resulting Classes

Dataset Ciulla, Fabio ; Varadharajan, Charuleka

This dataset serves as supplementary information for the paper by Ciulla F. and Varadharajan C. A Network Approach for Multiscale Catchment Classification using Traits (see reference 1). It contains environmental and physical catchment traits, such as temperatures, precipitation, land use and human interference, from 9067 sites across the contiguous United States (CONUS). The purpose of this dataset is to provide information for a better trait-based categorization of river catchments in the CONUS using networks as an analytical tool. The traits variables match the ones present in the GAGES-II dataset and the preprocessing steps are described in the Methods section (processed_dataset.csv).more »« less
https://doi.org/10.15485/1987555

View Dataset
Using Big Data Technologies for HEP Analysis

Journal Article Cremonesi, Matteo ; Bellini, Claudio ; Bian, Bianny ; ... - TBD

The HEP community is approaching an era were the excellent performances of the particle accelerators in delivering collision at high rate will force the experiments to record a large amount of information. The growing size of the datasets could potentially become a limiting factor in the capability to produce scientific results timely and efficiently. Recently, new technologies and new approaches have been developed in industry to answer to the necessity to retrieve information as quickly as possible to analyze PB and EB datasets. Providing the scientists with these modern computing tools will lead to rethinking the principles of data analysismore »« less
Full Text Available
CMS Analysis and Data Reduction with Apache Spark

Journal Article Gutsche, Oliver ; Canali, Luca ; Cremer, Illia ; ... - Journal of Physics. Conference Series

Experimental Particle Physics has been at the forefront of analyzing the world's largest datasets for decades. The HEP community was among the first to develop suitable software and computing tools for this task. In recent times, new toolkits and systems for distributed data processing, collectively called "Big Data" technologies have emerged from industry and open source projects to support the analysis of Petabyte and Exabyte datasets in industry. While the principles of data analysis in HEP have not changed (filtering and transforming experiment-specific data formats), these new technologies use different approaches and tools, promising a fresh look at analysis ofmore »« less
Cited by 3
https://doi.org/10.1088/1742-6596/1085/4/042030

Full Text Available
Expressing Parallelism with ROOT

Journal Article Piparo, D. ; Tejedor, E. ; Guiraud, E. ; ... - Journal of Physics. Conference Series

The need for processing the ever-increasing amount of data generated by the LHC experiments in a more efficient way has motivated ROOT to further develop its support for parallelism. Such support is being tackled both for shared-memory and distributed-memory environments. The incarnations of the aforementioned parallelism are multi-threading, multi-processing and cluster-wide executions. In the area of multi-threading, we discuss the new implicit parallelism and related interfaces, as well as the new building blocks to safely operate with ROOT objects in a multi-threaded environment. Regarding multi-processing, we review the new MultiProc framework, comparing it with similar tools (e.g. multiprocessing module inmore »« less
https://doi.org/10.1088/1742-6596/898/7/072022

Full Text Available

Similar Records

Title: Novel functional and distributed approaches to data analysis available in ROOT

Abstract

Citation Formats

Figures / Tables:

SWAN: A service for interactive analysis in the cloud journal, January 2018

SWAN: A service for interactive analysis in the cloud
journal, January 2018