ROOT I/O compression improvements for HEP analysis

Title: ROOT I/O compression improvements for HEP analysis

Full Record
Other Related Research

Abstract

We overview recent changes in the ROOT I/O system, increasing performance and enhancing it and improving its interaction with other data analysis ecosystems. Both the newly introduced compression algorithms, the much faster bulk I/O data path, and a few additional techniques have the potential to significantly to improve experiment's software performance. The need for efficient lossless data compression has grown significantly as the amount of HEP data collected, transmitted, and stored has dramatically increased during the LHC era. While compression reduces storage space and, potentially, I/O bandwidth usage, it should not be applied blindly: there are significant trade-offs between the increased CPU cost for reading and writing files and the reduce storage space.

Authors:

^[1];

^[2]; Piparo, Danilo ^[3]; Zhang, Zhe ^[1]; Bockelman, Brian Paul ^[4]

U. Nebraska, Lincoln
Fermilab
CERN
Unlisted, US, MA

Publication Date:: 2020-04-08

Research Org.:: Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), High Energy Physics (HEP)

OSTI Identifier:: 1656647

Report Number(s):: arXiv:2004.10531; FERMILAB-CONF-20-308-SCD
oai:inspirehep.net:1792200

DOE Contract Number:: AC02-07CH11359

Resource Type:: Conference

Resource Relation:: Conference: 24th International Conference on Computing in High Energy and Nuclear Physics, Adelaide, Australia, 11/04-11/08/2019

Country of Publication:: United States

Language:: English

Citation Formats


                    Shadura, Oksana, Canal, Philippe, Piparo, Danilo, Zhang, Zhe, and Bockelman, Brian Paul. ROOT I/O compression improvements for HEP analysis.  United States: N. p., 2020. 
        Web.

Copy to clipboard


                    Shadura, Oksana, Canal, Philippe, Piparo, Danilo, Zhang, Zhe, & Bockelman, Brian Paul. ROOT I/O compression improvements for HEP analysis.  United States.

Copy to clipboard


                    Shadura, Oksana, Canal, Philippe, Piparo, Danilo, Zhang, Zhe, and Bockelman, Brian Paul. Wed .  
        "ROOT I/O compression improvements for HEP analysis".  United States.  https://www.osti.gov/servlets/purl/1656647.

Copy to clipboard


                    
@article{osti_1656647,

  title        = {ROOT I/O compression improvements for HEP analysis},

  author       = {Shadura, Oksana and Canal, Philippe and Piparo, Danilo and Zhang, Zhe and Bockelman, Brian Paul},

  abstractNote = {We overview recent changes in the ROOT I/O system, increasing performance and enhancing it and improving its interaction with other data analysis ecosystems. Both the newly introduced compression algorithms, the much faster bulk I/O data path, and a few additional techniques have the potential to significantly to improve experiment's software performance. The need for efficient lossless data compression has grown significantly as the amount of HEP data collected, transmitted, and stored has dramatically increased during the LHC era. While compression reduces storage space and, potentially, I/O bandwidth usage, it should not be applied blindly: there are significant trade-offs between the increased CPU cost for reading and writing files and the reduce storage space.},

  doi          = {},

  url          = {https://www.osti.gov/biblio/1656647},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2020},

  month        = {4}

}

Copy to clipboard

Conference:

View Conference

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

A Parallelised ROOT for Future HEP Data Processing

Journal Article Piparo, Danilo ; Canal, Philippe ; Amadio, Guilherme ; ... - EPJ Web of Conferences

In the coming years, HEP data processing will need to exploit parallelism on present and future hardware resources to sustain the bandwidth requirements. As one of the cornerstones of the HEP software ecosystem, ROOT embraced an ambitious parallelisation plan which delivered compelling results. In this contribution the strategy is characterised as well as its evolution in the medium term. The units of the ROOT framework are discussed where task and data parallelism have been introduced, with runtime and scaling measurements. We will give an overview of concurrent operations in ROOT, for instance in the areas of I/O (reading and writingmore »« less
https://doi.org/10.1051/epjconf/201921405033

Full Text Available
Optimizing Lossy Compression Rate-Distortion from Automatic Online Selection between SZ and ZFP

Journal Article Tao, Dingwen ; Di, Sheng ; Liang, Xin ; ... - IEEE Transactions on Parallel and Distributed Systems

With ever-increasing volumes of scientific data produced by high-performance computing applications, significantly reducing data size is critical because of limited capacity of storage space and potential bottlenecks on I/O or networks in writing/reading or transferring data. SZ and ZFP are two leading BSD licensed open source C/C++ libraries for compressed floating-point arrays that support high throughput read and write random access. However, their performance is not consistent across different data sets and across different fields of some data sets, which raises the need for an automatic online (during compression) selection between SZ and ZFP, with minimal overhead. In this paper,more »« less
Cited by 2
https://doi.org/10.1109/TPDS.2019.2894404

Full Text Available
Reducing Disk Storage of Full-3D Seismic Waveform Tomography (F3DT) Through Lossy Online Compression

Journal Article Lindstrom, Peter ; Chen, Po ; Lee, En-Jui - Computers and Geosciences

Full-3D seismic waveform tomography (F3DT) is the latest seismic tomography technique that can assimilate broadband, multi-component seismic waveform observations into high-resolution 3D subsurface seismic structure models. The main drawback in the current F3DT implementation, in particular the scattering-integral implementation (F3DT-SI), is the high disk storage cost and the associated I/O overhead of archiving the 4D space-time wavefields of the receiver- or source-side strain tensors. The strain tensor fields are needed for computing the data sensitivity kernels, which are used for constructing the Jacobian matrix in the Gauss-Newton optimization algorithm. In this study, we have successfully integrated a lossy compression algorithmmore »« less
Cited by 3
https://doi.org/10.1016/j.cageo.2016.04.009

Full Text Available
Active Flash: Out-of-core Data Analytics on Flash Storage

Conference Boboila, Simona ; Kim, Youngjae ; Vazhkudai, Sudharshan S ; ...

Next generation science will increasingly come to rely on the ability to perform efficient, on-the-fly analytics of data generated by high-performance computing (HPC) simulations, modeling complex physical phenomena. Scientific computing workflows are stymied by the traditional chaining of simulation and data analysis, creating multiple rounds of redundant reads and writes to the storage system, which grows in cost with the ever-increasing gap between compute and storage speeds in HPC clusters. Recent HPC acquisitions have introduced compute node-local flash storage as a means to alleviate this I/O bottleneck. We propose a novel approach, Active Flash, to expedite data analysis pipelines bymore »« less
Improving I/O Throughput with PRIMACY: Preconditioning ID-Mapper for Compressing Incompressibility

Conference Shah, Neil ; Schendel, Eric R. ; Lakshminarasimhan, Sriram ; ... - 2012 IEEE INTERNATIONAL CONFERENCE ON CLUSTER COMPUTING (CLUSTER)

The ability to efficiently handle massive amounts of data is necessary for the continuing development towards exascale scientific data-mining applications and database systems. Unfortunately, recent years have shown a growing gap between the size and complexity of data produced from scientific applications and the limited I/O bandwidth available on modern high-performance computing systems. Utilizing data compression in order to lower the degree of I/O activity offers a promising means to addressing this problem. However, the standard compression algorithms previously explored for such use offer limited gains on both the end-to-end throughput and storage fronts. In this paper, we introduce anmore »« less
https://doi.org/10.1109/CLUSTER.2012.16

Similar Records