SIRIUS: Enabling Progressive Data Exploration for Extreme-Scale Scientific Data

Qiao, Zhenbo; Lu, Tao; Luo, Huizhang; Liu, Qing; Klasky, Scott A.; Podhorszki, Norbert; Wang, Jinzhen

doi:10.1109/TMSCS.2018.2886851

Title: SIRIUS: Enabling Progressive Data Exploration for Extreme-Scale Scientific Data

Full Record
Other Related Research

Abstract

Scientific simulations on high performance computing (HPC) platforms generate large quantities of data. To bridge the widening gap between compute and I/O, and enable data to be more efficiently stored and analyzed, simulation outputs need to be refactored, reduced, and appropriately mapped to storage tiers. However, a systematic solution to support these steps has been lacking in the current HPC software ecosystem. To that end, this paper develops SIRIUS, a progressive JPEG-like data management scheme for storing and analyzing big scientific data. It co-designs data decimation, compression, and data storage, taking the hardware characteristics of each storage tier into considerations. With reasonably low overhead, our approach refactors simulation data, using either topological or uniform decimation, into a much smaller, reduced-accuracy base dataset, and a series of deltas that is used to augment the accuracy if needed. The base dataset and deltas are compressed and written to multiple storage tiers. Data saved on different tiers can then be selectively retrieved to restore the level of accuracy that satisfies data analytics. Thus, SIRIUS provides a paradigm shift towards elastic data analytics and enables end users to make trade-offs between analysis speed and accuracy on-the-fly. This paper further develops algorithms to preserve statisticsmore »« less

Authors:

^[1]; Lu, Tao ^[1];

^[1];

^[2];

^[1]

New Jersey Inst. of Technology, Newark, NJ (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Publication Date:: Fri Dec 14 00:00:00 EST 2018

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1559600

Grant/Contract Number:: AC05-00OR22725

Resource Type:: Accepted Manuscript

Journal Name:: IEEE Transactions on Multi-Scale Computing Systems

Additional Journal Information:: Journal Volume: 4; Journal Issue: 4; Journal ID: ISSN 2372-207X

Publisher:: IEEE

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; high-performance computing; data analytics; storage; data reduction; compression; progressive refactoring

Citation Formats


                    Qiao, Zhenbo, Lu, Tao, Luo, Huizhang, Liu, Qing, Klasky, Scott A., Podhorszki, Norbert, and Wang, Jinzhen. SIRIUS: Enabling Progressive Data Exploration for Extreme-Scale Scientific Data.  United States: N. p., 2018. 
Web.  doi:10.1109/TMSCS.2018.2886851.

Copy to clipboard


                    Qiao, Zhenbo, Lu, Tao, Luo, Huizhang, Liu, Qing, Klasky, Scott A., Podhorszki, Norbert, & Wang, Jinzhen. SIRIUS: Enabling Progressive Data Exploration for Extreme-Scale Scientific Data.  United States.  https://doi.org/10.1109/TMSCS.2018.2886851

Copy to clipboard


                    Qiao, Zhenbo, Lu, Tao, Luo, Huizhang, Liu, Qing, Klasky, Scott A., Podhorszki, Norbert, and Wang, Jinzhen. Fri .  
"SIRIUS: Enabling Progressive Data Exploration for Extreme-Scale Scientific Data".  United States.  https://doi.org/10.1109/TMSCS.2018.2886851.  https://www.osti.gov/servlets/purl/1559600.

Copy to clipboard


                    
@article{osti_1559600,

  title        = {SIRIUS: Enabling Progressive Data Exploration for Extreme-Scale Scientific Data},

  author       = {Qiao, Zhenbo and Lu, Tao and Luo, Huizhang and Liu, Qing and Klasky, Scott A. and Podhorszki, Norbert and Wang, Jinzhen},

  abstractNote = {Scientific simulations on high performance computing (HPC) platforms generate large quantities of data. To bridge the widening gap between compute and I/O, and enable data to be more efficiently stored and analyzed, simulation outputs need to be refactored, reduced, and appropriately mapped to storage tiers. However, a systematic solution to support these steps has been lacking in the current HPC software ecosystem. To that end, this paper develops SIRIUS, a progressive JPEG-like data management scheme for storing and analyzing big scientific data. It co-designs data decimation, compression, and data storage, taking the hardware characteristics of each storage tier into considerations. With reasonably low overhead, our approach refactors simulation data, using either topological or uniform decimation, into a much smaller, reduced-accuracy base dataset, and a series of deltas that is used to augment the accuracy if needed. The base dataset and deltas are compressed and written to multiple storage tiers. Data saved on different tiers can then be selectively retrieved to restore the level of accuracy that satisfies data analytics. Thus, SIRIUS provides a paradigm shift towards elastic data analytics and enables end users to make trade-offs between analysis speed and accuracy on-the-fly. This paper further develops algorithms to preserve statistics for data decimation, a common requirement for reducing data. Here, we assess the impact of SIRIUS on unstructured triangular meshes, a pervasive data model used in scientific simulations. In particular, we evaluate two realistic use cases: the blob detection in fusion and high-pressure area extraction in computational fluid dynamics.},

  doi          = {10.1109/TMSCS.2018.2886851},

  journal      = {IEEE Transactions on Multi-Scale Computing Systems},

  number       = 4,

  volume       = 4,

  place        = {United States},

  year         = {Fri Dec 14 00:00:00 EST 2018},

  month        = {Fri Dec 14 00:00:00 EST 2018}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1109/TMSCS.2018.2886851

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

Canopus: A Paradigm Shift Towards Elastic Extreme-Scale Data Analytics on HPC

Conference Lu, Tao ; Suchyta, Eric ; Pugmire, Dave ; ...

Scientific simulations on high performance computing (HPC) platforms generate large quantities of data. To bridge the widening gap between compute and I/O, and enable data to be more efficiently stored and analyzed, simulation outputs need to be refactored, reduced, and appropriately mapped to storage tiers. However, a systematic solution to support these steps has been lacking on the current HPC software ecosystem. To that end, this paper develops Canopus, a progressive JPEGlike data management scheme for storing and analyzing big scientific data. It co-designs the data decimation, compression and data storage, taking the hardware characteristics of each storage tier intomore »« less
https://doi.org/10.1109/CLUSTER.2017.62

Full Text Available
Understanding and Modeling Lossy Compression Schemes on HPC Scientific Data

Conference Lu, Tao ; Liu, Qing Gary ; He, Xubin ; ...

Scientific simulations generate large amounts of floating-point data, which are often not very compressible using the traditional reduction schemes, such as deduplication or lossless compression. The emergence of lossy floating-point compression holds promise to satisfy the data reduction demand from HPC applications; however, lossy compression has not been widely adopted in science production. We believe a fundamental reason is that there is a lack of understanding of the benefits, pitfalls, and performance of lossy compression on scientific data. In this paper, we conduct a comprehensive study on state-of-the-art lossy compression, including ZFP, SZ, and ISABELA, using real and representative HPCmore »« less
https://doi.org/10.1109/IPDPS.2018.00044

Full Text Available
Multi-level Layout Optimization for Efficient Spatio-temporal Queries on ISABELA-compressed Data

Conference Gong, Zhenhuan ; Lakshminarasimhan, Sriram ; Jenkins, John ; ... - 2012 IEEE 26TH INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM (IPDPS)

The size and scope of cutting-edge scientific simulations are growing much faster than the I/O subsystems of their runtime environments, not only making I/O the primary bottleneck, but also consuming space that pushes the storage capacities of many computing facilities. These problems are exacerbated by the need to perform data-intensive analytics applications, such as querying the dataset by variable and spatio-temporal constraints, for what current database technologies commonly build query indices of size greater than that of the raw data. To help solve these problems, we present a parallel query-processing engine that can handle both range queries and queries withmore »« less
https://doi.org/10.1109/IPDPS.2012.83
Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs

Conference Chen, Jieyang ; Wan, Lipeng ; Liang, Xin ; ...

Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth make it increasingly infeasible to store and share all data produced by scientific simulations. Instead, we need methods for reducing data volumes: ideally, methods that can scale data volumes adaptively so as to enable negotiation of performance and fidelity tradeoffs in different situations. Multigrid-based hierarchical data representations hold promise as a solution to this problem, allowing for flexible conversion between different fidelities so that, for example, data can be created at high fidelity and then transferred or stored at lower fidelity via logically simple andmore »« less
Full Text Available
Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs

Conference Chen, Jieyang ; Wang, Lipeng ; Liang, Xin ; ...

Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth make it increasingly infeasible to store and share all data produced by scientific simulations. Instead, we need methods for reducing data volumes: ideally, methods that can scale data volumes adaptively so as to enable negotiation of performance and fidelity tradeoffs in different situations. Multigrid-based hierarchical data representations hold promise as a solution to this problem, allowing for flexible conversion between different fidelities so that, for example, data can be created at high fidelity and then transferred or stored at lower fidelity via logically simple andmore »« less
https://doi.org/10.1109/IPDPS49936.2021.00095

Similar Records