Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale

Logan, Jeremy; Ainsworth, Mark; Atkins, Chuck; Chen, Jieyang; Choi, Jong Youl; Gu, Junmin; Kress, James M.; Eisenhauer, Greg; Geveci, Berk; Godoy, William; Kim, Mark B.; Kurc, Tahsin; Liu, Qing; Mehta, Kshitij V.; Ostrouchov, George; Podhorszki, Norbert; Pugmire, David; Suchyta, Eric D.; Thompson, Nicolas; Tugluk, Ozan; Wan, Lipeng; Wang, Ruonan; Whitney, Ben; Wolf, Matthew D.; Wu, Kesheng; Klasky, Scott A.

Title: Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale

Full Record
Other Related Research

Abstract

The Adaptable I/O System (ADIOS) represents the culmination of substantial investment in Scientific Data Management, and it has demonstrated success for several important extreme-scale science cases. However, looking towards the exascale and beyond, we see the development of yet more stringent data management requirements that require new abstractions. Therefore, there is an opportunity to attempt to connect the traditional realms of HPC I/O optimization with the Database / Data Management community. As such, in this paper we offer some specific examples from our ongoing work in managing data structures, services, and performance at the extreme scale for scientific computing. Using the publish/subscribe model afforded by ADIOS, we demonstrate a set of services that connect data format, metadata, queries, data reduction, and high-performance delivery. The resulting publish/subscribe framework facilitates connection to on-line workflow systems to enable the dynamic capabilities that will be required for exascale science.

Authors:

Logan, Jeremy ^[1]; Ainsworth, Mark ^[2]; Atkins, Chuck ^[3]; Chen, Jieyang ^[1];

^[1]; Gu, Junmin ^[4];

^[1]; Eisenhauer, Greg ^[5]; Geveci, Berk ^[3];

^[1];

^[1]; Kurc, Tahsin ^[1]; Liu, Qing ^[6];

^[1];

^[1]; Thompson, Nicolas ^[1]; Tugluk, Ozan ^[2] more »

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Brown Univ., Providence, RI (United States)
Kitware, Inc., Clifton Park, NY (United States)
Lawrence Berkeley National Laboratory (LBNL)
Georgia Inst. of Technology, Atlanta, GA (United States)
New Jersey Institute of Technology, Newark, NJ (United States)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Publication Date:: 2020-03-01

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:: 1606642

Grant/Contract Number:: AC05-00OR22725; AC02-05CH11231

Resource Type:: Accepted Manuscript

Journal Name:: Bulletin of the IEEE Technical Committee on Data Engineering

Additional Journal Information:: Journal Volume: 43; Journal Issue: 1; Related Information: http://sites.computer.org/debull/A20mar/issue1.htm; Journal ID: ISSN 1053-1238

Publisher:: IEEE Computer Society

Country of Publication:: United States

Language:: English

Subject:: 96 KNOWLEDGE MANAGEMENT AND PRESERVATION

Citation Formats


                    Logan, Jeremy, Ainsworth, Mark, Atkins, Chuck, Chen, Jieyang, Choi, Jong Youl, Gu, Junmin, Kress, James M., Eisenhauer, Greg, Geveci, Berk, Godoy, William, Kim, Mark B., Kurc, Tahsin, Liu, Qing, Mehta, Kshitij V., Ostrouchov, George, Podhorszki, Norbert, Pugmire, David, Suchyta, Eric D., Thompson, Nicolas, Tugluk, Ozan, Wan, Lipeng, Wang, Ruonan, Whitney, Ben, Wolf, Matthew D., Wu, Kesheng, and Klasky, Scott A.. Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale.  United States: N. p., 2020. 
Web.

Copy to clipboard


                    Logan, Jeremy, Ainsworth, Mark, Atkins, Chuck, Chen, Jieyang, Choi, Jong Youl, Gu, Junmin, Kress, James M., Eisenhauer, Greg, Geveci, Berk, Godoy, William, Kim, Mark B., Kurc, Tahsin, Liu, Qing, Mehta, Kshitij V., Ostrouchov, George, Podhorszki, Norbert, Pugmire, David, Suchyta, Eric D., Thompson, Nicolas, Tugluk, Ozan, Wan, Lipeng, Wang, Ruonan, Whitney, Ben, Wolf, Matthew D., Wu, Kesheng, & Klasky, Scott A.. Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale.  United States.

Copy to clipboard


                    Logan, Jeremy, Ainsworth, Mark, Atkins, Chuck, Chen, Jieyang, Choi, Jong Youl, Gu, Junmin, Kress, James M., Eisenhauer, Greg, Geveci, Berk, Godoy, William, Kim, Mark B., Kurc, Tahsin, Liu, Qing, Mehta, Kshitij V., Ostrouchov, George, Podhorszki, Norbert, Pugmire, David, Suchyta, Eric D., Thompson, Nicolas, Tugluk, Ozan, Wan, Lipeng, Wang, Ruonan, Whitney, Ben, Wolf, Matthew D., Wu, Kesheng, and Klasky, Scott A.. Sun .  
"Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale".  United States.  https://www.osti.gov/servlets/purl/1606642.

Copy to clipboard


                    
@article{osti_1606642,

  title        = {Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale},

  author       = {Logan, Jeremy and Ainsworth, Mark and Atkins, Chuck and Chen, Jieyang and Choi, Jong Youl and Gu, Junmin and Kress, James M. and Eisenhauer, Greg and Geveci, Berk and Godoy, William and Kim, Mark B. and Kurc, Tahsin and Liu, Qing and Mehta, Kshitij V. and Ostrouchov, George and Podhorszki, Norbert and Pugmire, David and Suchyta, Eric D. and Thompson, Nicolas and Tugluk, Ozan and Wan, Lipeng and Wang, Ruonan and Whitney, Ben and Wolf, Matthew D. and Wu, Kesheng and Klasky, Scott A.},

  abstractNote = {The Adaptable I/O System (ADIOS) represents the culmination of substantial investment in Scientific Data Management, and it has demonstrated success for several important extreme-scale science cases. However, looking towards the exascale and beyond, we see the development of yet more stringent data management requirements that require new abstractions. Therefore, there is an opportunity to attempt to connect the traditional realms of HPC I/O optimization with the Database / Data Management community. As such, in this paper we offer some specific examples from our ongoing work in managing data structures, services, and performance at the extreme scale for scientific computing. Using the publish/subscribe model afforded by ADIOS, we demonstrate a set of services that connect data format, metadata, queries, data reduction, and high-performance delivery. The resulting publish/subscribe framework facilitates connection to on-line workflow systems to enable the dynamic capabilities that will be required for exascale science.},

  doi          = {},

  journal      = {Bulletin of the IEEE Technical Committee on Data Engineering},

  number       = 1,

  volume       = 43,

  place        = {United States},

  year         = {2020},

  month        = {3}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

The DOI is not currently available

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

Final Technical Report - Proactive Data Containers for Scientific Storage

Technical Report Soumagne, Jerome ; Warren, Richard ; Mu, Jingqing ; ...

Emerging HPC systems are expected to be deployed with an unprecedented level of complexity, due to a deep system memory/storage hierarchy and heterogeneity of the storage hardware. This hierarchy is expected to range from CPU cache through several levels of volatile memory to nonvolatile memory, traditional hard disks, and tape. Simple and efficient methods of data management and movement through this hierarchy is critical for scientific applications using exascale systems. Existing storage system and I/O (SSIO) technologies face severe challenges in dealing with these requirements. POSIX and MPI I/O standards that are the basis for existing I/O libraries and parallelmore »« less
https://doi.org/10.2172/1577855

Full Text Available
Toward designing effective exascale scientific computing workflows: experiences and best practices

Conference Coletti, Mark ; Davidson, Russ ; Sedova, Ada

Many fields within scientific computing have embraced advances in big-data analysis and machine learning, which often requires the deployment of large, distributed and complicated workflows that may combine training neural networks, performing simulations, running inference, and performing database queries and data analysis in asynchronous, parallel and pipelined execution frameworks. Such a shift has brought into focus the need for scalable, efficient workflow management solutions with reproducibility, error and provenance handling, traceability, and checkpoint-restart capabilities, among other needs. Here, we discuss challenges and best-practices for deploying exascale-generation computational science workflows on resources at the Oak Ridge Leadership Computing Facility (OLCF). Wemore »« less
Full Text Available
A View from ORNL: Scientific Data Research Opportunities in the Big Data Age

Conference Klasky, Scott A. ; Wolf, Matthew D. ; Ainsworth, Mark ; ...

One of the core issues across computer and computational science today is adapting to, managing, and learning from the influx of "Big Data". In the commercial space, this problem has led to a huge investment in new technologies and capabilities that are well adapted to dealing with the sorts of human-generated logs, videos, texts, and other large-data artifacts that are processed and resulted in an explosion of useful platforms and languages (Hadoop, Spark, Pandas, etc.). However, translating this work from the enterprise space to the computational science and HPC community has proven somewhat difficult, in part because of some ofmore »« less
https://doi.org/10.1109/ICDCS.2018.00136

Full Text Available
PARLO: PArallel Run-Time Layout Optimization for Scientific Data Explorations with Heterogeneous Access Pattern

Conference Gong, Zhenhuan ; Boyuka, David ; Zou, X ; ...

Download Citation Email Print Request Permissions Save to Project The size and scope of cutting-edge scientific simulations are growing much faster than the I/O and storage capabilities of their run-time environments. The growing gap is exacerbated by exploratory, data-intensive analytics, such as querying simulation data with multivariate, spatio-temporal constraints, which induces heterogeneous access patterns that stress the performance of the underlying storage system. Previous work addresses data layout and indexing techniques to improve query performance for a single access pattern, which is not sufficient for complex analytics jobs. We present PARLO a parallel run-time layout optimization framework, to achieve multi-levelmore »« less
PARLO: PArallel Run-Time Layout Optimization for Scientific Data Explorations with Heterogeneous Access Patterns

Conference Gong, Zhenhuan ; Boyuka, David A. ; Zou, Xiaocheng ; ... - PROCEEDINGS OF THE 2013 13TH IEEE/ACM INTERNATIONAL SYMPOSIUM ON CLUSTER, CLOUD AND GRID COMPUTING (CCGRID 2013)

The size and scope of cutting-edge scientific simulations are growing much faster than the I/O and storage capabilities of their run-time environments. The growing gap is exacerbated by exploratory, data-intensive analytics, such as querying simulation data with multivariate, spatio-temporal constraints, which induces heterogeneous access patterns that stress the performance of the underlying storage system. Previous work addresses data layout and indexing techniques to improve query performance for a single access pattern, which is not sufficient for complex analytics jobs. We present PARLO a parallel run-time layout optimization framework, to achieve multi-level data layout optimization for scientific applications at run-time beforemore »« less
https://doi.org/10.1109/CCGrid.2013.58

Similar Records