Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale

Journal Article · · Bulletin of the IEEE Technical Committee on Data Engineering
OSTI ID:1606642
 [1];  [2];  [3];  [1];  [1];  [4];  [1];  [5];  [3];  [1];  [1];  [1];  [6];  [1];  [1];  [1];  [1];  [1];  [1];  [2] more »;  [1];  [1];  [1];  [1];  [7];  [1] « less
  1. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
  2. Brown Univ., Providence, RI (United States)
  3. Kitware, Inc., Clifton Park, NY (United States)
  4. Lawrence Berkeley National Laboratory (LBNL)
  5. Georgia Inst. of Technology, Atlanta, GA (United States)
  6. New Jersey Institute of Technology, Newark, NJ (United States)
  7. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
+ Show Author Affiliations
The Adaptable I/O System (ADIOS) represents the culmination of substantial investment in Scientific Data Management, and it has demonstrated success for several important extreme-scale science cases. However, looking towards the exascale and beyond, we see the development of yet more stringent data management requirements that require new abstractions. Therefore, there is an opportunity to attempt to connect the traditional realms of HPC I/O optimization with the Database / Data Management community. As such, in this paper we offer some specific examples from our ongoing work in managing data structures, services, and performance at the extreme scale for scientific computing. Using the publish/subscribe model afforded by ADIOS, we demonstrate a set of services that connect data format, metadata, queries, data reduction, and high-performance delivery. The resulting publish/subscribe framework facilitates connection to on-line workflow systems to enable the dynamic capabilities that will be required for exascale science.
Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)
Sponsoring Organization:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
Grant/Contract Number:
AC05-00OR22725; AC02-05CH11231
OSTI ID:
1606642
Journal Information:
Bulletin of the IEEE Technical Committee on Data Engineering, Journal Name: Bulletin of the IEEE Technical Committee on Data Engineering Journal Issue: 1 Vol. 43; ISSN 1053-1238
Publisher:
IEEE Computer SocietyCopyright Statement
Country of Publication:
United States
Language:
English

Similar Records

Organizing Large Data Sets for Efficient Analyses on HPC Systems
Conference · Fri Apr 01 00:00:00 EDT 2022 · OSTI ID:1871082
Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O
Conference · Wed May 01 00:00:00 EDT 2024 · OSTI ID:2439834
Proactive Data Containers for Scientific Storage (Final Report)
Technical Report · Mon Dec 09 23:00:00 EST 2019 · OSTI ID:1577855

Related Subjects

96 KNOWLEDGE MANAGEMENT AND PRESERVATION