Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Toward Transparent Data Management in Multi-layer Storage Hierarchy for HPC Systems

Journal Article ·
 [1];  [2];  [1]
  1. Virginia Polytechnic Inst. and State Univ. (Virginia Tech), Blacksburg, VA (United States). Dept. of Computer Science
  2. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
+ Show Author Affiliations
Upcoming exascale high performance computing (HPC) systems are expected to comprise multi-tier storage hierarchy, and thus will necessitate innovative storage and I/O mechanisms. Traditional disk and block-based interfaces and file systems face severe challenges in utilizing capabilities of storage hierarchies due to the lack of hierarchy support and semantic interfaces. Object-based and semantically-rich data abstractions for scientific data management on large scale systems offer a sustainable solution to these challenges. Such data abstractions can also simplify users involvement in data movement. Here, we take the first steps of realizing such an object abstraction and explore storage mechanisms for these objects to enhance I/O performance, especially for scientific applications. We explore how an object-based interface can facilitate next generation scalable computing systems by presenting the mapping of data I/O from two real world HPC scientific use cases: a plasma physics simulation code (VPIC) and a cosmology simulation code (HACC). Our storage model stores data objects in different physical organizations to support data movement across layers of memory/storage hierarchy. Our implementation sclaes well to 16K parallel processes, and compared to the state of the art, such as MPI-IO and HDF5, our object-based data abstractions and data placement strategy in multi-level storage hierarchy achieves up to 7 X I/O performance improvement for scientific data.
Research Organization:
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
Sponsoring Organization:
National Science Foundation (NSF); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
Grant/Contract Number:
AC02-05CH11231
OSTI ID:
1435128
Country of Publication:
United States
Language:
English

References (9)

OSSD: A case for object-based solid state drives conference May 2013
Parallel I/O, analysis, and visualization of a trillion particle simulation
  • Byna, Surendra; Chou, Jerry; Rubel, Oliver
  • 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.92
conference November 2012
MapReduce: simplified data processing on large clusters journal January 2008
Integrating parallel file systems with object-based storage devices conference November 2007
RADOS: a scalable, reliable storage service for petabyte-scale storage clusters
  • Weil, Sage A.; Leung, Andrew W.; Brandt, Scott A.
  • Proceedings of the 2nd international workshop on Petascale data storage held in conjunction with Supercomputing '07 - PDSW '07 https://doi.org/10.1145/1374596.1374606
conference January 2007
An overview of the HDF5 technology suite and its applications conference January 2011
HACC: extreme scaling and performance across diverse architectures
  • Habib, Salman; Morozov, Vitali; Frontiere, Nicholas
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13 https://doi.org/10.1145/2503210.2504566
conference January 2013
A cost-effective, high-bandwidth storage architecture journal November 1998
On implementing MPI-IO portably and with high performance conference January 1999

Cited By (1)

Predicting file lifetimes for data placement in multi-tiered storage systems for HPC journal June 2021

Similar Records

Data Management in the Continuum: Cross-facility Object-based Data Transfers
Conference · Thu Oct 30 20:00:00 EDT 2025 · OSTI ID:3024529
Proactive Data Containers for Scientific Storage (Final Report)
Technical Report · Mon Dec 09 23:00:00 EST 2019 · OSTI ID:1577855
Interfacing HDF5 with a scalable object‐centric storage system on hierarchical storage
Journal Article · Sun Mar 08 20:00:00 EDT 2020 · Concurrency and Computation. Practice and Experience · OSTI ID:1603709

Related Subjects

97 MATHEMATICS AND COMPUTING