Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

A Quantitative Approach to Architecting All-Flash Lustre File Systems

Conference ·
; ; ; ; ;
New experimental and AI-driven workloads are moving into the realm of extreme-scale HPC systems at the same time that high-performance flash is becoming cost-effective to deploy at scale. This confluence poses a number of new technical and economic challenges and opportunities in designing the next generation of HPC storage and I/O subsystems to achieve the right balance of bandwidth, latency, endurance, and cost. In this work, we present quantitative models that use workload data from existing, disk-based file systems to project the architectural requirements of all-flash Lustre file systems. Using data from NERSC’s Cori I/O subsystem, we then demonstrate the minimum required capacity for data, capacity for metadata and data-on-MDT, and SSD endurance for a future all-flash Lustre file system.
Research Organization:
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
Sponsoring Organization:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
DOE Contract Number:
AC02-05CH11231
OSTI ID:
1827652
Country of Publication:
United States
Language:
English

References (14)

Group-based variant calling leveraging next-generation supercomputing for large-scale whole-genome sequencing studies journal September 2015
Comparative I/O workload characterization of two leadership class storage clusters conference January 2015
Optimization of SAMtools sorting using OpenMP tasks journal April 2017
Diving into petascale production file systems through large scale profiling and analysis
  • Wang, Feiyi; Sim, Hyogi; Harr, Cameron
  • Proceedings of the 2nd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems - PDSW-DISCS '17 https://doi.org/10.1145/3149393.3149399
conference January 2017
Storage utilization in the long tail of science
  • Lockwood, Glenn K.; Tatineni, Mahidhar; Wagner, Rick
  • Proceedings of the 2015 XSEDE Conference on Scientific Advancements Enabled by Enhanced Cyberinfrastructure - XSEDE '15 https://doi.org/10.1145/2792745.2792777
conference January 2015
GUIDE: a scalable information directory service to collect, federate, and analyze logs for operational insights into a leadership HPC facility
  • Vazhkudai, Sudharshan S.; Miller, Ross; Tiwari, Devesh
  • SC '17: The International Conference for High Performance Computing, Networking, Storage and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3126908.3126946
conference November 2017
A Metric for Evaluating Supercomputer Performance in the Era of Extreme Heterogeneity conference November 2018
Gordon: design, performance, and experiences deploying and supporting a data intensive supercomputer
  • Strande, Shawn M.; Cicotti, Pietro; Sinkovits, Robert S.
  • Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment on Bridging from the eXtreme to the campus and beyond - XSEDE '12 https://doi.org/10.1145/2335755.2335789
conference January 2012
Performance characterization of scientific workflows for the optimal use of Burst Buffers journal September 2020
Moore’s law realities for recording systems and memory storage components: HDD, tape, NAND, and optical journal May 2018
Best practices for managing large CryoEM facilities journal September 2017
Extreme I/O on HPC for HEP using the Burst Buffer at NERSC journal October 2017
Data systems for the Linac Coherent Light Source journal July 2016
Cataloging the Visible Universe Through Bayesian Inference at Petascale conference May 2018

Similar Records

Architecture and Performance of Perlmutter’s 35 PB ClusterStor E1000 All-Flash File System
Conference · Thu Dec 31 23:00:00 EST 2020 · OSTI ID:1798757
Architecture and performance of Perlmutter's 35 PB ClusterStor E1000 all-flash file system
Journal Article · Tue Jul 23 00:00:00 EDT 2024 · Concurrency and Computation. Practice and Experience · OSTI ID:2440410

Related Subjects