Ad Hoc File Systems for High-Performance Computing

Brinkmann, André; Mohror, Kathryn; Yu, Weikuan; Carns, Philip; Cortes, Toni; Klasky, Scott A.; Miranda, Alberto; Pfreundt, Franz-Josef; Ross, Robert B.; Vef, Marc-André

doi:10.1007/s11390-020-9801-1

Title: Ad Hoc File Systems for High-Performance Computing

Journal Article · 17 January 2020 · Journal of Computer Science and Technology

DOI: https://doi.org/10.1007/s11390-020-9801-1 · OSTI ID:1596689

Brinkmann, André ^[1]; Mohror, Kathryn ^[2]; Yu, Weikuan ^[3]; Carns, Philip ^[4]; Cortes, Toni ^[5]; Klasky, Scott A. ^[6]; Miranda, Alberto ^[7]; Pfreundt, Franz-Josef ^[8]; Ross, Robert B. ^[4]; Vef, Marc-André ^[1]

Johannes Gutenberg Univ., Mainz (Germany)
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
Florida State Univ., Tallahassee, FL (United States)
Argonne National Lab. (ANL), Argonne, IL (United States)
Univ. Politecnica de Catalunya, Barcelona (Spain)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Barcelona Supercomputing Center, Barcelona (Spain)
Fraunhofer Inst. for Industrial Mathematics ITWM, Kaiserslautern (Germany)

Storage backends of parallel compute clusters are still based mostly on magnetic disks, while newer and faster storage technologies such as flash-based SSDs or non-volatile random access memory (NVRAM) are deployed within compute nodes. Including these new storage technologies into scientific workflows is unfortunately today a mostly manual task, and most scientists therefore do not take advantage of the faster storage media. One approach to systematically include nodelocal SSDs or NVRAMs into scientific workflows is to deploy ad hoc file systems over a set of compute nodes, which serve as temporary storage systems for single applications or longer-running campaigns. This paper presents results from the Dagstuhl Seminar 17202 “Challenges and Opportunities of User-Level File Systems for HPC” and discusses application scenarios as well as design strategies for ad hoc file systems using node-local storage media. The discussion includes open research questions, such as how to couple ad hoc file systems with the batch scheduling environment and how to schedule stage-in and stage-out processes of data between the storage backend and the ad hoc file systems. Also presented are strategies to build ad hoc file systems by using reusable components for networking and how to improve storage device compatibility. Various interfaces and semantics are presented, for example those used by the three ad hoc file systems BeeOND, GekkoFS, and BurstFS. Their presentation covers a range from file systems running in production to cutting-edge research focusing on reaching the performance limits of the underlying devices.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States); Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); German Research Foundation (DFG); European Union (EU); Spanish Ministry of Science and Innovation (MICINN); National Science Foundation (NSF); USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: AC02-06CH11357; 1561041; 1564647; 1744336; 1763547; 1822737; 2014-SGR-1051; TIN2015-65316; 671591; AC52-07NA27344

OSTI ID:: 1596689

Alternate ID(s):: OSTI ID: 1606092

Report Number(s):: LLNL-JRNL-779789; 155300

Journal Information:: Journal of Computer Science and Technology, Vol. 35, Issue 1; ISSN 1000-9000

Publisher:: Springer NatureCopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 13 works

Citation information provided by
Web of Science

References (62)

Characterizing output bottlenecks in a supercomputer Xie, Bing; Chase, Jeffrey; Dillow, David 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.28	conference	November 2012
TRIO: Burst Buffer Based I/O Orchestration Wang, Teng; Oral, Sarp; Pritchard, Michael 2015 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2015.38	conference	September 2015
Scaling Embedded In-Situ Indexing with DeltaFS Zheng, Qing; Cranor, Charles D.; Guo, Danhao SC18: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2018.00006	conference	November 2018
An introduction to disk drive modeling Ruemmler, C.; Wilkes, J. Computer, Vol. 27, Issue 3 https://doi.org/10.1109/2.268881	journal	March 1994
‘Big data’, Hadoop and cloud computing in genomics O’Driscoll, Aisling; Daugelaite, Jurate; Sleator, Roy D. Journal of Biomedical Informatics, Vol. 46, Issue 5 https://doi.org/10.1016/j.jbi.2013.07.001	journal	October 2013
Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS) Lofstead, Jay F.; Klasky, Scott; Schwan, Karsten Proceedings of the 6th international workshop on Challenges of large applications in distributed environments - CLADE '08 https://doi.org/10.1145/1383529.1383533	conference	January 2008
The IBM Blue Gene/Q interconnection network and message unit Chen, Dong; Parker, Jeffrey J.; Eisley, Noel A. Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063419	conference	January 2011
Entropy-Aware I/O Pipelining for Large-Scale Deep Learning on HPC Systems Zhu, Yue; Chowdhury, Fahim; Fu, Huansong 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) https://doi.org/10.1109/MASCOTS.2018.00023	conference	September 2018
Brochure on Storage Systems and I/O: Organizing, Storing, and Accessing Data for Scientific Discovery Ross, Robert; Ward, Lee; Carns, Philip https://doi.org/10.2172/1544750	report	July 2019
LPCC: hierarchical persistent client caching for lustre Qian, Yingjin; Li, Xi; Ihara, Shuichi SC '19: The International Conference for High Performance Computing, Networking, Storage, and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3295500.3356139	conference	November 2019
Task-based programming in COMPSs to converge from HPC to big data Conejero, Javier; Corella, Sandra; Badia, Rosa M. The International Journal of High Performance Computing Applications, Vol. 32, Issue 1 https://doi.org/10.1177/1094342017701278	journal	April 2017
On the role of burst buffers in leadership-class storage systems Liu, Ning; Cope, Jason; Carns, Philip 2012 IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2012.6232369	conference	April 2012
FusionFS: Toward supporting data-intensive scientific applications on extreme-scale high-performance computing systems Zhao, Dongfang; Zhang, Zhao; Zhou, Xiaobing 2014 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2014.7004214	conference	October 2014
MCREngine: A scalable checkpointing system using data-aware aggregation and compression Islam, Tanzima Zerin; Mohror, Kathryn; Bagchi, Saurabh 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.77	conference	November 2012
Direct lookup and hash-based metadata placement for local file systems Lensing, Paul Hermann; Cortes, Toni; Brinkmann, André Proceedings of the 6th International Systems and Storage Conference on - SYSTOR '13 https://doi.org/10.1145/2485732.2485741	conference	January 2013
An overview of the HDF5 technology suite and its applications Folk, Mike; Heber, Gerd; Koziol, Quincey Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases - AD '11 https://doi.org/10.1145/1966895.1966900	conference	January 2011
Cray Cascade: A scalable HPC system based on a Dragonfly network Faanes, Greg; Bataineh, Abdulla; Roweth, Duncan 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.39	conference	November 2012
Efficient Data-Movement for Lightweight I/O Oldfield, Ron; Widener, Patrick; Maccabe, Arthur 2006 IEEE International Conference on Cluster Computing https://doi.org/10.1109/CLUSTR.2006.311897	conference	September 2006
Harmonia: An Interference-Aware Dynamic I/O Scheduler for Shared Non-volatile Burst Buffers Kougkas, Anthony; Devarajan, Hariharan; Sun, Xian-He 2018 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2018.00046	conference	September 2018
Design, Modeling, and Evaluation of a Scalable Multi-level Checkpointing System Moody, Adam; Bronevetsky, Greg; Mohror, Kathryn 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2010.18	conference	November 2010
Methodology for the Rapid Development of Scalable HPC Data Services Dorier, Matthieu; Settlemyer, Brad; Shipman, Galen 2018 IEEE/ACM 3rd International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems (PDSW-DISCS) https://doi.org/10.1109/PDSW-DISCS.2018.00013	conference	November 2018
Data Elevator: Low-Contention Data Movement in Hierarchical Storage System Dong, Bin; Byna, Suren; Wu, Kesheng 2016 IEEE 23rd International Conference on High Performance Computing (HiPC) https://doi.org/10.1109/HiPC.2016.026	conference	December 2016
Stacker: An Autonomic Data Movement Engine for Extreme-Scale Data Staging-Based In-Situ Workflows Subedi, Pradeep; Davis, Philip; Duan, Shaohua SC18: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2018.00076	conference	November 2018
ROOT — A C++ framework for petabyte data storage, statistical analysis and visualization Antcheva, I.; Ballintijn, M.; Bellenot, B. Computer Physics Communications, Vol. 182, Issue 6 https://doi.org/10.1016/j.cpc.2011.02.008	journal	June 2011
A Brief Introduction to the OpenFabrics Interfaces - A New Network API for Maximizing High Performance Application Efficiency Grun, Paul; Hefty, Sean; Sur, Sayantan 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects (HOTI) https://doi.org/10.1109/HOTI.2015.19	conference	August 2015
Deduplication Potential of HPC Applications’ Checkpoints Kaiser, Jurgen; Gad, Ramy; SuB, Tim 2016 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2016.32	conference	September 2016
UCX: An Open Source Framework for HPC Network APIs and Beyond Shamis, Pavel; Venkata, Manjunath Gorentla; Lopez, M. Graham 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects (HOTI) https://doi.org/10.1109/HOTI.2015.13	conference	August 2015
GekkoFS - A Temporary Distributed File System for HPC Applications Vef, Marc-Andre; Moti, Nafiseh; SuB, Tim 2018 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2018.00049	conference	September 2018
High-Performance Design of YARN MapReduce on Modern HPC Clusters with Lustre and RDMA Wasi-ur-Rahman, Md.; Lu, Xiaoyi; Islam, Nusrat Sharmin 2015 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2015.83	conference	May 2015
Improving Collective I/O Performance Using Non-volatile Memory Devices Congiu, Giuseppe; Narasimhamurthy, Sai; Suss, Tim 2016 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2016.37	conference	September 2016
On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems Yildiz, Orcun; Dorier, Matthieu; Ibrahim, Shadi 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2016.50	conference	May 2016
NetCDF: an interface for scientific data access Rew, R.; Davis, G. IEEE Computer Graphics and Applications, Vol. 10, Issue 4 https://doi.org/10.1109/38.56302	journal	July 1990
Optimizing a hybrid SSD/HDD HPC storage system based on file size distributions Welch, Brent; Noer, Geoffrey 2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2013.6558449	conference	May 2013
Apache Spark: a unified engine for big data processing Zaharia, Matei; Franklin, Michael J.; Ghodsi, Ali Communications of the ACM, Vol. 59, Issue 11 https://doi.org/10.1145/2934664	journal	October 2016
Search and clustering orders of magnitude faster than BLAST Edgar, Robert C. Bioinformatics, Vol. 26, Issue 19, p. 2460-2461 https://doi.org/10.1093/bioinformatics/btq461	journal	August 2010
A configurable rule based classful token bucket filter network request scheduler for the lustre file system Qian, Yingjin; Li, Xi; Ihara, Shuichi Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17 https://doi.org/10.1145/3126908.3126932	conference	January 2017
The Hadoop Distributed File System Shvachko, Konstantin; Kuang, Hairong; Radia, Sanjay 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2010.5496972	conference	May 2010
Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the World Wide Web Karger, David; Lehman, Eric; Leighton, Tom Proceedings of the twenty-ninth annual ACM symposium on Theory of computing - STOC '97 https://doi.org/10.1145/258533.258660	conference	January 1997
Scientific computing meets big data technology: An astronomy use case Zhang, Zhao; Barbary, Kyle; Nothaft, Frank Austin 2015 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2015.7363840	conference	October 2015
Challenges and Solutions for Tracing Storage Systems: A Case Study with Spectrum Scale Vef, Marc-André; Tarasov, Vasily; Hildebrand, Dean ACM Transactions on Storage, Vol. 14, Issue 2 https://doi.org/10.1145/3149376	journal	April 2018
Understanding and Improving Computational Science Storage Access through Continuous Characterization Carns, Philip; Harms, Kevin; Allcock, William ACM Transactions on Storage, Vol. 7, Issue 3, p. 1-26 https://doi.org/10.1145/2027066.2027068	journal	October 2011
PLFS: a checkpoint filesystem for parallel applications Bent, John; Gibson, Garth; Grider, Gary https://doi.org/10.1145/1654059.1654081	conference	January 2009
An Overview of the Atmospheric Component of the Energy Exascale Earth System Model Rasch, P. J.; Xie, S.; Ma, P. ‐L. Journal of Advances in Modeling Earth Systems, Vol. 11, Issue 8 https://doi.org/10.1029/2019MS001629	journal	August 2019
On the Quality of Wall Time Estimates for Resource Allocation Prediction Soysal, Mehmet; Berghoff, Marco; Klusáček, Dalibor Proceedings of the 48th International Conference on Parallel Processing: Workshops - ICPP 2019 https://doi.org/10.1145/3339186.3339204	conference	January 2019
Mercury: Enabling remote procedure call for high-performance computing Soumagne, Jerome; Kimpe, Dries; Zounmevo, Judicael 2013 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2013.6702617	conference	September 2013
Qthreads: An API for programming with millions of lightweight threads Wheeler, Kyle B.; Murphy, Richard C.; Thain, Douglas Distributed Processing Symposium (IPDPS), 2008 IEEE International Symposium on Parallel and Distributed Processing https://doi.org/10.1109/IPDPS.2008.4536359	conference	April 2008
Managing Variability in the IO Performance of Petascale Storage Systems Lofstead, Jay; Zheng, Fang; Liu, Qing 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2010.32	conference	November 2010
Snakemake—a scalable bioinformatics workflow engine Köster, Johannes; Rahmann, Sven Bioinformatics, Vol. 34, Issue 20 https://doi.org/10.1093/bioinformatics/bty350	journal	May 2018
Managing I/O Interference in a Shared Burst Buffer System Thapaliya, Sagar; Bangalore, Purushotham; Lofstead, Jat 2016 45th International Conference on Parallel Processing (ICPP) https://doi.org/10.1109/ICPP.2016.54	conference	August 2016
SSD Failures in Datacenters: What? When? and Why? Narayanan, Iyswarya; Vaid, Kushagra; Wang, Di Proceedings of the 9th ACM International on Systems and Storage Conference - SYSTOR '16 https://doi.org/10.1145/2928275.2928278	conference	January 2016
Argobots: A Lightweight Low-Level Threading and Tasking Framework Seo, Sangmin; Amer, Abdelhalim; Balaji, Pavan IEEE Transactions on Parallel and Distributed Systems, Vol. 29, Issue 3 https://doi.org/10.1109/TPDS.2017.2766062	journal	March 2018
Exascale Deep Learning for Climate Analytics Kurth, Thorsten; Treichler, Sean; Romero, Joshua SC18: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2018.00054	conference	November 2018
Poster: Portals 4 Network Programming Interface Barrett, Brian; Brightwell, Ron; Underwood, Keith 2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion: High Performance Computing, Networking Storage and Analysis https://doi.org/10.1109/SC.Companion.2012.264	conference	November 2012
NORNS: Extending Slurm to Support Data-Driven Workflows through Asynchronous Data Staging Miranda, Alberto; Jackson, Adrian; Tocci, Tommaso 2019 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2019.8891014	conference	September 2019
A Large-Scale Study of Flash Memory Failures in the Field Meza, Justin; Wu, Qiang; Kumar, Sanjev ACM SIGMETRICS Performance Evaluation Review, Vol. 43, Issue 1 https://doi.org/10.1145/2796314.2745848	journal	June 2015
High Performance RDMA-Based MPI Implementation over InfiniBand Liu, Jiuxing; Wu, Jiesheng; Panda, Dhabaleswar K. International Journal of Parallel Programming, Vol. 32, Issue 3 https://doi.org/10.1023/B:IJPP.0000029272.69895.c1	journal	June 2004
A 1 PB/s file system to checkpoint three million MPI tasks Rajachandrasekar, Raghunath; Moody, Adam; Mohror, Kathryn Proceedings of the 22nd international symposium on High-performance parallel and distributed computing - HPDC '13 https://doi.org/10.1145/2493123.2462908	conference	January 2013
Parallel netCDF: A High-Performance Scientific I/O Interface Li, Jianwei; Zingale, Michael; Liao, Wei-keng Proceedings of the 2003 ACM/IEEE conference on Supercomputing - SC '03 https://doi.org/10.1145/1048935.1050189	conference	January 2003
Performance and extension of user space file systems Rajgarhia, Aditya; Gehani, Ashish Proceedings of the 2010 ACM Symposium on Applied Computing - SAC '10 https://doi.org/10.1145/1774088.1774130	conference	January 2010
An Ephemeral Burst-Buffer File System for Scientific Applications Wang, Teng; Mohror, Kathryn; Moody, Adam SC16: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2016.68	conference	November 2016
File System Scalability with Highly Decentralized Metadata on Independent Storage Devices Lensing, Paul Hermann; Cortes, Toni; Hughes, Jim 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) https://doi.org/10.1109/CCGrid.2016.28	conference	May 2016
A 1 PB/s file system to checkpoint three million MPI tasks Rajachandrasekar, Raghunath; Moody, Adam; Mohror, Kathryn HPDC'13: The 22nd International Symposium on High-Performance Parallel and Distributed Computing, Proceedings of the 22nd international symposium on High-performance parallel and distributed computing https://doi.org/10.1145/2462902.2462908	conference	October 2018

Similar Records

Programming Abstractions for Managing Workflows on Tiered Storage Systems

Journal Article · 2021 · ACM Transactions on Storage · OSTI ID:1898543

Ghoshal, Devarshi; Ramakrishnan, Lavanya

Towards a Scalable and Adaptive Application Support Platform for Large-Scale Distributed E-Sciences in High-Performance Network Environments

Technical Report · 2016 · OSTI ID:1255487

Wu, Chase Qishi; Zhu, Michelle Mengxia

Virtual machine-based simulation platform for mobile ad-hoc network-based cyber infrastructure

Journal Article · 2015 · Journal of Defense Modeling and Simulation · OSTI ID:1254082

Yoginath, Srikanth B.; Perumalla, Kayla S.; Henz, Brian J.

Related Subjects

97 MATHEMATICS AND COMPUTING
Burst Buffers
Distributed File Systems
High-Performance Computing
POSIX
Parallel Architectures

Title: Ad Hoc File Systems for High-Performance Computing

Citation Formats

References (62)

Similar Records

Related Subjects