Scientific User Behavior and Data-Sharing Trends in a Petascale File System
Abstract
The Oak Ridge Leadership Computing Facility (OLCF) runs the No. 4 supercomputer in the world, supported by a petascale file system, to facilitate scientific discovery. In this paper, using the daily file system metadata snapshots collected over 500 days, we have studied the behavioral trends of 1, 362 active users and 380 projects across 35 science domains. In particular, we have analyzed both individual and collective behavior of users and projects, highlighting needs from individual communities and the overall requirements to operate the file system. We have analyzed the metadata across three dimensions, namely (i) the projects' file generation and usage trends, using quantitative file system-centric metrics, (ii) scientific user behavior on the file system, and (iii) the data sharing trends of users and projects. To the best of our knowledge, our work is the first of its kind to provide comprehensive insights on user behavior from multiple science domains through metadata analysis of a large-scale shared file system. We envision that this OLCF case study will provide valuable insights for the design, operation, and management of storage systems at scale, and also encourage other HPC centers to undertake similar such efforts.
- Authors:
-
- ORNL
- Publication Date:
- Research Org.:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1427663
- DOE Contract Number:
- AC05-00OR22725
- Resource Type:
- Conference
- Resource Relation:
- Conference: SC17: International Conference for High Performance Computing, Networking, Storage and Analysis - Denver, Colorado, United States of America - 11/12/2017 5:00:00 AM-
- Country of Publication:
- United States
- Language:
- English
Citation Formats
Lim, Seung-Hwan, Sim, Hyogi, Vazhkudai, Sudharshan, and Gunasekaran, Raghul. Scientific User Behavior and Data-Sharing Trends in a Petascale File System. United States: N. p., 2017.
Web. doi:10.1145/3126908.3126924.
Lim, Seung-Hwan, Sim, Hyogi, Vazhkudai, Sudharshan, & Gunasekaran, Raghul. Scientific User Behavior and Data-Sharing Trends in a Petascale File System. United States. https://doi.org/10.1145/3126908.3126924
Lim, Seung-Hwan, Sim, Hyogi, Vazhkudai, Sudharshan, and Gunasekaran, Raghul. 2017.
"Scientific User Behavior and Data-Sharing Trends in a Petascale File System". United States. https://doi.org/10.1145/3126908.3126924. https://www.osti.gov/servlets/purl/1427663.
@article{osti_1427663,
title = {Scientific User Behavior and Data-Sharing Trends in a Petascale File System},
author = {Lim, Seung-Hwan and Sim, Hyogi and Vazhkudai, Sudharshan and Gunasekaran, Raghul},
abstractNote = {The Oak Ridge Leadership Computing Facility (OLCF) runs the No. 4 supercomputer in the world, supported by a petascale file system, to facilitate scientific discovery. In this paper, using the daily file system metadata snapshots collected over 500 days, we have studied the behavioral trends of 1, 362 active users and 380 projects across 35 science domains. In particular, we have analyzed both individual and collective behavior of users and projects, highlighting needs from individual communities and the overall requirements to operate the file system. We have analyzed the metadata across three dimensions, namely (i) the projects' file generation and usage trends, using quantitative file system-centric metrics, (ii) scientific user behavior on the file system, and (iii) the data sharing trends of users and projects. To the best of our knowledge, our work is the first of its kind to provide comprehensive insights on user behavior from multiple science domains through metadata analysis of a large-scale shared file system. We envision that this OLCF case study will provide valuable insights for the design, operation, and management of storage systems at scale, and also encourage other HPC centers to undertake similar such efforts.},
doi = {10.1145/3126908.3126924},
url = {https://www.osti.gov/biblio/1427663},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Wed Nov 01 00:00:00 EDT 2017},
month = {Wed Nov 01 00:00:00 EDT 2017}
}