Machine Learning Assisted HPC Workload Trace Generation for Leadership Scale Storage Systems

Paul, Arnab; Choi, Jong Youl; Karimi, Ahmad Maroof; Wang, Feiyi

doi:10.1145/3502181.3531457

Machine Learning Assisted HPC Workload Trace Generation for Leadership Scale Storage Systems

Conference · Wed Jun 01 04:00:00 EDT 2022

DOI:https://doi.org/10.1145/3502181.3531457· OSTI ID:1883852

Paul, Arnab ^[1]; ^[1]; ^[1]; ^[1]

ORNL

Monitoring and analyzing a wide range of I/O activities in an HPC cluster is important in maintaining mission-critical performance in a large-scale, multi-user, parallel storage system. Center-wide I/O traces can provide high-level information and fine-grained activities per application or per user running in the system. Studying such large-scale traces can provide helpful insights into the system. It can be used to develop predictive methods for making predictive decisions, adjusting scheduling policies, or providing decisions for the design of next-generation systems. However, sharing real-world I/O traces to expedite such research efforts leaves a few concerns; i) the cost of sharing the large traces is expensive due to this large size, and ii) privacy concern is an issue.We address such issues by building an end-to-end machine learn- ing (ML) workflow that can generate I/O traces for large-scale HPC applications. We leverage ML based feature selection and gener- ative models for I/O trace generation. The generative models are trained on I/O traces collected by the darshan I/O characterization tool over a period of one year. We present a two-step generation process consisting of two deep-learning models, called the feature generator and the trace generator. The combination of two-step generative models provides robustness by reducing the bias of the model and accounting for the stochastic nature of the I/O traces across different runs of an application. We evaluate the performance of the generative models and show that the two-step model can generate time-series I/O traces with less than 20% root mean square error.

View Conference

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 1883852

Country of Publication:: United States

Language:: English

References (18)

HPC Workload Characterization Using Feature Selection and Clustering Bang, Jiwoo; Kim, Chungyong; Wu, Kesheng HPDC '20: The 29th International Symposium on High-Performance Parallel and Distributed Computing, Proceedings of the 3rd International Workshop on Systems and Network Telemetry and Analytics https://doi.org/10.1145/3391812.3396270	conference	June 2020
Scalable I/O tracing and analysis Vijayakumar, Karthik; Mueller, Frank; Ma, Xiaosong Proceedings of the 4th Annual Workshop on Petascale Data Storage - PDSW '09 https://doi.org/10.1145/1713072.1713080	conference	January 2009
24/7 Characterization of petascale I/O workloads Carns, Philip; Latham, Robert; Ross, Robert 2009 IEEE International Conference on Cluster Computing and Workshops https://doi.org/10.1109/CLUSTR.2009.5289150	conference	August 2009
Techniques for modeling large-scale HPC I/O workloads Snyder, Shane; Carns, Philip Proceedings of the 6th International Workshop on Performance Modeling, Benchmarking, and Simulation of High Performance Computing Systems - PMBS '15 https://doi.org/10.1145/2832087.2832091	conference	January 2015
Analysis and Correlation of Application I/O Performance and System-Wide I/O Activity Madireddy, Sandeep; Balaprakash, Prasanna; Carns, Philip 2017 International Conference on Networking, Architecture, and Storage (NAS) https://doi.org/10.1109/NAS.2017.8026844	conference	August 2017
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
The Design and Implementation of a Domain-Specific Language for Network Performance Testing Pakin, Scott IEEE Transactions on Parallel and Distributed Systems, Vol. 18, Issue 10 https://doi.org/10.1109/TPDS.2007.1065	journal	October 2007
Normalized Mutual Information Feature Selection Estevez, P. A.; Tesmer, M.; Perez, C. A. IEEE Transactions on Neural Networks, Vol. 20, Issue 2 https://doi.org/10.1109/TNN.2008.2005601	journal	February 2009
Characterizing and Understanding HPC Job Failures Over The 2K-Day Life of IBM BlueGene/Q System Di, Sheng; Guo, Hanqi; Pershey, Eric 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) https://doi.org/10.1109/DSN.2019.00055	conference	June 2019
Modeling I/O Performance Variability Using Conditional Variational Autoencoders Madireddy, Sandeep; Balaprakash, Prasanna; Carns, Philip 2018 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2018.00022	conference	September 2018
Systematically inferring I/O performance variability by examining repetitive job behavior Costa, Emily; Patel, Tirthak; Schwaller, Benjamin Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3458817.3476186	conference	November 2021
Tbbt Zhu, Ningning; Chen, Jiawu; Chiueh, Tzi-cker ACM SIGMETRICS Performance Evaluation Review, Vol. 33, Issue 1 https://doi.org/10.1145/1071690.1064270	journal	June 2005
Characterizing Machine Learning I/O Workloads on Leadership Scale HPC Systems Paul, Arnab K.; Karimi, Ahmad Maroof; Wang, Feiyi 2021 29th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) https://doi.org/10.1109/MASCOTS53633.2021.9614303	conference	November 2021
A general regression neural network Specht, D. F. IEEE Transactions on Neural Networks, Vol. 2, Issue 6 https://doi.org/10.1109/72.97934	journal	January 1991
Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data Zhu, Yinhao; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios Journal of Computational Physics, Vol. 394 https://doi.org/10.1016/j.jcp.2019.05.024	journal	October 2019
Enabling Parallel Simulation of Large-Scale HPC Network Systems Mubarak, Misbah; Carothers, Christopher D.; Ross, Robert B. IEEE Transactions on Parallel and Distributed Systems, Vol. 28, Issue 1 https://doi.org/10.1109/TPDS.2016.2543725	journal	January 2017
Understanding HPC Application I/O Behavior Using System Level Statistics Paul, Arnab K.; Faaland, Olaf; Moody, Adam 2020 IEEE 27th International Conference on High Performance Computing, Data, and Analytics (HiPC) https://doi.org/10.1109/HiPC50609.2020.00034	conference	December 2020
Automatic Generation of I/O Kernels for HPC Applications Behzad, Babak; Dang, Hoang-Vu; Hariri, Farah 2014 9th Parallel Data Storage Workshop https://doi.org/10.1109/PDSW.2014.6	conference	November 2014

Similar Records

Characterizing Machine Learning I/O Workloads on Leadership Scale HPC Systems

Conference · Mon Nov 01 00:00:00 EDT 2021 · OSTI ID:1885376

DXT: Darshan eXtended Tracing

Conference · Tue Jan 08 23:00:00 EST 2019 · OSTI ID:1490709

Performance Evaluation of Darshan 3.0.0 on the Cray XC30

Technical Report · Fri Apr 01 00:00:00 EDT 2016 · OSTI ID:1250469

Machine Learning Assisted HPC Workload Trace Generation for Leadership Scale Storage Systems

Citation Formats

References (18)

Similar Records

Related Subjects