AIIO: Using Artificial Intelligence for Job-Level and Automatic I/O Performance Bottleneck Diagnosis

Dong, Bin; Bez, Jean Luca; Byna, Suren

doi:10.1145/3588195.3592986

AIIO: Using Artificial Intelligence for Job-Level and Automatic I/O Performance Bottleneck Diagnosis

Conference · 07 August 2023 · Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing

DOI:https://doi.org/10.1145/3588195.3592986· OSTI ID:2228856

^[1]; ^[1]; ^[2]

Lawrence Berkeley National Laboratory, Berkeley, CA, USA
The Ohio State University & Lawrence Berkeley National Laboratory, Columbus, OH, USA

Manually diagnosing the I/O performance bottleneck for a single application (hereinafter referred to as the "job level'') is a tedious and error-prone procedure requiring domain scientists to have deep knowledge of complex storage systems. However, existing automatic methods for I/O performance bottleneck diagnosis have one major issue: the granularity of the analysis is at the platform or group level and the diagnosis results cannot be applied to the individual application. To address this issue, we designed and developed a method named "Artificial Intelligence for I/O"(AIIO), which uses AI and its interpretation technology to diagnose I/O performance bottlenecks at the job level automatically. By considering the sparsity of I/O log files, employing multiple AI models for performance prediction, merging diagnosis results across multiple models, and generalizing its performance prediction and diagnosis functions, AIIO can accurately and robustly identify the bottleneck of an even unseen application. Experimental results show that real and unseen applications can use the diagnosis results from AIIO to improve their I/O performance by at most 146 times.

View Conference

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

DOE Contract Number:: AC02-05CH11231

OSTI ID:: 2228856

Journal Information:: Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing, Conference: HPDC '23: 32. International Symposium on High-Performance Parallel and Distributed Computing, Orlando, FL (United States), 16-23 Jun 2023

Country of Publication:: United States

Language:: English

References (26)

Characterizing Machine Learning I/O Workloads on Leadership Scale HPC Systems Paul, Arnab K.; Karimi, Ahmad Maroof; Wang, Feiyi 2021 29th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) https://doi.org/10.1109/MASCOTS53633.2021.9614303	conference	November 2021
"Why Should I Trust You?": Explaining the Predictions of Any Classifier Ribeiro, Marco Tulio; Singh, Sameer; Guestrin, Carlos Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '16 https://doi.org/10.1145/2939672.2939778	conference	January 2016
Access Patterns and Performance Behaviors of Multi-layer Supercomputer I/O Subsystems under Production Load Bez, Jean Luca; Karimi, Ahmad Maroof; Paul, Arnab K. Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing https://doi.org/10.1145/3502181.3531461	conference	June 2022
Statistical scalability analysis of communication operations in distributed applications Vetter, Jeffrey S.; McCracken, Michael O. ACM SIGPLAN Notices, Vol. 36, Issue 7 https://doi.org/10.1145/568014.379590	journal	June 2001
Interpreting Write Performance of Supercomputer I/O Systems with Regression Models Xie, Bing; Tan, Zilong; Carns, Philip 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS49936.2021.00064	conference	May 2021
Stack Trace Analysis for Large Scale Debugging Arnold, Dorian C.; Ahn, Dong H.; de Supinski, Bronis R. 2007 IEEE International Parallel and Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2007.370254	conference	March 2007
Artificial intelligence for throughput bottleneck analysis – State-of-the-art and future directions Subramaniyan, Mukund; Skoogh, Anders; Bokrantz, Jon Journal of Manufacturing Systems, Vol. 60 https://doi.org/10.1016/j.jmsy.2021.07.021	journal	July 2021
An In-Depth I/O Pattern Analysis in HPC Systems Bang, Jiwoo; Kim, Chungyong; Wu, Kesheng 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC) https://doi.org/10.1109/HiPC53243.2021.00056	conference	December 2021
Parallel I/O, analysis, and visualization of a trillion particle simulation Byna, Surendra; Chou, Jerry; Rubel, Oliver 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.92	conference	November 2012
Systematically inferring I/O performance variability by examining repetitive job behavior Costa, Emily; Patel, Tirthak; Schwaller, Benjamin Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3458817.3476186	conference	November 2021
Six degrees of scientific data: reading patterns for extreme scale science IO Lofstead, Jay; Polte, Milo; Gibson, Garth Proceedings of the 20th international symposium on High performance distributed computing - HPDC '11 https://doi.org/10.1145/1996130.1996139	conference	January 2011
Early experiences in application level I/O tracing on blue gene systems Seelam, Seetharami 2008 IEEE International Symposium on Parallel and Distributed Processing https://doi.org/10.1109/IPDPS.2008.4536550	conference	April 2008
IOMiner: Large-Scale Analytics Framework for Gaining Knowledge from I/O Logs Wang, Teng; Snyder, Shane; Lockwood, Glenn 2018 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2018.00062	conference	September 2018
Gauge: An Interactive Data-Driven Visualization Tool for HPC Application I/O Performance Analysis del Rosario, Eliakin; Currier, Mikaela; Isakov, Mihailo 2020 IEEE/ACM Fifth International Parallel Data Systems Workshop (PDSW) https://doi.org/10.1109/PDSW51947.2020.00008	conference	November 2020
Drishti: Guiding End-Users in the I/O Optimization Journey Bez, Jean Luca; Ather, Hammad; Byna, Suren 2022 IEEE/ACM International Parallel Data Systems Workshop (PDSW) https://doi.org/10.1109/PDSW56643.2022.00006	conference	November 2022
Applying Machine Learning to Understand Write Performance of Large-scale Parallel Filesystems Xie, Bing; Tan, Zilong; Carns, Philip 2019 IEEE/ACM Fourth International Parallel Data Systems Workshop (PDSW) https://doi.org/10.1109/PDSW49588.2019.00008	conference	November 2019
XGBoost: A Scalable Tree Boosting System Chen, Tianqi; Guestrin, Carlos Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '16 https://doi.org/10.1145/2939672.2939785	conference	January 2016
Characterizing the I/O behavior of scientific applications on the Cray XT Roth, Philip C. Proceedings of the 2nd international workshop on Petascale data storage: held in conjunction with Supercomputing '07 https://doi.org/10.1145/1374596.1374609	conference	November 2007
A New File-Specific Stripe Size Selection Method for Highly Concurrent Data Access Dong, Bin; Li, Xiuqiao; Xiao, Limin 2012 ACM/IEEE 13th International Conference on Grid Computing https://doi.org/10.1109/Grid.2012.11	conference	September 2012
pNFS, POSIX, and MPI-IO Hildebrand, Dean; Nisar, Arifa; Haskin, Roger Proceedings of the 4th Annual Workshop on Petascale Data Storage https://doi.org/10.1145/1713072.1713082	conference	November 2009
I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis Bez, Jean Luca; Tang, Houjun; Xie, Bing 2021 IEEE/ACM Sixth International Parallel Data Systems Workshop (PDSW) https://doi.org/10.1109/PDSW54622.2021.00008	conference	November 2021
An analytic performance model of disk arrays Lee, Edward K.; Katz, Randy H. Proceedings of the 1993 ACM SIGMETRICS conference on Measurement and modeling of computer systems https://doi.org/10.1145/166955.166994	conference	June 1993
Data partitioning and load balancing in parallel disk systems Scheuermann, Peter; Weikum, Gerhard; Zabback, Peter The VLDB Journal The International Journal on Very Large Data Bases, Vol. 7, Issue 1 https://doi.org/10.1007/s007780050053	journal	February 1998
ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems Byna, Suren; Breitenfeld, M. Scot; Dong, Bin Journal of Computer Science and Technology, Vol. 35, Issue 1 https://doi.org/10.1007/s11390-020-9822-9	journal	January 2020
machine. Friedman, Jerome H. The Annals of Statistics, Vol. 29, Issue 5 https://doi.org/10.1214/aos/1013203451	journal	October 2001
File-access characteristics of parallel scientific workloads Nieuwejaar, N.; Kotz, D.; Purakayastha, A. IEEE Transactions on Parallel and Distributed Systems, Vol. 7, Issue 10 https://doi.org/10.1109/71.539739	journal	January 1996

Similar Records

Application of artificial intelligence in electronic circuit diagnosis

Thesis/Dissertation · 1989 · OSTI ID:5896740

Sabour, F

A Zoom-in Analysis of I/O Logs to Detect Root Causes of I/O Performance Bottlenecks

Conference · 2019 · OSTI ID:1562878

Wang, Teng; Byna, Suren; Lockwood, Glenn K.; +4 more

IEA/AIE-89; Proceedings of the Second International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems, University of Tennessee, Tullahoma, June 6-9, 1989. Volumes 1 2

Conference · 1989 · OSTI ID:6584908

Ali, M

AIIO: Using Artificial Intelligence for Job-Level and Automatic I/O Performance Bottleneck Diagnosis

Citation Formats

References (26)

Similar Records

Related Subjects