Enabling Agile Analysis of I/O Performance Data with PyDarshan

Luettgau, Jakob; Snyder, Shane; Reddy, Tyler; Awtrey, Nikolaus; Harms, Kevin; Bez, Jean Luca; Wang, Rui; Latham, Rob; Carns, Philip

doi:10.1145/3624062.3624207

Enabling Agile Analysis of I/O Performance Data with PyDarshan

Conference · 12 November 2023

DOI:https://doi.org/10.1145/3624062.3624207· OSTI ID:2228850

Luettgau, Jakob; Snyder, Shane; Reddy, Tyler; Awtrey, Nikolaus; Harms, Kevin; Bez, Jean Luca; Wang, Rui; Latham, Rob; Carns, Philip

View Conference

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

DOE Contract Number:: AC02-05CH11231

OSTI ID:: 2228850

Resource Relation:: Conference: SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, Denver, CO (United States), 12-17 Nov 2023

Country of Publication:: United States

Language:: English

References (20)

root-project/root: v6.18/02 No authors listed https://doi.org/10.5281/zenodo.3895860		August 2019
Recorder 2.0: Efficient Parallel I/O Tracing and Analysis No authors listed 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) https://doi.org/10.1109/IPDPSW50202.2020.00176		May 2020
Score-P: A Joint Performance Measurement Run-Time Infrastructure for Periscope, Scalasca, TAU, and Vampir No authors listed Tools for High Performance Computing 2011 https://doi.org/10.1007/978-3-642-31476-6_7		January 2012
A characterization of workflow management systems for extreme-scale applications No authors listed Future Generation Computer Systems https://doi.org/10.1016/j.future.2017.02.026		October 2017
Array programming with NumPy No authors listed Nature https://doi.org/10.1038/s41586-020-2649-2		September 2020
Drishti: Guiding End-Users in the I/O Optimization Journey No authors listed 2022 IEEE/ACM International Parallel Data Systems Workshop (PDSW) https://doi.org/10.1109/PDSW56643.2022.00006		November 2022
Transparent Asynchronous Parallel I/O Using Background Threads No authors listed IEEE Transactions on Parallel and Distributed Systems https://doi.org/10.1109/TPDS.2021.3090322		April 2022
The Tau Parallel Performance System No authors listed The International Journal of High Performance Computing Applications https://doi.org/10.1177/1094342006064482		May 2006
Data Structures for Statistical Computing in Python No authors listed Proceedings of the Python in Science Conference https://doi.org/10.25080/Majora-92bf1922-00a		January 2010
I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis No authors listed 2021 IEEE/ACM Sixth International Parallel Data Systems Workshop (PDSW) https://doi.org/10.1109/PDSW54622.2021.00008		November 2021
HPCTOOLKIT: tools for performance analysis of optimized parallel programs No authors listed Concurrency and Computation: Practice and Experience https://doi.org/10.1002/cpe.1553		January 2009
pandas-dev/pandas: Pandas No authors listed https://doi.org/10.5281/zenodo.3509134		January 2024
Visualizing a Moving Target: A Design Study on Task Parallel Programs in the Presence of Evolving Data and Concerns No authors listed IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2019.2934285		January 2019
Illuminating the I/O Optimization Path of Scientific Applications No authors listed Lecture Notes in Computer Science https://doi.org/10.1007/978-3-031-32041-5_2		January 2023
Characterization of scientific workflows No authors listed 2008 Third Workshop on Workflows in Support of Large-Scale Science (WORKS 2008) https://doi.org/10.1109/WORKS.2008.4723958		November 2008
AMReX: Block-structured adaptive mesh refinement for multiphysics applications No authors listed The International Journal of High Performance Computing Applications https://doi.org/10.1177/10943420211022811		June 2021
xarray: N-D labeled Arrays and Datasets in Python No authors listed Journal of Open Research Software https://doi.org/10.5334/jors.148		April 2017
Dask: Parallel Computation with Blocked algorithms and Task Scheduling No authors listed Proceedings of the Python in Science Conference https://doi.org/10.25080/Majora-7b98e3ed-013		January 2015
Hatchet No authors listed Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3295500.3356219		November 2019
Caliper: Performance Introspection for HPC Software Stacks No authors listed SC16: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2016.46		November 2016

Similar Records

TOKIO on ClusterStor: Connecting Standard Tools to Enable Holistic I/O Performance Analysis

Conference · 2018 · OSTI ID:1632125

Lockwood, Glenn K.; Wright, Nicholas J.; Snyder, Shane; +3 more

UMAMI: A Recipe for Generating Meaningful Metrics through Holistic I/O Performance Analysis

Conference · 2017 · OSTI ID:1426744

Lockwood, Glenn K.; Yoo, Wucherl; Byna, Suren; +5 more

A Zoom-in Analysis of I/O Logs to Detect Root Causes of I/O Performance Bottlenecks

Conference · 2019 · OSTI ID:1562878

Wang, Teng; Byna, Suren; Lockwood, Glenn K.; +4 more

Enabling Agile Analysis of I/O Performance Data with PyDarshan

Citation Formats

References (20)

Similar Records

Related Subjects