Darshan for HEP applications
Modern HEP workflows must manage increasingly large and complex data collections. HPC facilities may be employed to help meet these workflows' growing data processing needs. However, a better understanding of the I/O patterns and underlying bottlenecks of these workflows is necessary to meet the performance expectations of HPC systems. Darshan is a lightweight I/O characterization tool that captures concise views of HPC application I/O behavior. It intercepts application I/O calls at runtime, records file access statistics for each process, and generates log files detailing application I/O access patterns. Typical HEP workflows include event generation, detector simulation, event reconstruction, and subsequent analysis stages. A study of the I/O behavior of the ATLAS simulation and filtering stage, and the CMS simulation workflow using Darshan is presented, including insights into the I/O operations and data access size.
- Research Organization:
- Argonne National Laboratory (ANL)
- Sponsoring Organization:
- USDOE Office of Science - Office of High Energy Physics; USDOE Exascale Computing Project (ECP); USDOE Office of Science - Office of Advanced Scientific Computing Research (ASCR) - Scientific Discovery through Advanced Computing (SciDAC)
- DOE Contract Number:
- AC02-06CH11357
- OSTI ID:
- 2447365
- Country of Publication:
- United States
- Language:
- English
24/7 Characterization of petascale I/O workloads
|
conference | August 2009 |
Experience with the CMS event data model
|
journal | April 2010 |
Similar Records
Performance Evaluation of Darshan 3.0.0 on the Cray XC30
DXT: Darshan eXtended Tracing