Performance characterization of scientific workflows for the optimal use of Burst Buffers

Journal Article · Fri Dec 01 00:00:00 EST 2017 · Future Generations Computer Systems

DOI:https://doi.org/10.1016/j.future.2017.12.022· OSTI ID:1544343

Daley, C. S. ^[1]; Ghoshal, D. ^[1]; Lockwood, G. K. ^[1]; Dosanjh, S. ^[1]; Ramakrishnan, L. ^[1]; Wright, N. J. ^[1]

---

1. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Scientific discoveries are increasingly dependent upon the analysis of large volumes of data from observations and simulations of complex phenomena. Scientists compose the complex analyses as workflows and execute them on large-scale HPC systems. The workflow structures are in contrast with monolithic single simulations that have often been the primary use case on HPC systems. Simultaneously, new storage paradigms such as Burst Buffers are becoming available on HPC platforms. In this paper, we analyze the performance characteristics of a Burst Buffer and two representative scientific workflows with the aim of optimizing the usage of a Burst Buffer, extending our previous analyses (Daley et al., 2016). Our key contributions are (a) developing a performance analysis methodology pertinent to Burst Buffers, (b) improving the use of a Burst Buffer in workflows with bandwidth-sensitive and metadata-sensitive I/O workloads, (c) highlighting the key data management challenges when incorporating a Burst Buffer in the studied scientific workflows.

• Research Organization: Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
• DOE Contract Number: AC02-05CH11231
• OSTI ID: 1544343
• Journal Information: Future Generations Computer Systems, Journal Name: Future Generations Computer Systems; ISSN 0167-739X
• Publisher: Elsevier
• Country of Publication: United States
• Language: English