Performance characterization of scientific workflows for the optimal use of Burst Buffers

Daley, C. S.; Ghoshal, D.; Lockwood, G. K.; Dosanjh, S.; Ramakrishnan, L.; Wright, N. J.

doi:10.1016/j.future.2017.12.022

Performance characterization of scientific workflows for the optimal use of Burst Buffers

Journal Article · Thu Dec 28 00:00:00 EST 2017 · Future Generations Computer Systems

DOI:https://doi.org/10.1016/j.future.2017.12.022· OSTI ID:1573795

Daley, C. S. ^[1]; Ghoshal, D. ^[1]; Lockwood, G. K. ^[1]; Dosanjh, S. ^[1]; Ramakrishnan, L. ^[1]; Wright, N. J. ^[1]

Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Scientific discoveries are significantly rely on the analysis of large volumes of data from observations and simulations of complex phenomena. Scientists compose the complex analyses as workflows and execute them on large-scale HPC systems. The workflow structures are in contrast with monolithic single simulations that have often been the primary use case on HPC systems. At the same time, new storage paradigms such as Burst Buffers are becoming available on HPC platforms. In this paper, we analyze the performance characteristics of a Burst Buffer and two representative scientific workflows with the aim of optimizing the usage of a Burst Buffer, extending our previous analyses (Daley et al., 2016). Our key contributions are (a) developing a performance analysis methodology pertinent to Burst Buffers, (b) improving the use of a Burst Buffer in workflows with bandwidth-sensitive and metadata-sensitive I/O workloads, (c) highlighting the key data management challenges when incorporating a Burst Buffer in the studied scientific workflows.

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1573795

Alternate ID(s):: OSTI ID: 1702552
OSTI ID: 1544343

Journal Information:: Future Generations Computer Systems, Journal Name: Future Generations Computer Systems Vol. 110; ISSN 0167-739X

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (7)

CAMP: Community Access MODIS Pipeline Hendrix, Valerie; Ramakrishnan, Lavanya; Ryu, Youngryel Future Generation Computer Systems, Vol. 36 https://doi.org/10.1016/j.future.2013.09.023	journal	July 2014
GPAW - massively parallel electronic structure calculations with Python-based software Enkovaara, Jussi; Romero, Nichols A.; Shende, Sameer Procedia Computer Science, Vol. 4 https://doi.org/10.1016/j.procs.2011.04.003	journal	January 2011
MODIS land data storage, gridding, and compositing methodology: Level 2 grid Wolfe, R. E.; Roy, D. P.; Vermote, E. IEEE Transactions on Geoscience and Remote Sensing, Vol. 36, Issue 4 https://doi.org/10.1109/36.701082	journal	July 1998
24/7 Characterization of petascale I/O workloads Carns, Philip; Latham, Robert; Ross, Robert 2009 IEEE International Conference on Cluster Computing and Workshops https://doi.org/10.1109/CLUSTR.2009.5289150	conference	August 2009
On Timely Staging of HPC Job Input Data Monti, Henry M.; Butt, Ali R.; Vazhkudai, Sudharshan S. IEEE Transactions on Parallel and Distributed Systems, Vol. 24, Issue 9 https://doi.org/10.1109/TPDS.2012.279	journal	September 2013
Montage: a grid-enabled engine for delivering custom science-grade mosaics on demand Berriman, G. B.; Deelman, Ewa; Good, John C. SPIE Astronomical Telescopes + Instrumentation, SPIE Proceedings https://doi.org/10.1117/12.550551	conference	September 2004
Group-based variant calling leveraging next-generation supercomputing for large-scale whole-genome sequencing studies Standish, Kristopher A.; Carland, Tristan M.; Lockwood, Glenn K. BMC Bioinformatics, Vol. 16, Issue 1 https://doi.org/10.1186/s12859-015-0736-4	journal	September 2015

Cited By (1)

A slice-based decentralized NFV framework for an end-to-end QoS-based dynamic resource allocation Djouela Kamgang, Inès Raïssa; Zhioua, Ghayet El Mouna; Tabbane, Nabil Journal of Ambient Intelligence and Humanized Computing, Vol. 11, Issue 11 https://doi.org/10.1007/s12652-020-01709-5	journal	January 2020

Similar Records

Performance characterization of scientific workflows for the optimal use of Burst Buffers

Journal Article · Thu Nov 30 23:00:00 EST 2017 · Future Generations Computer Systems · OSTI ID:1544343

Final Report for File System Support for Burst Buffers on HPC Systems

Technical Report · Sun Nov 26 23:00:00 EST 2017 · OSTI ID:1411671

Measuring the impact of burst buffers on data-intensive scientific workflows

Journal Article · Sun Jun 16 20:00:00 EDT 2019 · Future Generations Computer Systems · OSTI ID:1603369

Related Subjects

97 MATHEMATICS AND COMPUTING
Burst Buffer
DataWarp
HPC
Workflow

Performance characterization of scientific workflows for the optimal use of Burst Buffers

Citation Formats

References (7)

Cited By (1)

Similar Records

Related Subjects