Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Measuring the impact of burst buffers on data-intensive scientific workflows

Journal Article · · Future Generations Computer Systems
 [1];  [2];  [2];  [2];  [2]
  1. Univ. of Southern California, Los Angeles, CA (United States); University of Southern California
  2. Univ. of Southern California, Los Angeles, CA (United States)
+ Show Author Affiliations
Science applications frequently produce and consume large volumes of data, but delivering this data to and from compute resources can be concerning, as parallel file system performance is not keeping up with compute and memory performance. To alter this I/O bottleneck, some systems have deployed burst buffers, but their impact on performance for real-world scientific workflow applications is still not clear. In this paper, we examine the impact of burst buffers through the remote-shared, allocatable burst buffers on the Cori system at NERSC. By running two data-intensive workflows, a high-throughput genome analysis workflow, and a subset of the SCEC high-performance CyberShake workflow, a production seismic hazard analysis workflow, we find that using burst buffers offers read and write improvements of an order of magnitude, and these improvements lead to increased job performance, and thereby increased overall workflow performance, even for long-running CPU-bound jobs.
Research Organization:
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Rensselaer Polytechnic Inst., Troy, NY (United States); Univ. of Southern California, Los Angeles, CA (United States)
Sponsoring Organization:
National Science Foundation (NSF); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
Grant/Contract Number:
AC02-05CH11231; SC0012636
OSTI ID:
1603369
Alternate ID(s):
OSTI ID: 1568907
Journal Information:
Future Generations Computer Systems, Journal Name: Future Generations Computer Systems Journal Issue: C Vol. 101; ISSN 0167-739X
Publisher:
ElsevierCopyright Statement
Country of Publication:
United States
Language:
English

Figures / Tables (16)

Figure 1(p. 3)figure Figure 1
Figure 2(p. 4)figure Figure 2
Figure 3(p. 5)figure Figure 3
Figure 4(p. 6)figure Figure 4
Figure 5(p. 6)figure Figure 5
Figure 6(p. 7)figure Figure 6
Figure 7(p. 8)figure Figure 7
Figure 8(p. 8)figure Figure 8
Figure 9(p. 9)figure Figure 9
Figure 10(p. 10)figure Figure 10
Figure 11(p. 10)figure Figure 11
Figure 12(p. 11)figure Figure 12
Figure 13(p. 11)figure Figure 13
Figure 14(p. 11)figure Figure 14
Figure 15(p. 11)figure Figure 15
Table 1(p. 12)table Table 1

Similar Records

Performance characterization of scientific workflows for the optimal use of Burst Buffers
Journal Article · Thu Nov 30 23:00:00 EST 2017 · Future Generations Computer Systems · OSTI ID:1544343
Performance characterization of scientific workflows for the optimal use of Burst Buffers
Journal Article · Wed Dec 27 19:00:00 EST 2017 · Future Generations Computer Systems · OSTI ID:1573795
Accelerating Science with the NERSC Burst Buffer Early User Program
Conference · Thu Dec 31 23:00:00 EST 2015 · OSTI ID:1393591

Related Subjects

42 ENGINEERING
Burst Buffers
High-Performance Computing
In Transit Processing
Scientific Workflows