skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: An Ephemeral Burst-Buffer File System for Scientific Applications

Abstract

BurstFS is a distributed file system for node-local burst buffers on high performance computing systems. BurstFS presents a shared file system space across the burst buffers so that applications that use shared files can access the highly-scalable burst buffers without changing their applications.

Authors:
 [1];  [2];  [1];  [2];  [2]
  1. Florida State University
  2. LLNL
Publication Date:
Research Org.:
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA)
Contributing Org.:
LLNL
OSTI Identifier:
1351607
Report Number(s):
BurstFS; 005217WKSTN00
LLNL-CODE-728877
DOE Contract Number:
AC52-07NA27344
Resource Type:
Software
Software Revision:
00
Software Package Number:
005217
Software CPU:
WKSTN
Source Code Available:
Yes
Country of Publication:
United States

Citation Formats

Wang, Teng, Moody, Adam, Yu, Weikuan, Sato, Kento, and Mohror, Kathryn. An Ephemeral Burst-Buffer File System for Scientific Applications. Computer software. Vers. 00. USDOE National Nuclear Security Administration (NNSA). 11 Apr. 2017. Web.
Wang, Teng, Moody, Adam, Yu, Weikuan, Sato, Kento, & Mohror, Kathryn. (2017, April 11). An Ephemeral Burst-Buffer File System for Scientific Applications (Version 00) [Computer software].
Wang, Teng, Moody, Adam, Yu, Weikuan, Sato, Kento, and Mohror, Kathryn. An Ephemeral Burst-Buffer File System for Scientific Applications. Computer software. Version 00. April 11, 2017.
@misc{osti_1351607,
title = {An Ephemeral Burst-Buffer File System for Scientific Applications, Version 00},
author = {Wang, Teng and Moody, Adam and Yu, Weikuan and Sato, Kento and Mohror, Kathryn},
abstractNote = {BurstFS is a distributed file system for node-local burst buffers on high performance computing systems. BurstFS presents a shared file system space across the burst buffers so that applications that use shared files can access the highly-scalable burst buffers without changing their applications.},
doi = {},
year = {Tue Apr 11 00:00:00 EDT 2017},
month = {Tue Apr 11 00:00:00 EDT 2017},
note =
}

Software:
To order this software, request consultation services, or receive further information, please fill out the following request.

Save / Share:
  • The growth of computing power on large-scale sys- tems requires commensurate high-bandwidth I/O system. Many parallel file systems are designed to provide fast sustainable I/O in response to applications soaring requirements. To meet this need, a novel system is imperative to temporarily buffer the bursty I/O and gradually flush datasets to long-term parallel file systems. In this paper, we introduce the design of BurstMem, a high- performance burst buffer system. BurstMem provides a storage framework with efficient storage and communication manage- ment strategies. Our experiments demonstrate that BurstMem is able to speed up the I/O performance of scientific applications bymore » up to 8.5 on leadership computer systems.« less
  • Parallel scientific applications require high-performance I/O support from underlying file systems. A comprehensive understanding of the expected workload is therefore essential for the design of high-performance parallel file systems. We re-examine the workload characteristics in parallel computing environments in the light of recent technology advances and new applications. We analyze application traces from a cluster with hundreds of nodes. On average, each application has only one or two typical request sizes. Large requests from several hundred kilobytes to several megabytes are very common. Although in some applications, small requests account for more than 90% of all requests, almost all ofmore » the I/O data are transferred by large requests. All of these applications show bursty access patterns. More than 65% of write requests have inter-arrival times within one millisecond in most applications. By running the same benchmark on different file models, we also find that the write throughput of using an individual output file for each node exceeds that of using a shared file for all nodes by a factor of 5. This indicates that current file systems are not well optimized for file sharing.« less
  • We present some theoretical and experimental results of animportant caching problem that arises frequently in data intensivescientific applications. In such applications, jobs need to processseveral files simultaneously, i.e., a job can only be serviced if all itsneeded files are present in the disk cache. The set of files requested bya job is called a file-bundle. This requirement introduces the need forcache replacement algorithms based on file-bundles rather then individualfiles. We show that traditional caching algorithms such Least RecentlyUsed (LRU), and GreedyDual-Size (GDS), are not optimal in this case sincethey are not sensitive to file-bundles and may hold in the cachenon-relevantmore » combinations of files. In this paper we propose and analyzea new cache replacement algorithm specifically adapted to deal withfile-bundles. We tested the new algorithm using a disk cache simulationmodel under a wide range of parameters such as file requestdistributions, relative cache size, file size distribution,and queuesize. In all these tests, the results show significant improvement overtraditional caching algorithms such as GDS.« less

To initiate an order for this software, request consultation services, or receive further information, fill out the request form below. You may also reach us by email at: .

OSTI staff will begin to process an order for scientific and technical software once the payment and signed site license agreement are received. If the forms are not in order, OSTI will contact you. No further action will be taken until all required information and/or payment is received. Orders are usually processed within three to five business days.

Software Request

(required)
(required)
(required)
(required)
(required)
(required)
(required)
(required)