Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Improving the Effectiveness of Burst Buffers for Big Data Processing in HPC Systems with Eley

Journal Article · · Future Generations Computer Systems
 [1];  [2];  [3]
  1. Univ. Rennes, Rennes (France); Argonne National Lab. (ANL), Lemont, IL (United States)
  2. Shenzhen Univ., Shenzhen (China)
  3. Inria, Nantes (France)
+ Show Author Affiliations
Here, burst buffer is an effective solution for reducing the data transfer time and the I/O interference in HPC systems. Extending Burst Buffers (BBs) to handle Big Data applications is challenging because BBs must account for the large data inputs of Big Data applications and the Quality-of-Service (QoS) of HPC applications which are considered as first-class citizens in HPC systems. Existing BBs focus on only intermediate data of Big Data applications and incur a high performance degradation of both Big Data and HPC applications. We present Eley, a burst buffer solution that helps to accelerate the performance of Big Data applications while guaranteeing the QoS of HPC applications. To achieve this goal, Eley embraces interference-aware prefetching technique that makes reading data input faster while introducing low interference for HPC applications. Evaluations using a wide range of Big Data and HPC applications demonstrate that Eley improves the performance of Big Data applications by up to 30% compared to existing BBs while maintaining the QoS of HPC applications.
Research Organization:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Organization:
National Science Foundation (NSF); Shenzhen University; USDOE; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
Grant/Contract Number:
AC02-06CH11357
OSTI ID:
1467628
Alternate ID(s):
OSTI ID: 1583001
Journal Information:
Future Generations Computer Systems, Journal Name: Future Generations Computer Systems Journal Issue: C Vol. 86; ISSN 0167-739X
Publisher:
ElsevierCopyright Statement
Country of Publication:
United States
Language:
English

References (7)

Enabling fast failure recovery in shared Hadoop clusters: Towards failure-aware scheduling journal September 2017
I/O-Aware Batch Scheduling for Petascale Computing Systems conference September 2015
CALCioM: Mitigating I/O Interference in HPC Systems through Cross-Application Coordination
  • Dorier, Matthieu; Antoniu, Gabriel; Ross, Rob
  • 2014 IEEE International Parallel & Distributed Processing Symposium (IPDPS), 2014 IEEE 28th International Parallel and Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2014.27
conference May 2014
On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems conference May 2016
MapReduce: simplified data processing on large clusters journal January 2008
Performance Modelling and Analysis of Software-Defined Networking under Bursty Multimedia Traffic
  • Miao, Wang; Min, Geyong; Wu, Yulei
  • ACM Transactions on Multimedia Computing, Communications, and Applications, Vol. 12, Issue 5s https://doi.org/10.1145/2983637
journal December 2016
Damaris: Addressing Performance Variability in Data Management for Post-Petascale Simulations journal October 2016

Cited By (1)

Approaches of enhancing interoperations among high performance computing and big data analytics via augmentation journal August 2019

Similar Records

Evaluating Burst Buffer Placement in HPC Systems
Conference · Sun Sep 01 00:00:00 EDT 2019 · OSTI ID:1566958
BBOS: Efficient HPC Storage Management via Burst Buffer Over-Subscription
Conference · Fri May 01 00:00:00 EDT 2020 · OSTI ID:1827659

Related Subjects

97 MATHEMATICS AND COMPUTING
Big Data
Burst Buffers
HPC
Interference
MapReduce
Parallel File Systems
Prefetch