Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting

Clemente-Castello, Francisco J.; Nicolae, Bogdan; Mayo, Rafael; Fernandez, Juan Carlos

doi:10.1109/TPDS.2018.2802932

Title: Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting

Abstract

Hybrid cloud bursting (i.e., leasing temporary off-premise cloud resources to boost the overall capacity during peak utilization) can be a cost-effective way to deal with the increasing complexity of big data analytics, especially for iterative applications. However, the low throughput, high latency network link between the on-premise and off-premise resources (“weak link”) makes maintaining scalability difficult. While several data locality techniques have been designed for big data bursting on hybrid clouds, their effectiveness is difficult to estimate in advance. Yet such estimations are critical, because they help users decide whether the extra pay-as-you-go cost incurred by using the off-premise resources justifies the runtime speed-up. To this end, the current paper presents a performance model and methodology to estimate the runtime of iterative MapReduce applications in a hybrid cloud-bursting scenario. The paper focuses on the overhead incurred by the weak link at fine granularity, for both the map and the reduce phases. This approach enables high estimation accuracy, as demonstrated by extensive experiments at scale using a mix of real-world iterative MapReduce applications from standard big data benchmarking suites that cover a broad spectrum of data patterns. As a result, not only are the produced estimations accurate in absolute terms comparedmore »« less

Authors:

^[1];

^[2];

^[1];

^[1]

Univ. Jaume I, Castellon (Spain)
Argonne National Lab. (ANL), Argonne, IL (United States)

Publication Date:: Tue Feb 06 00:00:00 EST 2018

Research Org.:: Argonne National Lab. (ANL), Argonne, IL (United States)

Sponsoring Org.:: National Science Foundation (NSF); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:: 1466407

Grant/Contract Number:: AC02-06CH11357

Resource Type:: Accepted Manuscript

Journal Name:: IEEE Transactions on Parallel and Distributed Systems

Additional Journal Information:: Journal Volume: 29; Journal Issue: 8; Journal ID: ISSN 1045-9219

Publisher:: IEEE

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Big Data Analytics; Hybrid Cloud; Iterative Applications; MapReduce; Performance Prediction; Runtime Estimation

Citation Formats


                    Clemente-Castello, Francisco J., Nicolae, Bogdan, Mayo, Rafael, and Fernandez, Juan Carlos. Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting.  United States: N. p., 2018. 
Web.  doi:10.1109/TPDS.2018.2802932.

Copy to clipboard


                    Clemente-Castello, Francisco J., Nicolae, Bogdan, Mayo, Rafael, & Fernandez, Juan Carlos. Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting.  United States.  https://doi.org/10.1109/TPDS.2018.2802932

Copy to clipboard


                    Clemente-Castello, Francisco J., Nicolae, Bogdan, Mayo, Rafael, and Fernandez, Juan Carlos. Tue .  
"Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting".  United States.  https://doi.org/10.1109/TPDS.2018.2802932.  https://www.osti.gov/servlets/purl/1466407.

Copy to clipboard


                    
@article{osti_1466407,

  title        = {Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting},

  author       = {Clemente-Castello, Francisco J. and Nicolae, Bogdan and Mayo, Rafael and Fernandez, Juan Carlos},

  abstractNote = {Hybrid cloud bursting (i.e., leasing temporary off-premise cloud resources to boost the overall capacity during peak utilization) can be a cost-effective way to deal with the increasing complexity of big data analytics, especially for iterative applications. However, the low throughput, high latency network link between the on-premise and off-premise resources (“weak link”) makes maintaining scalability difficult. While several data locality techniques have been designed for big data bursting on hybrid clouds, their effectiveness is difficult to estimate in advance. Yet such estimations are critical, because they help users decide whether the extra pay-as-you-go cost incurred by using the off-premise resources justifies the runtime speed-up. To this end, the current paper presents a performance model and methodology to estimate the runtime of iterative MapReduce applications in a hybrid cloud-bursting scenario. The paper focuses on the overhead incurred by the weak link at fine granularity, for both the map and the reduce phases. This approach enables high estimation accuracy, as demonstrated by extensive experiments at scale using a mix of real-world iterative MapReduce applications from standard big data benchmarking suites that cover a broad spectrum of data patterns. As a result, not only are the produced estimations accurate in absolute terms compared with experimental results, but they are also up to an order of magnitude more accurate than applying state-of-art estimation approaches originally designed for single-site MapReduce deployments.},

  doi          = {10.1109/TPDS.2018.2802932},

  journal      = {IEEE Transactions on Parallel and Distributed Systems},

  number       = 8,

  volume       = 29,

  place        = {United States},

  year         = {Tue Feb 06 00:00:00 EST 2018},

  month        = {Tue Feb 06 00:00:00 EST 2018}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1109/TPDS.2018.2802932

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 9 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referencing / citing this record:

A survey and taxonomy on workload scheduling and resource provisioning in hybrid clouds
journal, February 2020

Wang, Bo; Wang, Changhai; Song, Ying
Cluster Computing, Vol. 23, Issue 4
DOI: 10.1007/s10586-020-03048-8

Similar Records in DOE PAGES and OSTI.GOV collections:

Fault-Tolerant and Elastic Streaming MapReduce with Decentralized Coordination

Conference Kumbhare, Alok ; Frincu, Marc ; Simmhan, Yogesh ; ...

The MapReduce programming model, due to its simplicity and scalability, has become an essential tool for processing large data volumes in distributed environments. Recent Stream Processing Systems (SPS) extend this model to provide low-latency analysis of high-velocity continuous data streams. However, integrating MapReduce with streaming poses challenges: first, the runtime variations in data characteristics such as data-rates and key-distribution cause resource overload, that inturn leads to fluctuations in the Quality of the Service (QoS); and second, the stateful reducers, whose state depends on the complete tuple history, necessitates efficient fault-recovery mechanisms to maintain the desired QoS in the presence ofmore »« less
https://doi.org/10.1109/ICDCS.2015.41

Full Text Available
Center for Technology for Advanced Scientific Componet Software (TASCS)

Technical Report Govindaraju, Madhusudhan

Advanced Scientific Computing Research Computer Science FY 2010Report Center for Technology for Advanced Scientific Component Software: Distributed CCA State University of New York, Binghamton, NY, 13902 Summary The overall objective of Binghamton's involvement is to work on enhancements of the CCA environment, motivated by the applications and research initiatives discussed in the proposal. This year we are working on re-focusing our design and development efforts to develop proof-of-concept implementations that have the potential to significantly impact scientific components. We worked on developing parallel implementations for non-hydrostatic code and worked on a model coupling interface for biogeochemical computations coded in MATLAB.more »« less
https://doi.org/10.2172/1092881

Full Text Available
A case study of tuning MapReduce for efficient Bioinformatics in the cloud

Journal Article Shi, Lizhen ; Wang, Zhong ; Yu, Weikuan ; ... - Parallel Computing

The combination of the Hadoop MapReduce programming model and cloud computing allows biological scientists to analyze next-generation sequencing (NGS) data in a timely and cost-effective manner. Cloud computing platforms remove the burden of IT facility procurement and management from end users and provide ease of access to Hadoop clusters. However, biological scientists are still expected to choose appropriate Hadoop parameters for running their jobs. More importantly, the available Hadoop tuning guidelines are either obsolete or too general to capture the particular characteristics of bioinformatics applications. In this paper, we aim to minimize the cloud computing cost spent on bioinformatics datamore »« less
Cited by 9
https://doi.org/10.1016/j.parco.2016.10.002

Full Text Available
MapReduce SVM Game

Journal Article Vineyard, Craig M. ; Verzi, Stephen J. ; James, Conrad D. ; ... - Procedia Computer Science

Despite technological advances making computing devices faster, smaller, and more prevalent in today's age, data generation and collection has outpaced data processing capabilities. Simply having more compute platforms does not provide a means of addressing challenging problems in the big data era. Rather, alternative processing approaches are needed and the application of machine learning to big data is hugely important. The MapReduce programming paradigm is an alternative to conventional supercomputing approaches, and requires less stringent data passing constrained problem decompositions. Rather, MapReduce relies upon defining a means of partitioning the desired problem so that subsets may be computed independently andmore »« less
Cited by 1
https://doi.org/10.1016/j.procs.2015.07.307

Full Text Available
PLAStiCC: Predictive Look-Ahead Scheduling for Continuous dataflows on Clouds

Conference Kumbhare, Alok ; Simmhan, Yogesh ; Prasanna, Viktor K.

Scalable stream processing and continuous dataflow systems are gaining traction with the rise of big data due to the need for processing high velocity data in near real time. Unlike batch processing systems such as MapReduce and workflows, static scheduling strategies fall short for continuous dataflows due to the variations in the input data rates and the need for sustained throughput. The elastic resource provisioning of cloud infrastructure is valuable to meet the changing resource needs of such continuous applications. However, multi-tenant cloud resources introduce yet another dimension of performance variability that impacts the application’s throughput. In this paper wemore »« less
https://doi.org/10.1109/CCGrid.2014.60

Full Text Available

Similar Records

Title: Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting

Abstract

Citation Formats

A survey and taxonomy on workload scheduling and resource provisioning in hybrid clouds journal, February 2020

A survey and taxonomy on workload scheduling and resource provisioning in hybrid clouds
journal, February 2020