Exploiting Redundancy and Application Scalability for Cost-Effective, Time-Constrained Execution of HPC Applications on Amazon EC2

Marathe, Aniruddha P.; Harris, Rachel A.; Lowenthal, David K.; de Supinski, Bronis R.; Rountree, Barry L.; Schulz, Martin

doi:10.1109/TPDS.2015.2508457

Title: Exploiting Redundancy and Application Scalability for Cost-Effective, Time-Constrained Execution of HPC Applications on Amazon EC2

Full Record
Other Related Research

Abstract

The use of clouds to execute high-performance computing (HPC) applications has greatly increased recently. Clouds provide several potential advantages over traditional supercomputers and in-house clusters. The most popular cloud is currently Amazon EC2, which provides fixed-cost and variable-cost, auction-based options. The auction market trades lower cost for potential interruptions that necessitate checkpointing; if the market price exceeds the bid price, a node is taken away from the user without warning. We explore techniques to maximize performance per dollar given a time constraint within which an application must complete. Specifically, we design and implement multiple techniques to reduce expected cost by exploiting redundancy in the EC2 auction market. We then design an adaptive algorithm that selects a scheduling algorithm and determines the bid price. We show that our adaptive algorithm executes programs up to seven times cheaper than using the on-demand market and up to 44 percent cheaper than the best non-redundant, auction-market algorithm. We extend our adaptive algorithm to incorporate application scalability characteristics for further cost savings. In conclusion, we show that the adaptive algorithm informed with scalability characteristics of applications achieves up to 56 percent cost savings compared to the expected cost for the base adaptive algorithm run atmore » a fixed, user-defined scale.« less

Authors:

Marathe, Aniruddha P. ^[1]; Harris, Rachel A. ^[2]; Lowenthal, David K. ^[3]; de Supinski, Bronis R. ^[1]; Rountree, Barry L. ^[1]; Schulz, Martin ^[1]

Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
Google Corp., Mountain View, CA (United States)
The Univ. of Arizona, Tucson, AZ (United States)

Publication Date:: Thu Dec 17 00:00:00 EST 2015

Research Org.:: Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 1399726

Report Number(s):: LLNL-JRNL-676899
Journal ID: ISSN 1045-9219; TRN: US1702972

Grant/Contract Number:: AC52-07NA27344

Resource Type:: Accepted Manuscript

Journal Name:: IEEE Transactions on Parallel and Distributed Systems

Additional Journal Information:: Journal Volume: 27; Journal Issue: 9; Journal ID: ISSN 1045-9219

Publisher:: IEEE

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; fault tolerance; reliability; cloud computing; resource provisioning; cost optimization

Citation Formats


                    Marathe, Aniruddha P., Harris, Rachel A., Lowenthal, David K., de Supinski, Bronis R., Rountree, Barry L., and Schulz, Martin. Exploiting Redundancy and Application Scalability for Cost-Effective, Time-Constrained Execution of HPC Applications on Amazon EC2.  United States: N. p., 2015. 
Web.  doi:10.1109/TPDS.2015.2508457.

Copy to clipboard


                    Marathe, Aniruddha P., Harris, Rachel A., Lowenthal, David K., de Supinski, Bronis R., Rountree, Barry L., & Schulz, Martin. Exploiting Redundancy and Application Scalability for Cost-Effective, Time-Constrained Execution of HPC Applications on Amazon EC2.  United States.  https://doi.org/10.1109/TPDS.2015.2508457

Copy to clipboard


                    Marathe, Aniruddha P., Harris, Rachel A., Lowenthal, David K., de Supinski, Bronis R., Rountree, Barry L., and Schulz, Martin. Thu .  
"Exploiting Redundancy and Application Scalability for Cost-Effective, Time-Constrained Execution of HPC Applications on Amazon EC2".  United States.  https://doi.org/10.1109/TPDS.2015.2508457.  https://www.osti.gov/servlets/purl/1399726.

Copy to clipboard


                    
@article{osti_1399726,

  title        = {Exploiting Redundancy and Application Scalability for Cost-Effective, Time-Constrained Execution of HPC Applications on Amazon EC2},

  author       = {Marathe, Aniruddha P. and Harris, Rachel A. and Lowenthal, David K. and de Supinski, Bronis R. and Rountree, Barry L. and Schulz, Martin},

  abstractNote = {The use of clouds to execute high-performance computing (HPC) applications has greatly increased recently. Clouds provide several potential advantages over traditional supercomputers and in-house clusters. The most popular cloud is currently Amazon EC2, which provides fixed-cost and variable-cost, auction-based options. The auction market trades lower cost for potential interruptions that necessitate checkpointing; if the market price exceeds the bid price, a node is taken away from the user without warning. We explore techniques to maximize performance per dollar given a time constraint within which an application must complete. Specifically, we design and implement multiple techniques to reduce expected cost by exploiting redundancy in the EC2 auction market. We then design an adaptive algorithm that selects a scheduling algorithm and determines the bid price. We show that our adaptive algorithm executes programs up to seven times cheaper than using the on-demand market and up to 44 percent cheaper than the best non-redundant, auction-market algorithm. We extend our adaptive algorithm to incorporate application scalability characteristics for further cost savings. In conclusion, we show that the adaptive algorithm informed with scalability characteristics of applications achieves up to 56 percent cost savings compared to the expected cost for the base adaptive algorithm run at a fixed, user-defined scale.},

  doi          = {10.1109/TPDS.2015.2508457},

  journal      = {IEEE Transactions on Parallel and Distributed Systems},

  number       = 9,

  volume       = 27,

  place        = {United States},

  year         = {Thu Dec 17 00:00:00 EST 2015},

  month        = {Thu Dec 17 00:00:00 EST 2015}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1109/TPDS.2015.2508457

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 6 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

Combining Partial Redundancy and Checkpointing for HPC

Conference Elliott, James ; Kharbas, Kishor H ; Fiala, David J ; ...

Today's largest High Performance Computing (HPC) systems exceed one Petaflops (10^15 floating point operations per second) and exascale systems are projected within seven years. But reliability is becoming one of the major challenges faced by exascale computing. With billion-core parallelism, the mean time to failure is projected to be in the range of minutes or hours instead of days. Failures are becoming the norm rather than the exception during execution of HPC applications. Current fault tolerance techniques in HPC focus on reactive ways to mitigate faults, namely via checkpoint and restart (C/R). Apart from storage overheads, C/R-based fault recovery comesmore »« less
Commercial Software for Low Pressure Industrial Applications Leveraging DOE Technology

Technical Report Deschenes, Timothy R. ; Grot, Jonathan ; Guler, Nevzat ; ...

There is a critical need for accurate and efficient gas flow simulation methods for low pressure industrial chamber processes. SSI is leveraging the SPARTA library, developed by our partners at Sandia National Laboratories, within a web-enabled, cloud-based graphical user interface, termed A Rarefied gas, Industrial Simulation Tool On The cLoud Environment (ARISTOTLE). The Phase I effort entailed development of the prototype proof of concept version of ARISTOTLE. We demonstrated the feasibility of the ARISTOTLE concept by successfully leveraging the SPARTA library in a web-based cloud computing architecture. The prototype entails a cloud environment, which leverages Amazon Web Services’ (AWS) Elasticmore »« less
https://doi.org/10.2172/1559105
Distributed Logical Analytic Domains (DLADs) for Distributed HPC Security

Technical Report Ros-Giralt, Jordi

The DOE estimates that the costs of a cyber attack onto its main high performance computing (HPC) facilities could potentially be in the order of billions of US dollars. The same conclusion is reached in the commercial sector, with a recent study by Oxford Economics reporting a combined annualized loss of $12.5 billion in shareholder value for publicly traded companies (more than $50 billion aggregated during the past four years) due to cybersecurity breaches. On average, the cost of cybercrime for organizations is worsening, increasing at an annualized compound rate of 23 percent, according to Accenture’s 2017 report on themore »« less
Industrial Compositional Streamline Simulation for Efficient and Accurate Prediction of Gas Injection and WAG Processes

Technical Report Gerritsen, Margot

Gas-injection processes are widely and increasingly used for enhanced oil recovery (EOR). In the United States, for example, EOR production by gas injection accounts for approximately 45% of total EOR production and has tripled since 1986. The understanding of the multiphase, multicomponent flow taking place in any displacement process is essential for successful design of gas-injection projects. Due to complex reservoir geometry, reservoir fluid properties and phase behavior, the design of accurate and efficient numerical simulations for the multiphase, multicomponent flow governing these processes is nontrivial. In this work, we developed, implemented and tested a streamline based solver for gasmore »« less
https://doi.org/10.2172/950479

Full Text Available
File I/O for MPI Applications in Redundant Execution Scenarios

Conference Boehm, Swen ; Engelmann, Christian

As multi-petascale and exa-scale high-performance computing (HPC) systems inevitably have to deal with a number of resilience challenges, such as a significant growth in component count and smaller circuit sizes with lower circuit voltages, redundancy may offer an acceptable level of resilience that traditional fault tolerance techniques, such as checkpoint/restart, do not. Although redundancy in HPC is quite controversial due to the associated cost for redundant components, the constantly increasing number of cores-per-processor is tilting this cost calculation toward a system design where computation, such as for redundancy, is much cheaper and communication, needed for checkpoint/restart, is much more expensive.more »« less

Similar Records